Subfolders of Ignored Folder are being queued on every startup

Bugs and issues Windows
#1

Hey There,

I just recently added “.lrdata” (Adobe Lightroom Image Previews) as folder extension to my ignore list. It seemed to work perfectly fine until I rebooted my machine the next time. Insync now shows about 12k Items queued, walking through all of the sub folders in the .lrdata folder. saying “uploading” for each one. My SSD is utilized up to 100% until it has finished ALTHOUGH these folders are on my external drive which is EVEN DISCONNECTED.

after doing my homework i found out that insync is jamming the SSD with massive write requests to a temp file named “etilqs_zbTu8aFF5NZ65r6” which currently has about 4MB and of course a lot of write requests to the DB.

At least the log file is no longer spammed with entries since i just updated to the latest version (1.2.5.35099).

So my question: why does Insync iterate through all the subfolders of an ignored folder? and why is that happening even though the actual drive is not even connected?

One more thing: when i click on the magnifying glass while the external drive is not connected, a ton of these log entries appear:
[0402/231342:INFO:CONSOLE(0)] “Uncaught TypeError: Cannot read property ‘name’ of undefined,” source: file:///C:/Users/Wolfgang/AppData/Roaming/Insync/App/res/html/js/main.js(46360)
[0402/231342:INFO:CONSOLE(0)] “Uncaught TypeError: Converting circular structure to JSON,” source: file:///C:/Users/Wolfgang/AppData/Roaming/Insync/App/res/html/js/main.js(47102)

when i connect the external drive, a click on the magnifying glass leads to exactly that subfolder inside the ingored folder.

I added some screenshots:
Disc utilization

write requests to temp file:

item queue (went down from ~13k to 565 while i was writing)

cheers!
Wolfgang

#2

ok, so now i also ckecked the contents of the temp file. looks like another log file to me. There are a ton of write request to this file. Why are you writing so many logs? 1 text file 1 database and 1 temp file?!? i’m not surprised you had performance issues in the past. is there a possibility to turn off logging?

Furthermore I found out that you are also keeping trashed files in the database even though they have been permanently deleted from google drive and should be ignored locally. Is there a possiblity to clean up the DB? i don’t really want to reinstall insync

#3

ok, so now i moved the “ignored” folder from my sync path and first i was still observing the same problem. then i further analyzed your SQLite database and found out, that you are keeping entries for all google drive files and folders which have already been deleted permanently from google drive (not just trashed, but deleted permanently). So i just removed all those entries and the relationships from the DB and the described problem with the Item queue disappeared :blush:

although I have to say that the disc utilization is still up to 100% for quite some time at startup and I guess that you are checking each individual file for something (which are in my case about 65000 files). Is this absolutely necessary at startup? My Insync is always running and during a restart of my system, not much can happen to the files. It’s just annoying to have the disc jammed by your program!!!

cheers
Wolfgang

#4

no response? did you even to take note of this post?

#5

@silentdrummer: Really sorry that we didn’t get back to you in time.

Re: the items getting queued issue, I confirm that in case the folder is being ignored, on every re-run, Insync “un-ignores” the children first which leads to upload tasks which are then discarded as soon as it realizes that item’s parent is ignored. We will work on fixing this behaviour soon.

As to logging, the text file is there as a fall back for the errors that can’t be logged to the logs.db database. It should not affect the performance. The temporary file that you are talking about is being used by the account database (%appdata%\Insync\dbs\gd2-[account_id].db) for database activities such as transaction control.

Yes, currently, Insync does not delete the entries from the account database of the files that have been permanently deleted remotely. Unless there are way too many permanently deleted files, this is not a considerable overhead. Having said that, we will consider removing such entries as those are not useful except when some of the users seek clarification for the events happened in their drives. However, I fail to understand how removing such entries fixed the queued items issue for you. Did you remove entries for the ignored folder’s subfolders?

Regarding the high disk activity during startup, yes it is because Insync scans the entire local folder on high priority and fetches the drive changes and processes them during the start up. This involves consistent access to the database and the folder. We are aware of this issue and are working on improving the performance & resource utilization during the application startup.

Thanks!

#6

@dipesh: thank you very much for your detailed response! it’s good to know that you are working on it!

as for the queuing: I can’t exactly remember what I did but I ended up removing the folder from the synched folder structure and just linked the files I needed via a junction.

I appreciate this issue being fixed soon!

#7

any news here? would be nice to have this solved!

#8

@silentdrummer: The ignore files/folder issue has already been fixed. Re: performance improvement & resource utilization, those are being continually worked upon & shall get manifested to a greater extent in 1.3.

Thanks

#9

I’m don’t know if my problem is related to the above, but I’m having 340K files being removed from the queue without any further action ie uploading or downloading. I believe its because I tried to sync a folder and then decided not to sync it by disabling it in the selective sync option. Its been doing this for the last 2 days. Is there a way to clear the queue, and get insync to resync properly?

#10

@geek_club: The problem doesn’t seem to be related to ignore feature. If 340K files were once removed from the queue, those should not be added again. Although Insync does scan the changes for the unselected files/folders as well but the changes should not be that many. Could you send across the logs and compressed copy of the dbs folder (present next to the logs) to support@insynchq.com so that we could look further into it?

Thanks