Why Insync takes so long to actually start sync'ing?

Bugs and issues
#1

Ubuntu 14.04. Every time I turn on my computer, it always takes tens of minutes before Insync starts to actually sync files. The insync get_sync_progress shows No syncing activities, and the GUI shows a blue circle icon and No syncing activities or No ongoing transfers. The computer syncs data every day, so there are usually only dozens of files for syncing. How to speed up such initialization process?

EDIT:
I observed that when Insync actually starts sync’ing, the PROGRESS in the GUI shows there are tens of thousands of files waiting for sync’ing. Apparently, Insync spends a huge amount of time figuring out which files have to be updated. And most of the time when a file is processed, the GUI shows uploading/downloading 0.0 bytes of x.x bytes. A simple conclusion is that those are the files in the folders already in the IGNORE LIST, but Insync still processes each of them without uploading or downloading. This is really inefficient.

Why does not Insync just ignore the folders (not at the file level) directly in the first place, and not to examine the files one by one?

2 Likes
Have a not to upload and/or download option for Selective Sync
#2

Right now, this is the current behaviour of Insync when syncing. We are continuously improving Insync on how to optimized the syncing process.

Thank you for your understanding and patience.

#3

This doesn’t happen in each version and each system, though. It used to happen on my Ubuntu desktop (prolonged and irritating disk thrashing), but it stopped with version 1.2.10, I think. Now it just takes a few seconds But it still happends on my Windows 8.1 laptop. I’m going to replace it with Linux as well, so I hope this behaviour will stop on the laptop, too.

#4

A few seconds? OMG, that never happened to me. You’ve got to let me know your story.

#5

Actually, it was like that at the beginning, I used Insync for quite a long time without any such problems, then it went wrong probably with version 1.2.9 - the disk thrashing started each time I turned on the machine. I reported this problem and then - very soon - after I updated to 1.2.10, it disappeared. So I thought the developers had responded to my report - I was actually amazed how fast they fixed it. Now I see that perhaps it was just a coincidence. I don’t know, for some reason the problem disappeared. I’ve now got version 1.2.11 on both my Linux desktop and my - newly purchased - Windows 8.1 laptop. Unfortunately, the delay and disk thrashing does occur on the laptop. Which is very annoying, because I do things on my desktop at home and then I take the laptop to work and want to continue - but I have to wait some 10-15 minutes until the files are actually synced. When I turn my desktop on at home, there’s no delay, I can immediately access the changed files (well, maybe after half a minute, depending on how many files have been changed, but the syncing starts immediately). I’m going to install Ubuntu on the laptop soon and I hope the behaviour will be the same as on the desktop, i.e. no delay.
I’m surprised Roald here says that this is the current behaviour of Insync, because I don’t have this behaviour on the Ubuntu desktop, and the files are synced perfectly, so clearly this behaviour is not necessary.

#6

@yltang What version are you using? There was a change introduced in 1.2.8 that caused slowdown for some users which was fixed in 1.2.10. This might be what @Pawel encountered, although it should only affect Linux. If the initialization is slow in other OS or if it is still slow on versions after 1.2.9 then it’s a different issue.

#7

Version: 1.2.11

Eclipse has a lot of setting files which are stored in .metadata directory. And the Python virtual environment (I name the directory as virtual.env) also has a lot of files. I include .metadata and .env in the IGNORE LIST.

I think these two directories may be the cause of the problem. When initializing, Insync always shows there are tens of thousands of files in PROGRESS and it takes tens of minutes before Insync actually starts scync’ing files.

#8

It’s an issue nevertheless…

1 Like
#9

@yltang We are aware of that issue and are currently working on it. Thanks for your patience.

2 Likes
#10

I’ve asked this question a few times, so I apologize for the repetition, but I use Insync on Windows 7 (among other systems). I have a very large sync directory (~300GB), but on my Windows machine the initial sync completed months ago, so it’s all up to date. However, upon starting InSync, it still takes hours if not days for InSync to sync files I have synced from other computers. Right now I am sitting at work and waiting for InSync (which I opened 1.5 hours ago) to sync about twelve files that are sitting on my Google Drive. I am using InSync 1.2.13.

I will continue to ask this until someone answers me: Is this normal/common behavior? Or is this a bug?

2 Likes
#11

Update: The long syncing times are not just for files created and synced from other machines. I just created a directory and four text files on my work Windows 7 machine and it is now three hours later and they have not synced yet. InSync’s Progress tab simply says “Syncing Metadata…” and has said that since I started the software 3 hours ago.We’re talking about less than a single MB of data to be synced, and I can’t turn my computer off when I leave work in hopes that it will eventually sync this.

I ask again (because I paid for this software): Is this normal behavior, or am I an outlier?

#12

@Christopher_Stuetzle: How many number of files & folders are there in your Insync folder? If there are large number of files present locally, then the initial folder scan during the start or after you resume Insync may take long time and you may not observe many transfers during this phase.We are working on improving this.

Does syncing work fine after the syncing metadata phase during the start is over? If not, then kindly send send to us the logs.db and compressed copy of dbs folder (present next to logs.db) to support@insynchq.com

Thanks

#13

@Christopher_Stuetzle: Another thing, are you syncing the same/comparable number of files on Linux? Do you not experience the long “Syncing metadata” phase during the start on your Linux installation?

Thanks,

#14

After the metadata sync finishes, the actual file sync seems to work fine. But it takes days to get there. Yes, the directory is very large and has a ton of files, so what it sounds like is that this is typical behavior. I do not have the same issue on Linux, it seems to work quickly.

I ran an experiment, turned off all of my home machines, and made some changes to my Drive (small ones, a few MB at most). Then, I started my Linux machines and started up InSync. It signed in and synced the 4MB of files I had added while the machine was off in about 12 minutes on one machine in about twenty minutes on the other (still too slow for my taste, but still not awful). My Windows 8 machine took about 95 minutes to sync. My work computer (the slow one that I’m really worried about) is Windows 7, but while I’m home I can’t check that one. However, history tells me that it would take hours or days to sync these small changes.

My primary concern is that files I change while at work don’t sync in a timely manner so I can’t view them when I get home, and vice versa.

#15

@Christopher_Stuetzle: Is your Insync folder on Windows machine mapped to a network drive? Please share with us the compressed copies of the “dbs” folders from your linux and windows machines to support@insynchq.com. The dbs folder resides in the same directory as that of logs.db file. Please copy the dbs folders while Insync instances are stuck.

Thanks.

#16

I cannot, as the file size (even compressed) far exceeds the limit for attachments GMail. The database file inside the dbs directory is 675MB.

Is there another way to work around this?

P.S. I made one change to one text file in one folder 45 minutes ago on a fully up-to-date sync and it’s still not synced.

#17

@Christopher_Stuetzle: Please upload it to Google Drive and send to us the share link. Did you change the text file remotely? Are you waiting for it to download the changes or upload? Please create a copy of the database while it is running and stuck.

Thanks

#18

Just wanted to know if there is a solution to this problem yet ?

is this someting Insync is working on trying to fix?

I am no developer so dont know the reason behind having to resync meta data just cause we quit and star, cant we save the meta data information in a file when a shutdown is triggered and then when it start up meta data information is restored.

Many many users dont know about this metadata inssue with insync.

over 50% of the post here are about “not syncing”, “syncing stuck” , “metadata issue”. “taking long time”,

all these common errors are related to single problem “metadata resync” on start up.

this only effect users with many individial files we talk about minimum 25k and the more the longer time.

in my case for example i use headless version on my server, and it takes me over 24 hours to start sync if i do a simple quit and start command. now i know about this issue but many does not and believe its a bug or so.

because we live in 2015 and no one expect things to take this long.

I think its an important issue and insync should take this more serious and eventual hire external developers to fix the problem if its beyond their scope of knowledge. as the issue has a damaging effect on their product.

more than 80% properly don’t even bother to report the issue and just believe “its buggy”.

But the truth is insync is a GREAT! product i have studied it very well and it actually works super great. if just they fixed the above mentioned issue.

is there any plan for this ?

#19

Yes, we already started making steps towards this. Our goal for the next major version is file matching (so that you can re-use existing files and no need to re-upload/download). The next major version after that is fixing most of the speed issues.

Thank you for your feedback!

#20

Reviving an old topic here - I have an excessively large data structure (~1 TB) and the initial file checking takes 3-5 days. If I don’t turn off the computer, then it works fine, but I have to update sometimes.

I’m assuming the program is checking each file via md5 or other hash and this is why it takes so long? I don’t change most of the files on a regular basis - is there a way I can disable file content checking and just use a less intensive check? For example, a check by modified date would work. Linux rsync uses a live database that builds incrementally - maybe we could force something like that for large sets of files?

I’m willing to get my hands dirty and hack and program and debug if necessary.