Major sync issues

#1

Hi there
I have commented elsewhere that I am in the process of migrating from Dropbox to Google Drive, and have about 215,000 files - about 600gb - of data I need to push into G Drive.
I had erroneously assumed that this would be as simple as downloading my dropbox files to my local machine, then moving the folders into my g drive folders and syncing via Insync.

The speed of the transfer is completely untenable, and I am now stuck with all of my files sitting in a folder on a local machine, and not syncing up at the speed I need them to.

I ran a test by connecting to my G Drive with Transmit (ftp client, and dropped a 1gb file in, and it took less than 30 min to upload. Which should mean, at that speed, I should be able to move my entire structure over to G Drive in a few days.

I want to stop syncing w Insync, but am concerned because now my local and remote folders are completely out of sync. Some remote folders are correctly populated, and some local folders are correctly populate - and I have no way of determining which is which.

So what do I do? If I stop the Isync process, then I will have a combination of local and remote files, and if I move to the native G Suite application, or my ftp client, I will still not be able to determine what needs to be uploaded, and what needs to be downloaded.

I’m sure there’s a resolution to this, but from where I’m sitting right now, it seems as if I might be stuck with just waiting for the rest of the year for Insync to get all of my files where the belong.

#2

Have Insync started uploading the files or is it stuck in “processing”? If it’s already started uploading, then speed isn’t really something Insync can really control.
Google Drive’s public API has inherent limitations. Your average file size is quite small (<3MB), which is the worst case scenario for Google Drive’s public API. Your test with a 1GB file isn’t a fair comparison, as large files are inherently more efficient to upload / download.

If you want another tool to help you figure out the sync status, try rclone.

#3

Hi Hawk,
I saw your response to my other comment, so thanks for that.

Insync is in the process of syncing. The problem is that it’s pulling files down, AND pushing files up, and I have no way of knowing what’s what. There are no errors, just issues with the speed of transfer.
So I’m not entirely sure what to do. If I stop Insync from syncing, I will have local and remote files/folders, that are out of sync.
If I try to use an alternative method to sync - such as Transmit, or Google Stream (or whatever it’s called), then I may be overwriting remote files/folders from local files/folders that have no content (because they’re pulling from remote).

So if I don’t want to accidentally screw everything up, it seems I need to just wait for Insync to complete the interminable syncing process, that at this rate could take the better part of six months.

Thoughts?

#4

This sounds odd. What were the starting states of your Google Drive and local folders? Were there files on your Google Drive that were not in your local folder? Right after you set up Insync, what steps did you take to start syncing?

Insync’s initial sync is selective by default, and there shouldn’t be any downloads. If there’s some downloads, then either you took some extra actions, or there’s a bug in Insync. You might consider sending the logs to support.

This shouldn’t happen if the client is properly built and you use the right options.
To achieve a higher speed, Google’s official client (Backup & Sync for regular consumer accounts, Drive File Stream for GSuite accounts) is the only meaningful option. Any other third-party client would also have difficulty with public API’s restrictions.
You can start with some small test sets to make sure everything works just right. In particular, I think Backup & Sync has a “upload only / never delete” mode somewhere.

#5

Thanks for replying, @Hawk! Your Insync knowledge is all kinds of amazing and awesome. :muscle:t5:

Hey @Cameron_Burgess! As what Hawk said, it’d be great if you could shoot us an email at support@insynchq.com with a link to this Topic and your log files. You can learn how to get your log files here!

#6

Thanks Hawk,

This is not a new user sync - I’ve had Insync setup on this machine for years. The issue is that I have remote files in G Drive that are not on my machine, and I had local files from Dropbox that I moved to my local G Drive folder to sync UP to G Drive remote.

I had expected, as was the case when moving files around on Dropbox, that a couple of hundred gig would sync in a few days, and I would be able to see what folders were fully synced, partially synced, or not synced at all. Obviously I can’t do this from the Finder, but I can do this from the menu bar.

I have discovered some duplicate files and cleaned out about half of the files I was attempting to upload, but I am still stuck on over 130,000 files yet to synchronise.

As it stands, I am now contemplating porting everything back into Dropbox, and shutting down Insync altogether. Having all my files sitting on a machine that, while backed up, has no remote storage, is a real issue. I stand to lose 15 years of work in the event of a catastrophic failure.

#8

Thanks for the explanation. It’s much clearer now. So this seems to be a pure speed issue.

I would also expect a few hundred GB to take a few days. Your large number of files should add some overhead, but (even for Insync) I’d estimate it to be on the order of hours, certainly not weeks.

One possibility we need to consider is that your downloads may be given higher priority than your uploads, making the uploads seem unproportionally slow at this stage. How many and how large are the files to be downloaded?

You can also check Insync’s network usage stats through OS. I’m not familiar with Mac, but there should be some tools available to do this. If Insync is already using most of the bandwidth, then there isn’t much to worry about.

You can use Google’s official client and Insync side by side BTW (by using selective sync). You may upload the files migrated from OneDrive using Google’s Backup & Sync, and let Insync complete the downloads. Be sure to uncheck “sync new children of partial folder” in Insync if you use this setup.
Later, if you want to switch to full Insync setup, you can let Insync do a resync. A resync without actual uploads / downloads should take just a few hours.

#9

Wanted to advise that landing in Stockholm on a 100mbps uplink solved the upload issue.

1 Like