WARNING: The new custom location sync will delete thousands of files from Drive

Bugs and issues
#1

I reached out to customer service via email, but have not received a response. As this issue could be very costly for some, I wanted to post a warning here. Below is my email to Insync with a few edits to remove personal information:

I have ~ 100 GB of Google Drive files. I recently purchased a new desktop and wanted to sync all of my Google Drive files to my computer. I read about the new changes to Insync and was interested in being able to specify multiple folder locations.

My goal was:

  1. Have some top-level folders (Documents, Music, etc) synced to my fast SSD

  2. Have other top-level folders (Videos, Backup, Archive, etc) synced to my slower HDD

  3. Have every single file in Google Drive synced to my computer using one of the above

I set up the sync (see below), and started it. When I got up in the morning to check, all of the default location folders had synced successfully, but all of the custom location folders had not. None of the folders on my local machine had any files in them and Insync had deleted those folders and files from Google Drive! The files had never been downloaded, there was no .insync trash folder. In total, InSync deleted thousands and thousands of files.

This is the second time that the Insync software has deleted massive numbers of files from my cloud storage. Surely there must be a way to check if a sync configuration is going to perform large numbers of deletes and ask or warn the user? Or a way to validate what changes will be made when doing initial sync?

I followed Insync’s suggestions and blog post, and I think what I did is a pretty common use-case. What went wrong? What can I do in the future (if I continue to use InSync) to have some of my Google Drive folders go to one location, and some to another?

It looks like I may be able to recover the files through Google Drive, but I’m going to have to restore everything from Trash, which means spending hours figuring out which files really should be deleted, and which should not.

Setup:

  1. Downloaded InSync on 12/15/2018 (latest version, not going to start InSync to check version #)

  2. Signed in and read through the initial instructions, as well as the blog post (https://help.insynchq.com/key-features-and-syncing-explained/syncing-superpowers/sync-or-merge-a-top-level-folder-to-another-location-in-your-computer)

  3. I paused synching

  4. Changed my sync location to D:\gdrive

  5. From the file explorer, I used the non-selective Sync view (the bullet list 3 item icon to the top right was off/not-highlighted)

  6. For each folder in Google drive, I hovered over the folder and either selected the option to 'Sync to default location", or ‘Sync to custom location’.

  7. For all custom locations, I selected a root folder of E:\gdrive, and choose ‘Sync as a subfolder’

  8. This resulted in Insync creating sub-folders with names corresponding to the top level Google Drive folders (see screenshot #1).

  9. I then confirmed that every top-level Google Drive folder in the Insync file explorer had the Sync icon next to it, and the correct location (default or custom).

I then hit the play button on the bottom to resume syncing and watched the feed and messages for a few minutes. Everything seemed to be working, and I went to bed.

Screenshots:

Directory for the custom location folders - the folders were created but all are empty. InSync deleted each of these folders in Google Drive

screenshot 1

image%20(3)
image (3).png1110×643 33 KB

#2
  1. I paused syncing
  2. Changed my sync location to D:\gdrive
    …

This seems like a massive red flag, and probably the cause of the deletion.
Do you mean you used “account -> Files -> Folder Settings -> Move”? This is a massive (in terms of its impact) operation, and must be completed (not paused) before doing any other sync operations.
You changed sync location while pausing (so it’s not completed), and performed many other sync operations, only then did you resume syncing. This almost certainly confused Insync.

I routinely use the custom sync location feature, and performed many tests myself, have never seen mass deletion before.
Normally you shouldn’t pause syncing while doing these operations. It’s only when you’re hacking some unsupported/undocumented behaviors that pausing is needed.

#3

Do you mean you used “account -> Files -> Folder Settings -> Move”?

No, I mean that when running Insync for the first time, when prompted to use the default sync location, I choose the change option to specify a different location.

In terms of pausing, this seems like something standard I’d want to do. When doing a brand-new sync (really just a one-way, since I had no Google Drive files on my machine), I didn’t want to start the sync until I had all of the locations and settings configured. E.g. set-up, then sync.

I also had to set up an ignore list, which I forgot to include in the above steps. The only thing I ignored was the node_modules glob. From previous experience, I’ve found that if I don’t pause syncing before setting up the ignore list, I’ll end up with ignored files being downloaded/uploaded.

Also, the filed on D: (the default location) were synced correctly. It was the ones that were synced to the selective folders that were deleted from Google Drive.

If I can’t/shouldn’t pause syncing the next time I do this, what should the setup procedure be?

#4

Right, I just did a test. Turns out changing root sync location isn’t relevant in this issue.
You just can’t pause when asking cloud folders to sync down to custom local locations.
The folder is still in trash, so nothing you can’t recover.

This does seem like a bug in sync logic that should be addressed.

Full steps to reproduce this:

  1. Prepare a folder in the cloud, not synced with Insync
  2. Pause syncing
  3. In file manager interface, click “sync to custom location” on this folder
  4. Choose either “sync as subfolder” or “merge”
  5. Resume syncing
  6. Wait a while, then the entire cloud-side folder would be trashed

I’m on Windows 7 Pro 64-bit, Insync 1.5.5.37367

@mia

2 Likes
#5

@Grim_Echo

To avoid surprises like this, it’s still best to not pause. I understand that it’s easier for you to have a period of pause to set everything up, but having pauses in between sync makes it harder on Insync’s sync internal logic.
Ideally, Insync should handle most such cases, but bugs are unavoidable.

In your case, set up ignore list before you sync anything at all, then click the “sync to custom location” button on your folders one by one. No pausing necessary.

#6

Thanks for the verification @Hawk, and for distilling it to the core steps.

You mentioned in your previous post that you had used the custom sync location successfully. Since pausing seems to be the issue, I might try setting this up for round #2 without pausing first. But is there a sequence that will still allow me to ignore node_modules before any syncing occurs? If I add it to the ignore list after syncing starts, but before the first node_modules folder is actually transferred, will this be soon enough to prevent it? My understanding is that Insync first builds an index of what to sync, and its here where any checks are made against the ignore list.

What about using symbolic links as an alternative to custom locations? E.g. sync everything to the default location, (slow mechanical HDD), create sym links from the sync folder on the HDD to a new folder on a faster SSD, then move files from the sync folder to the linked folder via explorer?

#7

I believe addding ignore list before syncing anything should work.

This works, but is inefficient. The problem is that Insync doesn’t recognize a “move” when you move files across drive boundaries. Instead, it’s treated as a deletion + addition, so the file will be re-uploaded.

A workaround:

  • Don’t sync the original folder that holds your files.
  • Create an empty folder in the cloud, and an empty folder on your SSD.
  • Use symlink to sync these two empty folders.
  • On the cloud side (e.g. in GD web UI), move your files from their original folder to the new empty folder.
  • Files will be synced directly onto your SSD.
#8

Thanks for all the help @Hawk!

I’ll give that a shot (or I’ll try custom location sync again, but w/o pause).

Insync support responded with a short email saying that they let their developer know about the issue. I’m responding with a link to this form post.

#9

The pause button causes a lot of issue in this case and now I’m suspecting that in this and this issues I had, it was the same cause (at the beginning of setting up my new computer, I did a lot of moves, reallocation of folders, etc).

I give up to the many problems and I restart all insync related files, folders, etc and started a new and fresh insync installation, only using default locations. Now I have no problems related to (not)default, deleting, moving things around, unfortunately, this approach can’t be applied to your case.

Insync developers, could be a good idea to at least warm the user when it pauses the insync and it is doing some “maybe buged” operation?
Also, could this be reproduced shunting down inync, killing it, or just a computer freeze?, as this is something to worry about and to be very careful when doing some massive moving or operations. (all this while doing these conflicting operations)

1 Like
Does not keep Team Drive sub folders in sync since several weeks on Ubuntu 18.10
#10

@mia Any update on this? Confirmation of the issue from the developers and/or a planned fix?

#11

@Grim_Echo Having also an issue here: Does not keep Team Drive sub folders in sync since several weeks on Ubuntu 18.10

The support team does respond, but usually with a delay of several days. Their last email is not very comforting. It indicates that they are focusing on rebuilding the product from ground up and have not yet even determined since which version my reported bug appeared.

Tried downgrading to several older versions but no change in behavior. My fear is that downgrading requires not only finding a version that behaves correctly but also removing all accounts and starting fresh (which in my case is a really pain in the a%% as I currently have no idea which files are local only and which are in my online drive only).

Seeing you are asking for an update in a thread that was opened in Dec 2018 is not inspiring confidence in having any solution soon.

#12

@mia Still looking for an update or even a confirmation of this.