Feedback wanted: Ignore List improvements

Feature requests
#1

Hi guys!

We are in the midst of making improvements to a powerful but neglected feature: the Ignore List. But before embarking on a coding journey, we want to hear your stories and the circumstances that require the ignore list feature.

If you can, answer the following when describing you use case:

  1. What job are you performing that requires the Ignore List?
  2. What were you using as a solution before Insync?
  3. What jobs could the Ignore List be solving but currently isn’t?
  4. What difficulties are you having with the current ignore list that prevents you from doing the job or the work that you’re trying to do?
  5. How does the ignore list improve your daily work flow?

Please share your stories to help us build a better product. If you prefer to share privately, email me at josh.cabalinan@insynchq.com with subject: “Ignore List”.

Thank you! :slight_smile:

3 Likes
Feedback Wanted: Windows Server Users
#2
  1. Ignoring temp build files for software projects, basically a .gitignore.
  2. Dropbox, where I had to manually delete folder content and unselect it in UI.
  3. Would be cool if it supported text-file-based config like .gitignore for each folder, or even importing .gitignore. Also would be good to be able to backup the list or import/export on new PC, account etc
  4. Can’t ignore some files because there are other folders (projects) where files with the same name/extension should not be ignored (like *.obj files in C++ builds vs *.obj 3D models)
  5. Saves space and sync time, avoids build errors (sometimes there were random failures at least with dropbox when it tried to sync temp files during build)
1 Like
#3
#5

I actually purchased InSync for this feature. I have found no other comparable solution. I wanted it so there wasn’t constant syncing of .git files and other project-related files when I do software development. I could do something like de-select them in Dropbox but I did not want to have to do this for each project.

Thus far, I’ve had little success with InSync. This is partly because getting started made it difficult to implement the rules in time for the initial sync to ignore them. After that, removing them from selective sync removes them from my computer, which is a real pain to deal with since I then have to go recover them but in a way that doesn’t cause Insync to resync them. If I could tell InSync to ignore all the files that match the filters, regardless of whether they have previously been synced to Drive, I would be very happy.

1 Like
#6

yes, applying the rules retroactively would be helpful :slight_smile:

#7

hello all :slight_smile:

would simply adding .gitignore support per folder satisfy your “ignore list” requirement?

1 Like
#8

Using .gitnore does nothing for me. I agree that a per folder or path based feature would be fantastic! For instance I use MPLAB X and NetBeans. These generate many intermediate object and debug files that I don’t care to sync. Often times the sync hinge of these files causes IDE errors. If I could make a rule like: “MPLAB_Projects/*.o; *.o.d” would solve my issue. Right now I have the extension exclusions set globally which could cause inadvertent problems in other folders.

This also brings up the point that we currently seem required to enter every exception on a separate line. Many other applications allow the concatenation of exclusions with a comma or semicolon delimiter.

#9

Hi @J_Garnero

What other intermediate object and debug files do you not want to sync (I’m not a dev so I’m curious on how users who are developers would use the Ignore list)?

What does your work flow look like? What problems do you run into without an ignore list (you mentioned that the syncing of the aforementioned file types cause IDE errors and that extension exclusions set globally could cause problems in other folders)? How would the ignore list (or the inclusion of the feature upgrades you mentioned) make your work flow better? How would it look like if the feature upgrades you mentioned were included?

#10

Hi @Alex_P

What does your work flow look like? How would an ignore list that ignores temp/project files by path help in that work flow? Can you give us a specific example so that we can imagine what really happens: the problems that you experience, the workarounds that you have to do, without an ignore list that does such features?

I’m trying to imagine what it looks like on your side and really trying to put myself inside your shoes since I’m not a dev and I’m curious as to how improving our ignore list feature would help you do your work better.

#11

In most software projects there are lots of generated/temp files, caches etc. (such as files generated during intermediate compilation steps or third party libraries downloaded by package managers like Maven, Composer, Pip) that should not be uploaded anywhere because they can take a lot of space and/or regenerated often, like every time I build/run a program to test it (and sometimes this can even cause some file access conflicts if compiler and InSync try to access/delete files at the same time), and they can be easily regenerated.

So current ignore list works mostly fine, but there are things that can be improved like I mentioned above.

  • Separate ignore lists per folder would simplify filters and solve the issue with similar file names like the one I mentioned: in C++ projects *.obj files are generated by compiler and should be ignored while in other folders it can be a 3D model or something else that should not be ignored.
  • Backup/import/export: I don’t know where InSync stores ignore list, is it per account or per device? If per device, then it would be very inconvenient to retype all filters if I move to a new PC. And if per device, then export still could be useful to share with other people.
  • Also import (or just copying) would be very useful for per folder filters (if you add this of course :slight_smile: ) to avoid retyping filters manually for all similar projects.
  • Importing per folder filters from .gitignore or just supporting it can be useful (and would probably solve all of the issues for me), but probably not very important, usually there is not that much file types/folders in a project that have to be ignored by InSync and sometimes it can be even useful not to ignore some files from .gitignore in InSync (for example user settings). So maybe a separate file (.insyncignore) would be better

I hope I managed to clarify it, feel free to ask more questions :slight_smile:

2 Likes
#12

It would be useful to be able to specify the full path of the to-be-ignored file or directory, or perhaps its path relative to the root of the synchronization.

For instance, currently I sync all files under ~/gdrive, and if I add foo to the ignore list, any file or directory anywhere in the subdirectory tree under ~/gdrive will be ignored. If I wanted to exclude only ~/gdrive/foo, it would be useful to be able to specify ~/gdrive/foo or perhaps /foo (relative to the root of the synchronization).

#13

Hi @jbec

@Alex_P and I appear to be on the same page. I use Insync to keep my software development projects on gdrive so I can access them from my office, home, or portable computers. The use of the ignore list allows me to leave the project in the synchronized folder because I do not have to worry about the frenzy a compile, assemble, link cycle creates for any sync client. When I build a project for debug it creates multiple intermediate output files along the way to creating the final .hex file. If these intermediate files are allowed to sync I often get runtime compile error of the file not accessible nature - likely due to multiple access of the sync client due to the build operation replacing old intermediate files with the newer ones. Insync, I would imagine gets behind the process due to the immediate replacement of a file and possibly before the “old” one is deleted from gdrive, a new one needs synching.

Why would I like folder (path) specific filters is easy. If I have more than just software development projects on gdrive (which I do) to globally exclude a file extension is asking for trouble somewhere else. Just consider the generic type of .obj and .dat. DAT could be configuration data or a database, a tempfile, or any possibility. To deny the sync of .dat could prove extremely detrimental.

Perhaps an example:: I create a scratch folder in my project folder in my Insync folder. Inside this folder is a jpeg image that I reference but don’t want it to go to the cloud. I create an ignore filter of “.jpg”. What happens to every other jpeg image on my gdrive from now and into the future? Yes, not the desired purpose of the ignore list, I suspect.

I hope this helps clarify.

#14

@marcelpaulo isn’t it just Selective Sync?

#15

@Alex_P, I wasn’t aware I could unsync a specific directory, but … To unsync it, it had to be synced first, and my idea was not to sync it to start with: to add it to the ignore list before it was created.

Perhaps if I explain what usage I had in mind, my request will be clearer. I’m backing up a local directory to an external USB drive in a way that renames and directory changes are detected (it would be fabulous if insync could also do this !), so, following these tips, I do:

  1. Create a shadow directory under the source directory with the same directory structure as the source directory, but hard-linking all regular files
  2. rsync the contents of the source directory — which include the shadow directory —, preserving hard links, to the destination directory
  3. Update the shadow directory in the source directory
  4. Update the shadow directory in the destination directory

This will prevent renamed files (or files which have been moved) to be transferred again. So the ~/gdrive directory will have a shadow directory directly under it which I don’t want to synchronize with my Google Drive. In order to selectively unsync it, I would have to create it, let insync synchronize it with Google Drive, and then unsync it. The result would be: as the shadow directory has hard links to ALL files under ~/gdrive, insync would upload again a second copy of ALL files. I wasn’t aware of the ignore list feature, so that’s what happened why I tried out this shadow backup.

In any case, it would be useful, if not necessary, to be able to specify files and folders in the ignore list anchoring them somewhere in the directory structure, either with an absolute path or relative to the root of the synchronization. rsync has very flexible and comprehensive ways of specifying the exclude patterns, it would be useful to have some it implemented in insync’s ignore list.

#16

Not directly related, but it’d be nice to have a possibility to be notified of big changes that are about to happen - e.g. a 3 GB file appeared in a synced folder, or 3 000 files appeared at once. I’d like to be notified of such extreme case and be asked if I really want to sync these files or rather add them to the ignore list. The same on other machines after these big changes have been made somewhere else - ask the user if he really wants to sync such an amount of data/files, or if he wants to selectively unsync them (before the actual sync happens).

1 Like
#17

First of all, thank you so much for looking into improving Ignore List!

  1. What job are you performing that requires the Ignore List? - Development. For example node_modules folder might contain more than 50k files. Some ML files are super large and should not be synced.
  2. What were you using as a solution before Insync? - Bought Insync and swithed from Dropbox to Google Drive because of the Ignore List.
  3. What jobs could the Ignore List be solving but currently isn’t? - As mentioned above, folder specific ignore lists, notifications/confirmation on adding big or large amount of items, retroactive rules. I would love something like .insyncignore just using .gitignore wouldn’t be enough because I want to sync some project files that I do not want to commit to git.
  4. What difficulties are you having with the current ignore list that prevents you from doing the job or the work that you’re trying to do? - Its hard not to accidentally sync unwanted files and then to get them out of syncing again.
  5. How does the ignore list improve your daily work flow? - My macbook would otherwise be constantly on 99% CPU trying to sync up all the dist, node_modules, etc folders with many, large or rapidly changing files.
2 Likes
#18

As mentioned by @mart.vaha, the negative consequence of not ignoring the temporary files created by dev projects is lots and lots of syncing. Just the .git directory in large complex can be many hundreds of files. Other caches and temporary files can be constantly changing meaning in many cases a large number of files are constantly being synced in the background. Some programs will even run into issues with cloud backup apps when both are trying to access the same file.

#19

Hi @Joe_G and all,

Thanks for the great reply! That helps me see what you, and a lot of other people it seems, have problems with when it comes to developing your own projects. I’d also like to thank everyone who took the time to reply to this thread and help us see how we can improve our app and make it better for you guys to use: @jacob2, @mart.vaha, @peci1, @marcelpaulo, @Alex_P

To sum it up, the problem seems to be that a lot of generated/temp/cache files get created when developing projects. If left to sync or if these files are unignored, these can cause problems/errors when running the project itself as well as leave issues with syncing, space, and etc. Ideally, the ignore list would solve this problem by ignoring those files (in specific folders) by not allowing them to upload/download to the Drive or to the local computer.

Did I get that right?

Also, I understand that you can store/backup projects on Github. What do you guys use Insync for then? Again, I’m not too familiar with Github and how you guys work so I’d appreciate it if someone could walk me through your use cases or how you guys like to do things.

#20

@jbec I personally don’t sync git projects on Google Drive. Maybe unintentionally when I forget some folder is synced.

The bigger problem is this: I set the Documents folder to sync except some ignored folders. Then I install an app like video editing software, which thinks that saving its media and intermediate files in Documents is a good idea. And it takes some time to figure out that there’s a new folder in Documents that’s growing quite big… This is when I’d want to be warned.

#21

Perhaps an option to not automatically synch new folders or files? But this could also create a mess down the road.