Upload stuck on file at 0%, no errors

Bugs and issues Linux
#1

Running insync-headless 1.3.14.36131 on Linux (Mint 17.03). Uploaded nearly all of more than 200,000 files, but is now stuck. Been showing this for hours:
VESTA ~ # insync-headless get_sync_progress
Uploading
19 (0% of 1.1 GB)

I assume that ‘19’ is the name of the file it’s syncing. Is there any way to see where that file is? I have hundreds of files and directories named ‘19’.

There are no errors. I’ve stopped and restarted it with no effect.

Update: lsof does not show ANY files open by insync in the directory tree that’s being uploaded. Other changed files seem to be getting synced. Insync task shows continuous high CPU and I/O - top CPU task, one of top three or four I/O tasks.

#2

tagging our engineer @lpugoy

#3

@Bill_Kuhns: Apologies for the trouble. Please send your logs to support@insynchq.com for investigation: How to find the log files.

#4

Log file is 30 meg. Can I delete it and run insync again to create smaller log? I assume I’ll still have the problem.

Update: Did that, got same size logfile and same problem. I can extract the last n records from the logfile if that would help:

sqlite> select * from event_logs order by timestamp desc limit 2;
8e4174333a5c495da364b95f878d6bea|1|b90ce254e56a46bb945ff8fdcf4f7f6b|idesksyncer.syncworks.GDDeleter.__run|87f69fd4a909d60f0afc3196b5daaf08|1485949120.30623|20|root in info|logging/__init__.py|1600|117656662660099360660|wkuhns@vecs.org|{"message": "File %r was not deleted.", "params": [641140]}|||{}
70f0397ec9d74807ab821fab4e552553|1|b90ce254e56a46bb945ff8fdcf4f7f6b|idesksyncer.syncworks.GDDeleter.__run|e3f4627c2968f7c6fa12d84b36441ad7|1485949120.29987|40|ideskfs.fs in add_modes|logging/__init__.py|1575|117656662660099360660|wkuhns@vecs.org|{"message": "While deleting %r at %r.", "params": [641140, "/proc/30036/fd/11"]}|{"type": "OSError", "value": "[Errno 1] Operation not permitted: '/proc/30036/fd'", "module": "exceptions"}|{"frames": [{"function": "delete_file", "abs_path": "idesksyncer/fsstate.py", "vars": {"path": "/proc/30036/fd/11", "self": "<idesksyncer.fsstate.FSState object at 0x26e4dd0>", "file_info": ["'L'", "1485946292.628", "64", "'4/12272409'"], "fs_id": 641140, "permanently": false}, "module": "idesksyncer.fsstate", "filename": "desksyncer/fsstate.py", "lineno": 722}, {"function": "delete_link", "abs_path": "ideskfs/fs.py", "vars": {"self": "<isyncd.linux.fsimpl.LinuxFSImpl object at 0x26695d0>", "link_path": "/proc/30036/fd/11"}, "module": "ideskfs.fs", "filename": "deskfs/fs.py", "lineno": 117}, {"function": "make_removable", "abs_path": "ideskfs/fs.py", "vars": {"self": "<isyncd.linux.fsimpl.LinuxFSImpl object at 0x26695d0>", "full_path": "/proc/30036/fd/11"}, "module": "ideskfs.fs", "filename": "deskfs/fs.py", "lineno": 311}, {"function": "add_modes", "abs_path": "ideskfs/fs.py", "vars": {"curr_mode": 320, "self": "<isyncd.linux.fsimpl.LinuxFSImpl object at 0x26695d0>", "st": "posix.stat_result(st_mode=16704, st_ino=12272397, st_dev=4L, st_nlink=2, st_uid=0, st_gid=0, st_size=0, st_atime=1485946292, st_mtime=1485946292, st_ctime=1485946292)", "mode_bits": 128, "new_mode": 448, "full_path": "/proc/30036/fd"}, "module": "ideskfs.fs", "filename": "deskfs/fs.py", "lineno": 318}]}|{}
#5

Update: Same problem now, two files, different file names (if that’s what those numbers are):

VESTA Insync # insync-headless get_sync_progress
Uploading
21 (0% of 1.1 GB)
11 (0% of 26.2 MB)
1 files queued

More update: It settles down to a single 1GB file. According to the log, that file is in the /proc directory, and is a symlink to Insync’s own log file(!) - /proc/30036/fd/11 currently.

It also appears to be watching a lot of certificate files that are nowhere near the directory that it’s supposed to be syncing. When I pause, it gives a long series of messages like this:

unwatching /usr/share/ca-certificates/mozilla/AffirmTrust_Networking.crt

What’s that all about? Here’s the ONLY directory tree it should be looking at:

VESTA Insync # insync-headless get_account_information
xxxxx@xxxx.org
Quota:
  Total: 117.0 GB
  Drive: 29.6 GB (25.28%)
  Others: 2.0 GB (1.73%)
  Trash: 5.3 GB (4.54%)
  Remaining: 80.1 GB (68.46%)
Folder location: /home/insync/2017-01-16-vecs
License: Plus
Export option: link
#6

@Bill_Kuhns: If you have symlinks under /home/insync/2017-01-16-vecs then Insync will follow them, which could explain the line unwatching /usr/share/ca-certificates/mozilla/AffirmTrust_Networking.crt. Please confirm if this is the case.

For the logs if size is the issue you can compress it before sending to save space. If that is still too big please send instead the entries for the last run_id. Though it would be preferable to send all of the logs.

#7

OK - the two are related, I think. The directory tree that I’m syncing contains a disk image for an embedded system that contains absolute symlinks t0 /usr/share and /proc, among other things. Can I supress following symlinks for selected directories?

#8

@Bill_Kuhns: I see. No, that’s not possible unfortunately.