Multi-Thread Upload/Downlaod

Feature requests
#1

Can you add the ability to add multi-threaded upload and download and also set the number of threads?

Kind regards
David

PS Great software

18 Likes
#2

Would second this. Currently, the upload slows to 10Kb/s when Insync is uploading small files eg php files, since Insync uploads 2 files at a time. If there was a way to set the number of threads, it would speed up the uploading considerably.

1 Like
#3

I’ll +1 you on this. :smile: Thank you for your support

1 Like
#4

+1

Currently I am uploading ~200.000 small files and it looks like its going to take several days. :frowning:

2 Likes
#5

Because the uploading process was painfully slow (less then 70.000 of 200.000 files transferred after more then 24 hours) I decided to do the change myself. Here is how:

  1. decompile /usr/lib/insync/isyncd.deskcore-0.0.0-py2.7.egg/idesksyncer/syncworks.py:

     uncompyle2 syncworks.pyc | grep -v "+++ okay decompyling syncworks.pyc" > syncworks.py
 mv syncworks.pyc syncworks.pyc.bak

  2. Change the number of threads in syncworks.py: replace size=3 and size=2 with whatever you want.

     self.__pools[gddb.WorkCategory.Two] = gevent.pool.Pool(size=3)
 self.__pools[gddb.WorkCategory.Three] = gevent.pool.Pool(size=2)

  3. Restart Insync

The sync is now much faster to me (30 files/min -> 200 files/min with size=30 and size=20).

5 Likes
#6

Hey,

Can you upload the file online for me, im getting syntax errors all over the place when i try to decompile it ;9

ty

#7

Due to legal reasons this is probably not a good idea but if you show me what syntax error you get I may be able to fix it for you :slight_smile:

#8

I think that the problem is that it fails to decompile.
Maybe its the decompiler that i use because i cant get uncompyle2 to work ;/

#9

What error message did you get?

#10

357 POP_TOP None

358 LOAD_GLOBAL ‘gevent’
361 LOAD_ATTR ‘sleep’
364 LOAD_FAST ‘next_time’
367 CALL_FUNCTION_1 None
370 POP_TOP None
371 JUMP_BACK ‘129’
374 POP_BLOCK None
375_0 COME_FROM ‘126’

Syntax error at or near `POP_BLOCK’ token at offset 374

decompiled 0 files: 0 okay, 1 failed, 0 verify failed

2015.09.08 19:02:27 West-Europa (zomertijd)

#11

Did you use Python 2.7? On Linux? What distribution?

#12

Python 2.7 on Windows 10.

#13

+1 This is a must have feature. I have hundreds of files and it takes weeks while InSync uploads everything. It’s very annoying.

Does anybody knows what we need to do to have this feature?

1 Like
#14

Can you tell me how to do this on a Mac? Thanks!

#15

100% this. I switched to Insync because as a developer, having exclude rules is HUGE (excluding node_modules and vendor folders).

Unfortunately, when I started syncing, I found out it’s going very slowly. 2 files at the same time? It’s gonna take ages. Seems to me like uploading multiple files is a very basic feature. Each of those 2 files are going painfully slow, and I still have almost 70.000 more.

Any word from Insync about this?

#16

@johnRivs We are aware of this issue, but our devs is working hard on improving the syncing speed. Syncing is noticeably faster after the initial sync.

Happy New Year :smile:

1 Like
#17

When will this feature be available? It takes forever to sync a large amount of files and it only uses a small part of the available bandwidth. (Syncing from Linux with Insync 1.3.6)

#18

@marte Could someone send me a syncworks.py file with 20 or 30 threads enabled so I can test if download is any faster?

#19

@Peter: we’ll add support for changing it via environment variable / command-line argument on the next release. Thanks for the feedback.

#20

i agree with the feature request.

another idea is that insync figures out by itself what the best number of parallel transfers is. call it “intelligent multi thread” and make it a major feature. for very small files it would transfer up to something like 20 files, in the middle area maybe 5 parallel connection and for large files it sticks to 2 parallel transfers.