Bandwidth of rsync much lower than connection speed

7 posts / 0 new
Last post
Anonymous
Bandwidth of rsync much lower than connection speed

I am trying to use RSync to speed up a nightly get of around 3 gigs of data from the east coast to the west over a VPN. I have around 10Mbits of bandwidth between the two locations at a latency of around 110ms to the server. The problem I have is that I can't get the transfer rate to exceed 70kByte/s. If I do a raw Windows copy, I can get 220kB/s, so this seems really low. I have tried turning on compression, running a TCP/IP optimizer on the server (the client is Vista, which claims not to need this), switching to DeltaCopy, and removing the network share (it was copying from a network drive at the server location, now it's direct local drive). None of these had any impact on transfer speed.

I am seeing reduced overall bandwidth usage (rsync cuts 50% of the data transfer), but it's 3x slower, so it's not a net gain. Both machines are very fast (quad core, 10k drives). Both cpus are idling at <1% used by rsync. It seems like it has to be a network issue, but I can't find anything in Rsync to tweak that.

Thoughts? Did I miss something in trying to get this transfer to go faster?

itefix
Offline
Last seen: 1 hour 39 min ago
Joined: 01.05.2008 - 21:33
How does your rsync command

How does your rsync command look like ?

Jurney
Command line I'm using

rsync --compress-level=9 --recursive --times --archive --human-readable --verbose --progress --delete rsync://SERVERNAME/FOLDER/ /cygdrive/D/LocalFolder/LocalFolder

Jurney
More data

Another interesting tidbit... the rate of transfer isn't really affected by other transfers running on the same connection. Specifically, if I run rsync by itself, then I try it again while doing a file copy from another server, it gets the same transfer rate (~70kB/s). Weird.

Jurney
Weirder and weirder

Just on a hunch, I tried syncing 2 separate files at the same time from the 1 rsync server. They both get ~70kB/s. wtfbbq.  Either rsync has a built in transfer cap of some kind, or I am getting traffic shaped somewhere.

itefix
Offline
Last seen: 1 hour 39 min ago
Joined: 01.05.2008 - 21:33
AFAIK, there is no built-in

AFAIK, there is no built-in performance cap in rsync. I have now tested it between two windows machines (client to daemon) via a 2 Mbps link and I can easily achieve 150 KB or more.

Jurney
Maybe its cygwin?

It may be cygwin and its interaction with TCP/IP... I can't reproduce this over a lan, just my highish latency VPN link, so I can't say much for certain, but the "80kBps" cap in these articles is suspiciously close to what I see if you add in some protocol overhead:

http://www.mail-archive.com/cygwin-patches@cygwin.com/msg03644.html

http://support.microsoft.com/kb/329781

I'm gonna try doing the sync to a public rsync server to see if it's the VPN or just the link speed.