Connection Reset Error

6 posts / 0 new
Last post
itefix
Offline
Last seen: 1 week 6 days ago
Joined: 01.05.2008 - 21:33
Connection Reset Error

Hi Everyone,

 

I have been fighting with this issue for many, many months now and I still have not found the fix.  I have this problem from multiple sources going to multiple destinations.  I setup my script to try multiple times if the writefd_unbuffered error occurs.  I recently changed the script to try the different cwrsync versions (both rsync.exe / cygwin1.dll files) after a failure but all fail.  Source drive P is a shadow copy volume on Win2K3 and destination server is Win2K3 running cwrsync 1.2.8 in daemon mode.  I’ve tried cwrsync 2.0.6, 2.0.8 and 2.0.9 before and all had the same issues.  I’ve found the cwrsync 1.2.8 files to be the least problematic.

I do have several (4-6) rsyncs at the same time. Should I be running separate services?  How many connections can rsync handle?

 

++++++++++++++++++++++++   SERVER LOGS   +++++++++++++++++++++++++++

 

2006/10/11 21:52:08 rsync: read error: Connection reset by peer (104)
2006/10/11 21:52:08 rsync: writefd_unbuffered failed to write 54 bytes: phase "unknown" : Connection reset by peer (104)

2006/10/11 21:52:08 rsync error: error in rsync protocol data stream (code 12) at io.c(909)

 

++++++++++++++++++++++++   CLIENT LOGS   +++++++++++++++++++++++++++

 

STARTING BACKUP OPTION 4 FOLDER - /Exchange Data - 22:07:09 ###

"rsync.exe -r -t -l -v --modify-window=2 --exclude="" --exclude-from=Files_To_Exclude.lti --stats --delete "/cygdrive/p/Exchange Data/" "rsync://Backup@test.domain.com:55555/Backup1/d/Exchange Data/" "

Backup of "p:/Exchange Data" has restarted due to errors at 22:07:11.

Attempting backup with cwrsync version 2.0.0. on attempt 1

building file list ... done
./
E00.chk
E00.log
E0002660.log
E0002661.log
E00tmp.log
Secondary Storage Group/E01.chk
priv1.edb
rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown" : Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(584)
 

Backup of "p:/Exchange Data" has restarted due to errors at 22:51:02.

 

Attempting backup with cwrsync version 2.0.6. on attempt 2

building file list ... done
deleting .priv1.edb.006320
./
Secondary Storage Group/
priv1.edb
rsync: writefd_unbuffered failed to write 4 bytes : Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(614)
 

Backup of "p:/Exchange Data" has restarted due to errors at 23:29:00.

 

Attempting backup with cwrsync version 2.0.8. on attempt 3

building file list ... done
deleting .priv1.edb.000836
./
priv1.edb
rsync: writefd_unbuffered failed to write 4 bytes : Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(610)
 

Backup of "p:/Exchange Data" has restarted due to errors at 00:06:49.

Attempting backup with cwrsync version 2.0.9. on attempt 4

building file list ... done
deleting .priv1.edb.006368
./
priv1.edb
rsync: writefd_unbuffered failed to write 4 bytes : Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(610)

 

Backup of "p:/Exchange Data" has restarted due to errors at 00:46:47.

+++++++++++++++++     END OF CLIENT LOGS     ++++++++++++++++++++++++

 

itefix
Offline
Last seen: 1 week 6 days ago
Joined: 01.05.2008 - 21:33
Re: Connection Reset Error

Some thoughts: How big is your private storage priv1.edb ? Did you check the integrity of it (isinteg?) ? As of SP1, Exchange 2003 can tolerate one-bit errors. I am not sure if rsync can. The problem may also be related to the shadowing. Do you experience the same problem if you stop Exchange services and directly rsync priv1.edb instead of shadow copy ?

itefix
Offline
Last seen: 1 week 6 days ago
Joined: 01.05.2008 - 21:33
Re: Connection Reset Error

Hi Tev,

 I've had this issue before I ever started using shadow copy. I also have this same issue on different files including exchange files, access DB files, PST files and rarely I see on regular files. 

 Derek

itefix
Offline
Last seen: 1 week 6 days ago
Joined: 01.05.2008 - 21:33
Re: Connection Reset Error

I have wrestling with the "rsync: read error: Connection reset by peer (104)" problem with cwRsync 2.0.8 and 2.0.9. Files are being transferred from a file server (around 120GB of data across approx 100,000 files) that sits behind a NAT router isolated from the main LAN which has the destination Win2k file server. The source Win2k3 server is running the Rsync daemon as a service and the NAT router has been configured for port forwarding the Rsync connection from the client. This setup is using an Rsync client command of the form:

 rsync <RemoteHost>::<RsyncShare>\<SubPath> <DestPath>

After much experimentation, it seems to be due to the router dropping the TCP connection as part of its inactive TCP connection cleanup because the channel is idle for too long while Rsync does its work. A similar problem was encountered in the FTP control channel, as discussed in http://www.ncftp.com/ncftpd/doc/misc/ftp_and_firewalls.html. Another similar problem was encountered with Microsoft's Remote Desktop Protocol, as discussed in newsgroup items "RDC Client get black screen lockup connecting to terminal server" and "Client Session Timeout Settings".

I have been able to get Rsync to run through its procedure numerous times since I added "-timeout 600" (ie. 10 minutes) to the Rsync client command line. There is an entry in the Rsync 2.6.4 release notes that mentions a keep-alive packet that becomes active when a timeout is specified. Before applying the timeout parameter, the procedure had only run to completion once when the initial file transfer occurred but had never completed a subsequent invocation. Since applying the parameter, it has run to completion every time. Only a few dozen of the 100,000 files are transferred or deleted on a typical run.

I have also experimented with a remote SSH shell connection using an Rsync command of the form:

 rsync -azv -e "ssh -i <PrivateKeyFile> -l <SSHUser>" <RsyncUser>@<RemoteHost>:<RemotePath> <DestPath>

It exhibits the same problem but the timeout parameter does not overcome the problem. It seems to fail a similar distance into a run, so it's probably due to the same cause but the details of the keep-alive mechanism might be different in this case.

If the cause of this type of problem is inactive TCP connection cleanup in routers and firewalls (and possibly switches) then perhaps a keep-alive mechanism would be appropriate regardless of Rsync operating mode and timeout parameters.

itefix
Offline
Last seen: 1 week 6 days ago
Joined: 01.05.2008 - 21:33
Re: Connection Reset Error

I have also had some timeout issues. Using ssh option ServerAliveInterval solved my problems:

ssh.exe .... -o ServerAliveInterval=60

itefix
Offline
Last seen: 1 week 6 days ago
Joined: 01.05.2008 - 21:33
Re: Connection Reset Error

Tev,

Your suggestion has resolved the "Connection Reset Error" problem when using cwRsync through SSH on a large backup operation.

Thanks

Murray

Topic locked

Release announcements