Facing migration problems where Rsync hangs on large files?
Quite often the Rsync operations go into a stuck state when copying large files from one server to another.
Mostly, Rsync hangs on large files due to the insufficient RAM size of the server.
At Bobcares we often get requests to fix Rsync errors, as a part of our Server Management Services.
Today, we’ll see the Rsync problem in detail and see how our Dedicated Engineers copy files efficiently.
What is Rsync?
Rsync is a great tool for replicating a single file or directories to remote file systems. This makes it a popular backup option.
Usually, Rsync is faster than other methods such as SCP. It uses an algorithm that minimizes the amount of data copied by only moving the portions of files that have changed.
Do you think why the Rsync is better than other commands like SCP?
Speed: Rsync replicates the entire content between the directories of destination and source.
Privileges: Rsync does not require the root privilege or sudo privilege. It comes handy for non-root users.
Security: It has a separate protocol, usually SSH, that must be specified for the transfer. Rsync allows encryption of data using ssh protocol during transfer.
There are basically two ways in which Rsync can copy/sync data:
1. Copying/syncing to/from another host over any remote shell like ssh, rsh.
2. Copying/Syncing through Rsync daemon using TCP.
Rsync hangs on large files- Reasons and fixes
Nowadays, we see many customers facing errors while transferring big files using the Rsync command.
Let’s see the main causes of this error and how our Support Engineers fix it.
1. Insufficient RAM space
Recently, one of our customers approached us with an error where Rsync hangs on large files.
Here, we checked and found that his server was running with insufficient RAM space. This was creating problems with Rsync too.
Usually, one of the important reasons for this error is insufficient RAM space on the server. Rsync’s memory usage is directly proportional to the size and number of files. In other words, large directories/files take a large amount of RAM.
Therefore, maintaining an adequate amount of free memory on the server comes as a prerequisite for Rsync operations.
So we always check the RAM space on the server using the free command and check the server space effectively.
We also analyze the results of the following commands and increase the memory on the server.
1. cat /proc/meminfo
2. vmstat -s
3. top -c
2. Wrong options
Sometimes the Rsync command hangs on large files due to the usage of wrong options with Rsync too. This becomes applicable in scenarios where we need to transfer large files with a slow network connection.
So we suggest to our customers to try it with the -z option ‘compress file data during the transfer‘. Also, our Support Engineers make sure to use the “–partial” option, which simply restarts an interrupted download/upload file. Further, we add the –timeout=0 option which means Rsync will not timeout.
Also, we set a big value for “–block-size” when there is a need to transfer the large files. A sample Rsync command will be:
To Copy a Directory from Local Server to a Remote Server
server## rsync --timeout=600 -r -zp --block-size=1048576 "filename/file path" root@remoteserverIP:/home/That fixes the error and copies the file effectively.
3. SSH options
Similarly, bad SSH configuration also affects the Rsync operation. We make sure to alter SSH settings in /etc/ssh/sshd_config
A sample set of entries looks like:
KeepAlive yes
ServerAliveInterval 30
ServerAliveCountMax 6This helps to avoid timeout errors when transferring large files.
4. The version of the Rsync
Last and not the least version of Rsync definitely affects a file transfer.
The older Rsync version causes incompatibility issues with the switches or protocol while transferring larger files. So we suggest our customers use the latest version of the Rsync.
We always use the command to check the current version of the rsync before any data transfer:
rsync --version[Stuck on large file transfer using Rsync? Our Experts are available 24×7 to help you.]
Conclusion
In short, the reasons where Rsync hangs on large files are insufficient RAM space on the server, bad connection, wrong SSH settings and so on. Today, we discussed the fixes and saw how our Support Team proactively avoid the rsync errors on large file transfer.
I’ve been using Linux since the mid 90s and learnt something about rsync. Never had any big files failing issues until now.
Cheers