Rsync

rsync (1996) command is used to copy and synchronization files and folders. Rsync is a single thread C application that includes a progress indicator.

• Man: http://man7.org/linux/man-pages/man1/rsync.1.html

Configuration files

• /etc/rsyncd.conf which includes don't compress options for some file extensions

Options

-W, --whole-file copy files whole (without delta-xfer algorithm)

-z, --compress compresses the file data as it is sent to the destination machine using zlib compression library

--temp-dir=DIR, -T create temporary files in directory DIR, by default temp file is created as a hidden file in the target directory and named .<FILE_NAME>.<RANDOM_STRING>

--partial

--delete delete files in the target not present in source

      --delete                 delete extraneous files from dest dirs
      --delete-before          receiver deletes before xfer, not during
      --delete-during          receiver deletes during the transfer
      --delete-delay           find deletions during, delete after
      --delete-after           receiver deletes after transfer, not during
      --delete-excluded        also delete excluded files from dest dirs
      --delete-missing-args    delete missing source args from destination

Basic commands

• rsync -Pr ORIGIN_FOLDER_OR_FILE DESTINATION_FOLDER

-P, --partial --progress (--partial keep partially transferred files)

-r --recursive

• rsync -P ORIGIN DESTINATION

-P, --partial --progress (--partial keep partially transferred files)

• Copy files from remote server to local machine: rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage (explainshell)

-c, --checksum option can slow transfers specially for large files

-h, --human-readable

-a, --archive. Equivalent to -rlptgoD

-v, --verbose

-z, --compress

-P, --partial --progress. --partial keep partially transferred files

--stats, print statistics

-a, --archive. Equivalent to -rlptgoD

-r --recursive recurse into directoriees

-l --links copy symlinks as symlinks

-p --perms preserve permissions

-t --times preserve modification times

-g --group preserve group

-o --owner preserve owner (super-user only)

-D same as --devices --specials

rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage
receiving incremental file list
.../...

Remove -c option to avoid calculating checksums which can slow transmissions.

rsync -havzP --stats [email protected]:/path/to/copy /path/to/local/storage
receiving incremental file list
.../...

.../...
sending incremental file list

• rsync -Pav --ignore-existing src dst

Learn difference between: --ignore-existing and --update

• rsync -av --ignore-existing src dst

Compression

rsync support compression, but you can also compress your data with other external compression tools before using rsync. Such as bzip, gzip or xz

-z, --compress compresses the file data as it is sent to the destination machine using zlib compression library

Compressing file before rsync using xz

• xz -v -T0 file_to_compress

-v provides progress bar

-T0 use all available Threads, multi-threaded compression supported since 2014, version 5.2.0.

Activities

1. Create a remote copy of your files for backup proposes using rsync
2. Read StackOverflow questions about rsync: https://stackoverflow.com/questions/tagged/rsync?tab=Votes
3. Understand the differences between cp, scp and rsync: https://stackoverflow.com/questions/20244585/how-does-scp-differ-from-rsync
4. Learn Effect of Trailing Slash / in rsync: https://www.alibabacloud.com/blog/speeding-up-network-file-transfers-with-rsync_594337
5. Learn use of --inplace (implies also --partial) for transferring large files and its important implications: http://man7.org/linux/man-pages/man1/rsync.1.html, also evaluate -c, --checksum impact on transfering large files
6. Learn about --sparse option and conflicts with --inplace

Performance

1. Understand how single thread rsync design can affect your transfer speeds.
2. Learn about -W, --whole-file copy files whole (without delta-xfer algorithm), implications and usage cases such as not making a diff copy.

Related terms

• aws s3 rsync
• gzip --rsyncable
• duplicity
• fork
• rsyncy status/progress bar for rsync
• Andrew Tridgell
• AWS S3 replication

See also

• rsync, scp, sftp, rsnapshot, rclone, rdiff
• cp, dd, sftp, scp, rsync, casync, mv, fio, ln, docker cp, kubectl cp, minikube cp, multipass transfer, Copy-Item, Xcopy
• dd, mv

Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy.

Source: https://en.wikiversity.org/wiki/Linux/Basic_commands/rsync