Difference between revisions of "Rsync"

Latest revision as of 14:56, 31 August 2023

rsync (1996) command is used to copy and synchronization files and folders. Rsync is a single thread C application that includes a progress indicator.

Man: http://man7.org/linux/man-pages/man1/rsync.1.html

Configuration files[edit]

/etc/rsyncd.conf which includes don't compress options for some file extensions

Options[edit]

-W, --whole-file copy files whole (without delta-xfer algorithm)

-z, --compress compresses the file data as it is sent to the destination machine using zlib compression library

--temp-dir=DIR, -T create temporary files in directory DIR, by default temp file is created as a hidden file in the target directory and named .<FILE_NAME>.<RANDOM_STRING>

--partial

--delete delete files in the target not present in source

      --delete                 delete extraneous files from dest dirs
      --delete-before          receiver deletes before xfer, not during
      --delete-during          receiver deletes during the transfer
      --delete-delay           find deletions during, delete after
      --delete-after           receiver deletes after transfer, not during
      --delete-excluded        also delete excluded files from dest dirs
      --delete-missing-args    delete missing source args from destination

Basic commands[edit]

rsync -Pr ORIGIN_FOLDER_OR_FILE DESTINATION_FOLDER

-P, --partial --progress (--partial keep partially transferred files)

-r --recursive

rsync -P ORIGIN DESTINATION

-P, --partial --progress (--partial keep partially transferred files)

Copy files from remote server to local machine: rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage^[1] (explainshell^[2])

-c, --checksum option can slow transfers specially for large files

-h, --human-readable

-a, --archive. Equivalent to -rlptgoD

-v, --verbose

-z, --compress

-P, --partial --progress. --partial keep partially transferred files

--stats, print statistics

-a, --archive. Equivalent to -rlptgoD

-r --recursive recurse into directoriees

-l --links copy symlinks as symlinks

-p --perms preserve permissions

-t --times preserve modification times

-g --group preserve group

-o --owner preserve owner (super-user only)

-D same as --devices --specials

rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage
receiving incremental file list
.../...

Remove -c option to avoid calculating checksums which can slow transmissions.

rsync -havzP --stats [email protected]:/path/to/copy /path/to/local/storage
receiving incremental file list
.../...

.../...
sending incremental file list

rsync -Pav --ignore-existing src dst

Learn difference between: --ignore-existing and --update

rsync -av --ignore-existing src dst

Compression[edit]

rsync support compression, but you can also compress your data with other external compression tools before using rsync. Such as bzip, gzip or xz

-z, --compress compresses the file data as it is sent to the destination machine using zlib compression library

Compressing file before rsync using xz^[3]

xz -v -T0 file_to_compress

-v provides progress bar

-T0 use all available Threads, multi-threaded compression supported since 2014, version 5.2.0. ^[4]

Activities[edit]

Create a remote copy of your files for backup proposes using rsync
Read StackOverflow questions about rsync: https://stackoverflow.com/questions/tagged/rsync?tab=Votes
Understand the differences between cp, scp and rsync: https://stackoverflow.com/questions/20244585/how-does-scp-differ-from-rsync
Learn Effect of Trailing Slash / in rsync: https://www.alibabacloud.com/blog/speeding-up-network-file-transfers-with-rsync_594337
Learn use of --inplace (implies also --partial) for transferring large files and its important implications: http://man7.org/linux/man-pages/man1/rsync.1.html, also evaluate -c, --checksum impact on transfering large files
Learn about --sparse option and conflicts with --inplace

Performance

Understand how single thread rsync design can affect your transfer speeds.^[5]
Learn about -W, --whole-file copy files whole (without delta-xfer algorithm), implications and usage cases such as not making a diff copy.

@@ Line 1: / Line 1: @@
-[[wikipedia:rsync|rsync]]<ref>http://man7.org/linux/man-pages/man1/rsync.1.html</ref> is an advanced copy and synchronization tool including progress indicator.
+{{lowercase}}
+<code>[[wikipedia:rsync|rsync]]</code> ([[1996]]) command is used to copy and synchronization files and folders. Rsync is a [[single thread]] C application that includes a [[progress]] indicator.
+* Man: http://man7.org/linux/man-pages/man1/rsync.1.html
 == Configuration files ==
@@ Line 5: / Line 8: @@
 == Options ==
-:<code>-W, --whole-file</code>
+:<code>-W, --whole-file</code> copy files whole (without [[delta-xfer]] algorithm)
-:<code>-z, --compress</code> compresses the file data as it is sent to the destination machine using [[zlib]] compression library
+:<code>-z, --[[compress]]</code> compresses the file data as it is sent to the destination machine using [[zlib]] compression library
+:<code>--temp-dir=DIR, -T</code> create temporary files in directory DIR, by default temp file is created as a [[hidden file]] in the target directory and named <code>.<FILE_NAME>.<RANDOM_STRING></code>
+:<code>--partial</code>
+:<code>[[--delete]]</code> delete files in the target not present in source
+       --delete                 delete extraneous files from dest dirs
+       --delete-before          receiver deletes before xfer, not during
+       --delete-during          receiver deletes during the transfer
+       --delete-delay           find deletions during, delete after
+       --delete-after           receiver deletes after transfer, not during
+       --delete-excluded        also delete excluded files from dest dirs
+       --delete-missing-args    delete missing source args from destination
 == Basic commands ==
+* <code>rsync -Pr ORIGIN_FOLDER_OR_FILE DESTINATION_FOLDER</code>
+: <code>-P, --partial [[--progress]] (--partial</code> keep partially transferred files)
+: <code>-r --recursive</code>
+* <code>rsync -P ORIGIN DESTINATION</code>
+: <code>-P, --partial --progress (--partial</code> keep partially transferred files)
 * Copy files from remote server to local machine: <code>rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage</code><ref>https://stackoverflow.com/questions/9090817/copying-files-using-rsync-from-remote-server-to-local-machine</ref> (explainshell<ref>https://explainshell.com/explain?cmd=rsync+-chavzP+--stats+user%40remote.host%3A%2Fpath%2Fto%2Fcopy+%2Fpath%2Fto%2Flocal%2Fstorage</ref>)
 : <code>-c, --checksum</code> option can slow transfers specially for large files
@@ Line 18: / Line 37: @@
 : <code>--stats</code>, print statistics
-:-a, --archive. Equivalent to <code>-rlptgoD</code>
+:<code>-a, --archive</code>. Equivalent to <code>-rlptgoD</code>
-::-r --recursive             recurse into directoriees
+::<code>-r --recursive</code>             recurse into directoriees
-::-l --links                 copy symlinks as symlinks
+::<code>-l --links</code>                 copy symlinks as symlinks
-::-p --perms                 preserve permissions
+::<code>-p --perms</code>                 preserve permissions
-::-t --times                 preserve modification times
+::<code>-t --times</code>                 preserve modification times
-::-g --group                 preserve group
+::<code>-g --group</code>                 preserve group
-::-o --owner                 preserve owner (super-user only)
+::<code>-o --owner</code>                 preserve owner (super-user only)
-::-D same as --devices --specials
+::<code>-D</code> same as <code>--devices --specials</code>
+<pre>rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage
+receiving incremental file list
+.../...
+</pre>
+Remove <code>-c</code> option to avoid calculating checksums which can slow transmissions.
-<pre>rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage
+<pre>rsync -havzP --stats [email protected]:/path/to/copy /path/to/local/storage
 receiving incremental file list
 .../...
 </pre>
+ .../...
+ sending incremental file list
-* <code>rsync -a -v --ignore-existing src dst</code>
+* <code>rsync -Pav --ignore-existing src dst</code>
+:Learn difference between: <code>--ignore-existing</code> and <code>--update</code>
+* <code>rsync -av --ignore-existing src dst</code>
 == Compression ==
 rsync support [[compression]], but you can also compress your data with other external compression tools before using rsync. Such as [[bzip]], [[gzip]] or <code>[[xz]]</code>
+:<code>-z, --[[compress]]</code> compresses the file data as it is sent to the destination machine using [[zlib]] compression library
-Compressing using <code>[[w:XZ Utils|xz]]</code><ref>https://linux.die.net/man/1/xz</ref>
+Compressing file before rsync using <code>[[xz]]</code><ref>https://linux.die.net/man/1/xz</ref>
-* <code> xz -v -T0 file_to_compress</code>
+* <code>[[xz]] -v -T0 file_to_compress</code>
-::-v provides progress bar
+::<code>-v</code> provides progress bar
-::-T0 use all available Threads, multi-threaded compression supported since 2014, version 5.2.0. <ref>https://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD</ref>
+::<code>-T0</code> use all available Threads, multi-threaded compression supported since 2014, version 5.2.0. <ref>https://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD</ref>
 == Activities ==
 # Create a remote copy of your files for backup proposes using <code>rsync</code>
 # Read StackOverflow questions about rsync: https://stackoverflow.com/questions/tagged/rsync?tab=Votes
-# Understand the differences between [[linux/Basic commands/cp|cp]], [[scp]] and [[linux/Basic commands/rsync|rsync]]: https://stackoverflow.com/questions/20244585/how-does-scp-differ-from-rsync
+# Understand the differences between <code>[[cp]]</code>, <code>[[scp]]</code> and <code>[[rsync]]</code>: https://stackoverflow.com/questions/20244585/how-does-scp-differ-from-rsync
 # Learn Effect of Trailing Slash / in rsync: https://www.alibabacloud.com/blog/speeding-up-network-file-transfers-with-rsync_594337
-# Learn use of <code>--inplace</code> for transferring large files and its important implications: http://man7.org/linux/man-pages/man1/rsync.1.html, also evaluate <code>-c, --checksum</code> impact on transfering large files
+# Learn use of <code>--inplace</code> (implies also <code>--partial</code>) for transferring large files and its important implications: http://man7.org/linux/man-pages/man1/rsync.1.html, also evaluate <code>-c, --checksum</code> impact on transfering large files
+# Learn about <code>--sparse</code> option and conflicts with  <code>--inplace</code>
+Performance
+# Understand how single thread <code>rsync</code> design can affect your transfer speeds.<ref>https://stackoverflow.com/questions/24058544/speed-up-rsync-with-simultaneous-concurrent-file-transfers</ref>
 # Learn about <code>-W, --whole-file</code> copy files whole (without delta-xfer algorithm), implications and usage cases such as not making a diff copy.
-# Learn about <code>--sparse</code> option and conflicts with  <code>--inplace</code>
+== Related terms ==
+* <code>[[aws s3 rsync]]</code>
+* <code>[[gzip]] --[[rsyncable]]</code>
+* <code>[[duplicity]]</code>
+* [[fork]]
+* <code>rsyncy</code> status/[[progress]] bar for [[rsync]]
+* [[Andrew Tridgell]]
+* [[AWS S3 replication]]
 == See also ==
-* <code>[[scp]]</code>, <code>cp</code>, <code>[[dd (Unix)|dd]]</code>, <code>[[mv]]</code>
+* {{rsync}}
+* {{cp}}
+* <code>[[dd]]</code>, <code>[[mv]]</code>
 [[Category:Linux]]

Difference between revisions of "Rsync"

Latest revision as of 14:56, 31 August 2023

Contents

Configuration files[edit]

Options[edit]

Basic commands[edit]

Compression[edit]

Activities[edit]

Related terms[edit]

See also[edit]

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

Tools