Nov 17, 2014 Tags: Linux, Howto

Using Rsync

This Howto explains how you can use SSH and Rsync to copy files from a remote server to your local machine fast and comfortably.

Updated on Sep 21, 2016

See also

Navigate this page:

The Code

I’m showing part of a script that I usually use to copy a web installation from a remote server to my local machine. Let’s start with the code right away:

#! /bin/bash

# settings
useratdomain=user@remote.com

# with / at the end of srcdir !!!
srcdir=/home/user/public_html/

# with / at the end of destdir !!!
destdir=/home/user/projectname/public_html/

if ((1)); then true;
    dryrun=--dry-run
else false;
    dryrun=
fi

# NO slash / at the end of workdir!
workdir=get-`date '+%Y-%m-%d'`

if ((1)); then true;
    echo "`date '+%Y-%m-%d %H:%M:%S'` Started."
fi

if ((1)); then true;
    echo "`date '+%Y-%m-%d %H:%M:%S'` Prepare log files in '$workdir'."
    log_v_file=$workdir/rsync_temp_log_v_file.txt
    log_v_stdout=$workdir/rsync_temp_log_v_stdout.txt
    rm -rf $workdir
    mkdir $workdir         &> /dev/null
fi

if ((1)); then true;
    echo "`date '+%Y-%m-%d %H:%M:%S'` Get public files from remote"
    rsync \
        $dryrun \
        -rlt  -v --safe-links --delete --log-file=$log_v_file \
        --exclude=/.htaccess \
        --exclude=*NOT_VERSIONED* \
        --exclude=/typo3conf/localconf.php \
        --exclude=/typo3conf/LocalConfiguration.php \
        --exclude=/typo3temp/ \
        -e 'ssh' $useratdomain:$srcdir $destdir \
        &>$log_v_stdout
fi

if ((1)); then true;
    echo "`date '+%Y-%m-%d %H:%M:%S'` Finished."
fi

How does it work?

In this example we mirror the remote folder srcdir to the local folder destdir. Beforehand I have uploaded my public key to the remote server so it will accept the SSH connection for useratdomain, which is user@remote.com. The slash at the end of both srcdir and destdir is important as otherwise you may end up with srcdir being copyied as a folder into destdir, In that case you see something like .../public_html/public_html/.

You can turn on and off each little part of the script by either writing ((1)) or ((0)).

The Rsync Command

I prefer splitting the command into several lines. A backslash at the end tells the shell that there will be a continuation line. Add more exclude lines if you have more things to exclude. We are using these options:

-r Copy recursively, with subdirectories.
-l Copy symlinks as symlinks.
-p Preserve permissions.
-t Preserve modification times.
-v Increase verbosity.
--safe-links Ignore symlinks that point outside the source tree.
--delete

Delete extraneous files from destination dirs.

Warning: Pay attention to provide correct paths! The --delete option turns on mirroring which means that files and folders in your destination directory can be deleted.

--log-file Have rsync “log what we’re doing to the specified FILE”.
--exclude Exclude files matching PATTERN.
-e Specify the remote shell to use.

Updated on Nov 17, 2014: More options recommended:

-z Compress file data during the transfer.
--progress Show progress during transfer.
-e Isn’t necessary sometimes.

Explaining ‘true’ and ‘false’

Writing:

if ((1)); then
    dryrun=--dry-run
else
    dryrun=
fi

would do the job. But I prefer to add ‘true’ or ‘false’ after ‘then’ and ‘else’ as in:

if ((1)); then true;
    # do something
else false;
    # or do this
fi

The sole reason for doing so is that there won’t be a syntax error if there isn’t a payload in the block and all lines are commented out – as in this example. Any command would do. But ‘true’ and ‘false’ seem perfectly suited and do not disturb at all.

Rsync Examples

# recurse, perms, links, times (better use -a!)
rsync -rplt -v --safe-links --delete \
   /media/marble/LinuxData200        \
   marble@192.168.1.203:/media/marble/BackupThinkbook

Enjoy!

HTH - I hope this helps. What “tricks” are you using?

More

Compare Directories

Use rsync for a comparison between local and remote: Do a simulation (-n, –dry-run) only, as if you were archiving (-a, –archive), comparison based checksum (-c. –checksum), output a change-summary for all updates ( -i, –itemize-changes), as if you were mirroring to destination (–delete):

# compare, dry-run
rsync -ain --delete /local/path/  user@domain.de:/remote/path/

# compare by checksum, dry-run
rsync -acin --delete /local/path/  user@domain.de:/remote/path/

# compare by checksum, dry-run, with special params for ssh
rsync -acin --delete /local/path/  -e "ssh -i /home/user/.ssh/sshkey" user@domain.de:/remote/path/

#

As blueprint:

# ==========================
# compare
# --------------------------

useratdomain=me@server-domain.com

# with / at the end!
srcdir=/home/me/local/path/

# with / at the end!
destdir=/home/there/remote/path/

rsync -ain --delete "${srcdir}" "${useratdomain}:${destdir}"

# change direction:

rsync -ain --delete "$useratdomain:$destdir" "$srcdir"

#

Exclude some:

rsync -ain --delete "${srcdir}" "${useratdomain}:${destdir}" \
   --exclude typo3temp/ \
   --exclude typo3/ \
   --exclude t3lib/
#

Show lines for non identical items only:

rsync -ain --delete "${srcdir}" "${useratdomain}:${destdir}" \
   --exclude typo3temp/ \
   --exclude typo3/ \
   --exclude t3lib/ \
   | grep -v "\.\.\.\.\.\.\.\.\."
#

See also:

From the rsync manpage: -i, –itemize-changes

Search the manual:

man rsync | less -p "--itemize-changes"

Find this:

Requests a simple itemized list of the changes that are being made to each file, including attribute changes. This is exactly the same as specifying --out-format='%i %n%L'. If you repeat the option, unchanged files will also be output, but only if the receiving rsync is at least version 2.6.7 (you can use -vv with older versions of rsync, but that also turns on the output of other verbose messages).

11 letters cryptic output

The “%i” escape has a cryptic output that is 11 letters long. The general format is like the string YXcstpoguax, where Y is replaced by the type of update being done, X is replaced by the file-type, and the other letters represent attributes that may be output if they are being modified.

Type of update

The update types that replace the Y are as follows:

<
A < means that a file is being transferred to the remote host (sent).
>
A > means that a file is being transferred to the local host (received).
c
A c means that a local change/creation is occurring for the item (such as the creation of a directory or the changing of a symlink, etc.).
h
A h means that the item is a hard link to another item (requires –hard-links).
.
A . means that the item is not being updated (though it might have attributes that are being modified).
*
A * means that the rest of the itemized-output area contains a message (e.g. “deleting”).

File type

The file-types that replace the X are:

f for a file, a
d for a directory, an
L for a symlink, a
D for a device, and a
S for a special file (e.g. named sockets and fifos).

Attributes

The other letters in the string above are the actual letters that will be output if the associated attribute for the item is being updated or a ”.” for no change. Three exceptions to this are:

(1) +
a newly created item replaces each letter with a “+”,
(2) spaces
an identical item replaces the dots with spaces, and
(3) ?
an unknown attribute replaces each letter with a ”?” (this can happen when talking to an older rsync).

The attribute that is associated with each letter is as follows:

.
A . means no change.
c
A c means either that a regular file has a different checksum (requires --checksum) or that a symlink, device, or special file has a changed value. Note that if you are sending files to an rsync prior to 3.0.1, this change flag will be present only for checksum-differing regular files.
s
A s means the size of a regular file is different and will be updated by the file transfer.
t
A t means the modification time is different and is being updated to the sender’s value (requires --times). An alternate value of T means that the modification time will be set to the transfer time, which happens when a file/symlink/device is updated without --times and when a symlink is changed and the receiver can’t set its time. (Note: when using an rsync 3.0.0 client, you might see the s flag combined with t instead of the proper T flag for this time-setting failure.)
p
A p means the permissions are different and are being updated to the sender’s value (requires --perms).
o
An o means the owner is different and is being updated to the sender’s value (requires --owner and super-user privileges).
g
A g means the group is different and is being updated to the sender’s value (requires --group and the author‐ ity to set the group).
u
The u slot is reserved for future use.
a
The a means that the ACL information changed.
x
The x means that the extended attribute information changed.

One other output is possible: when deleting files, the “%i” will output the string “*deleting” for each item that is being removed (assuming that you are talking to a recent enough rsync that it logs deletions instead of out‐ putting them as a verbose message).

Comments

comments powered by Disqus

Previous topic

ReST: Builders and Writers

Next topic

Basic Installation of Sphinx and Ablog

Tags

Archives

Languages

Recent Posts

This Page