rsync over SSH: backup your data securely

rsync-over-ssh

Using rsync over ssh is a secure method to backup your system. SSH encrypts your data over the internet and secures the transmission from hackers and other threats. The good news is that rsync uses the SSH protocol by default. In this tutorial I will show you how to use rsync over SSH and how to conifgure SSH keys for passwordless backups.

rsync over SSH requirements

  • SSH access to server
  • rsync client installed locally

Installing rsync

In most cases rsync will already be installed on your system. If it’s not, it can be easily installed.

Debian/Ubuntu

sudo apt-get install rsync

CentOS

sudo yum install rsync

Testing the connection

Test the connection to your server by connecting with SSH:

ssh user@remotehost.com

If all goes well you should be greeted by a password prompt:

Testing rsync

If you can connect with SSH you can connect with rsync over ssh. Test rsync by initiating a dry run backup of your home directory:

rsync -nav user@remotehost:~/ /path/to/local/backup/folder/

With any luck you will see your files and folders fly by the screen. Note that we did not have to use the -e switch. Since rsync uses SSH by default this is not necessary unless you need to specifiy additional connection parameters (non-standard port or SSH key locations.)

Setting up SSH keys

We can setup SSH keys so rsync doesn’t need a password to connect. This is useful for automating your backups with chron while staying completely secure. SSH keys are generated on your local computer and then copied to the remote host. Generate a key pair with the following command:

ssh-keygen -t rsa

Save the keys in the default location or specify another directory:

Enter file in which to save the key (/home/demo/.ssh/id_rsa):


Leave the passphrase blank by hitting enter through the following prompt:

Enter passphrase (empty for no passphrase):

Here is what you should see:

Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
4a:dd:0a:c6:35:4e:3f:ed:27:38:8c:74:44:4d:93:67 user@a
The key's randomart image is:
+--[ RSA 2048]----+
|          .oo.   |
|         .  o.E  |
|        + .  o   |
|     . = = .     |
|      = S = .    |
|     o + = +     |
|      . o + o .  |
|           . o   |
|                 |
+-----------------+

While it would be more secure to enter a passphrase, rsync can’t be automated in this way. However, this method is still secure because a hacker would need to obtain your private key to gain access to the server.

Next we copy the public key to the server. Security tip: Never grant your keys to the root account. Always copy your keys to a standard user account. This way if a hacker jacks your private key he would only have limited access to your box.

ssh-copy-id user@remotehost.com

If prompted, type yes to connect and then enter your password to complete the transfer:

The authenticity of host '12.34.56.78 (12.34.56.78)' can't be established.
RSA key fingerprint is b1:2d:33:67:ce:35:4d:5f:f3:a8:cd:c0:c4:48:86:12.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '12.34.56.78' (RSA) to the list of known hosts.
user@12.34.56.78's password: 
Now try logging into the machine, with "ssh 'user@12.34.56.78'", and check in:

  ~/.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

Boom goes the dynamite. You should now be able to run rsync over ssh without a password.

Resources

http://linux.die.net/man/1/rsync

 

8 Practical Robocopy Examples For Your Next Backup

robocopy-examples




Robocopy is a powerful backup utility that ships with most versions of Microsoft Windows. I’ve put this post together to showcase a few powerful Robocopy examples. You can use these examples for your next backup or as a general guide to using Robocopy.

Default Behavior

Robocopy does a raw file copy at a single directory level when used without any switches. By default, it won’t copy any directories unless you specify otherwise. In this example, I am using Robocopy to backup all the files in my Documents folder to a flash drive.

C:\>Robocopy C:\Documents\ E:\

Parameters / SwitchesDescription
C:\DocumentsSource directory. This is where we will be copying files FROM.
E:\Destination Directory. This is where we will copy files TO.

Mirror a complete directory stucture

Robocopy can backup your files in a number of ways. You can change how Robocopy operates using a combination of switches. In this example we tell Robocopy to create an exact mirror copy of the source directory. /MIR will copy both files and folders and mirror the source directory structure.

C:\>Robocopy C:\Documents E:\ /MIR

Parameters / SwitchesDescription
C:\DocumentsSource directory. This is where we will be copying files FROM.
E:\Destination Directory. This is where we will copy files TO.
/MIRMake a mirror copy of the directory structure on the destination drive. Delete any files that aren't in the source.

Move files (delete from source)

Robocopy can move files from a source directory to a destination. This is useful when you want to clear up space on your hard drive. This technically isn’t a backup since Robocopy will remove the files you copy. **Note** this example moves files only. If you want to move both files and folders you must use the /MOVE switch.

C:\>Robocopy C:\Documents E:\ /MOV

Parameters / SwitchesDescription
C:\DocumentsSource directory. This is where we will be copying files FROM.
E:\Destination Directory. This is where we will copy files TO.
/MOVMove source files to destination. Removes files from source.

Mirror directory structure but keep destination data

When using /MIR, Robocopy deletes files in the destination directory that aren’t in the source. You can specify for Robocopy to keep destination files with /XX switch. Useful when you’re doing a simple data dump to a backup drive that has other files you want to keep.

C:\>Robocopy C:\Documents E:\ /MIR /XX

Parameters / SwitchesDescription
C:\DocumentsSource directory. This is where we will be copying files FROM.
E:\Destination Directory. This is where we will copy files TO.
/MIRMake a mirror copy of the directory structure on the destination drive. Delete any files that aren't in the source.
/XXKeep destination files instead of deleting them.

Copy security permissions

Sometimes it is necessary to copy over the security attributes of files and folders. This is useful on a server where users have varying access to the file system. You want to make sure these security attributes stay intact when you copy data back to the server. Here I am mirroring my directory structure and copying over security permissions.

C:\>Robocopy C:\Documents E:\ /MIR /SEC

Parameters / SwitchesDescription
C:\DocumentsSource directory. This is where we will be copying files FROM.
E:\Destination Directory. This is where we will copy files TO.
/MIRMake a mirror copy of the directory structure on the destination drive. Delete any files that aren't in the source.
/SECCopy security permissions.

Specify retries and wait time for locked files

Robocopy has built in programming to handle locked files. If a file is locked Robocopy can retry the copy as many times as you want. Here we specify a 2 retry count with /R switch. Simultaneously, we are also telling Robocopy to wait 10 seconds before attempting the next retry with the /W switch.

C:\>Robocopy C:\Documents E:\ /R:2 /W:10 

Parameters / SwitchesDescription
C:\DocumentsSource directory. This is where we will be copying files FROM.
E:\Destination Directory. This is where we will copy files TO.
/R:2Retry 2 times before moving on to the next file.
/W:10Wait 10 seconds before going to the next retry cycle.

Mirror entire C:\ drive but exclude hidden and system files

You might be thinking about grabbing everything on your C:\ drive. Not a bad idea if you really want to snag absolutely everything. Do you really need all the system and hidden files though? You can tell Robocopy to exclude these specific files with the /XA switch. Here we tell Robocopy to exclude files with the system and hidden attributes set.

C:\>Robocopy C:\ E:\ /MIR /XA:SH

Parameters / SwitchesDescription
C:\Source directory. This is where we will be copying files FROM.
E:\Destination Directory. This is where we will copy files TO.
/MIR Make a mirror copy of the directory structure on the destination drive. Delete any files that aren't in the source.
/XA:SHExclude system and hidden files when copying.

Stringing it all together

The true power of Robocopy reveals itself when you use many of these switches together. This allows for a fine tuned backup that meets your requirements. In this last Robocopy example, I mirror my entire C:\ drive, exclude the system and hidden folders, Retry twice before moving on- waiting 10 seconds after each retry, and log the results to file on my hard drive.

C:\>Robocopy C:\ E:\ /MIR /XA:SH /R:2 /W:10 /LOG:C:\log.txt

Parameters / SwitchesDescription
C:\DocumentsSource directory. This is where we will be copying files FROM.
E:\Destination Directory. This is where we will copy files TO.
/MIRMake a mirror copy of the directory structure on the destination drive. Delete any files that aren't in the source.
/XA:SHExclude system and hidden files when copying.
/R:2Retry 2 times before moving on to the next file.
/W:10Wait 10 seconds before going to the next retry cycle.
/LOG:C:\log.txtLog events to a text file. Useful for troubleshooting if something goes wrong.

Resources

Did you enjoy these Robocopy examples? If you’d like to experiment with other switches you can reference this Microsoft article for a complete list. Alternatively, you can run the following command at your command prompt:

C:\>Robocopy /?

 

How To Run Robocopy in Backup Mode

robocopy-backup-mode

Robocopy is a powerful backup utility with plenty of switches to confuse even the savviest tech. Deep within the Robocopy programming is a function called Robocopy backup mode. This special mode is executed with the /B switch. That’s fine and dandy – but what the heck happens when you run Robocopy this way?

When Robocopy is executed in backup mode the software bypasses file permissions that would otherwise prevent a successful backup. This is useful for companies who have designated backup specialists that handle backups. Backup mode allows standard users to initiate backups while limiting their access to the files.

You must be an Administrator or a member of the backup operators group to run Robocopy in this way.

Example:

C:\>Robocopy C:\ E:\ /B

Reasons to use backup mode

You generally would not use backup mode unless you were a user with limited access and had the proper backup rights set by your Administrator. If you’re the Administrator of your machine the /B switch is not needed. Instead, refer to the other Robocopy command line switches to properly backup your machine. While backup mode sounds like the perfect mode to backup your system, it is only a useful switch for limited users who have been granted backup rights by an Administrator.

Common errors

When using the /B switch you may receive the following error:

You do not have the Backup and Restore Files user rights. You need these to perform Backup copies (/B or /ZB).

This error is easily solved by using an elevated command prompt. Simply right click the command console in Windows explorer and select run as administrator.

If you don’t have administrative access you will need to be added to the backup operators group to make backups of files that you do not own. Ask your Administrator to grant you this access.

Resources

Microsoft Technet  – Robocopy