Rclone

Rclone

Illustrative rclone image.

Rclone is a powerful, open-source, command-line program used for managing files on cloud storage and other remote storage systems. Often referred to as "The Swiss army knife of cloud storage," it provides features similar to Unix commands like rsync, cp, mv, mount, ls, and cat, but adapted for various cloud services.

Rclone supports over 70 cloud storage providers, including popular services like Google Drive, Amazon S3, Dropbox, Backblaze B2, Microsoft OneDrive, and standard protocols like SFTP, FTP, and WebDAV. It's designed to be efficient, reliable, and versatile for tasks ranging from backup and synchronization to data migration and mounting cloud storage as local drives.

Core Features

Rclone offers a wide array of functionalities for file management:

 Copy: Copies new or changed files from source to destination.

 Sync: Makes a source and destination directory identical, typically in one direction.

 Move: Moves files from source to destination, deleting them from the source after verification.

 Mount: Mounts any local, cloud, or virtual filesystem as a standard disk on your operating system.

 Crypt: Provides client-side encryption of files before they are uploaded to cloud storage, and decryption upon download.

 Check: Verifies file integrity by comparing hashes (e.g., MD5, SHA1) and checks for missing or extra files.

 List: Various commands (ls, lsl, lsd, lsjson) to list files and directories on remotes.

 Serve: Can serve remotes over protocols like SFTP, HTTP, WebDAV, and FTP.

Installation on Debian/Ubuntu

Rclone is readily available in the package repositories for Debian-based systems.

Update System Packages: It's good practice to update your package list and upgrade existing packages before installing new software.

 sudo apt update && sudo apt upgrade -y

Install Rclone: Install rclone directly from the Debian repositories.

 sudo apt install rclone

Verify Installation: Check the installed version to confirm rclone is working correctly.

 rclone version

This command will output the rclone version, OS/architecture, and Go language version it was compiled with.

Configuration (Adding a Cloud Storage Remote)

Before using rclone to transfer files, you need to configure it to connect to your specific cloud storage provider. This involves an interactive setup process.

Start Configuration: Initiate the interactive configuration by running:

 rclone config

If it's your first time, it will prompt: No remotes found, make a new one? (y/n)

Create a New Remote: Type n (for new remote) and press Enter. You will then be asked for a name for your new remote. This name will be used to refer to your cloud storage in rclone commands.

 n) New remote
 s) Set configuration password
 q) Quit config
 x) Advanced config
 y) Same as New remote but skip name
 e) Edit existing remote
 r) Rename remote
 c) Copy remote
 rc) Show remote configs
 d) Delete remote
 p) Print existing remotes

Current remotes:

 n) New remote
 s) Set configuration password
 q) Quit config
 e) Edit existing remote
 r) Rename remote
 c) Copy remote
 d) Delete remote
 p) Print existing remotes

Enter name for new remote. name> pulsedmedia.com Server

(Replace Pulsed Media Server with your preferred name).

Choose Storage Type: A list of supported cloud storage providers will appear. Type the number corresponding to your desired service.

Type of storage to configure.

   ... (long list of options) ...
   13 / pulsedmedia.com
    \ "drive"
 ... (rest of list) ...
 Storage> 13

Follow Provider-Specific Prompts: Rclone will then ask for details specific to your chosen cloud provider. Follow these prompts carefully. Many options can be left as default by pressing Enter.

Save Remote: Once all details are entered and authentication (if required) is complete, rclone will ask to save the remote.

 y) Yes this is OK (default)
 e) Edit this remote
 d) Delete this remote
 y/e/d> y

Type y and press Enter to save the remote.

Quit Configuration: You can now quit the configuration session.

 q) Quit config
 e) Edit existing remote

...

 q) Quit config
 > q

Basic Usage Examples

Here are some common rclone commands. Replace pulsedmedia.com with the name of your configured remote and /path/to/local/folder with your local directory.

List Contents of a Remote Directory: To see the files and folders in the root of your remote:

rclone ls pulsedmedia.com:

To list contents of a specific folder on your remote:

rclone ls pulsedmedia.com:pulsedmedia-server2.com/

Copy Files from Local to Remote: Copies a single file from your local machine to the remote:

 rclone copy /path/to/local/file.txt pulsedmedia.com:

Copies an entire folder from your local machine to a folder on the remote:

 rclone copy /path/to/local/folder pulsedmedia.com:pulsedmedia-server2.com

Copy Files from Remote to Local: Copies a file from the remote to your local machine:

 rclone copy MyGoogleDrive:MyRemoteFile.txt /path/to/local/downloads/

Copies an entire folder from the remote to your local machine:

 rclone copy pulsedmedia.com:pulsedmedia-server2.com
 /path/to/local/downloads/

Synchronize a Local Folder with a Remote Folder (One-Way): This command makes the destination identical to the source. Files that exist in the destination but not in the source will be deleted from the destination. Use with caution!

 rclone sync /path/to/local/folder pulsedmedia.com:RemoteBackupFolder

Always test with --dry-run first:

 rclone sync --dry-run /path/to/local/folder pulsedmedia.com:RemoteBackupFolder

Move Files from Local to Remote: Moves (copies and then deletes from source) files from your local machine to the remote:

 rclone move /path/to/local/folder pulsedmedia.com:ProcessedFiles

Mount a Remote as a Local Drive: Requires fuse to be installed (sudo apt install fuse). This allows you to access your cloud storage as if it were a local folder.

 mkdir ~/my_cloud_mount
 rclone mount pulsedmedia.com: ~/my_cloud_mount --vfs-cache-mode full &

The & puts the process in the background. To unmount:

 fusermount -u ~/my_cloud_mount

See Also

• Command-line interface
• rsync