Difference between revisions of "Rclone"

From Pulsed Media Wiki
(Created page with "== Rclone == '''Rclone''' is a free and open-source command-line program used to manage, sync, and transfer files across various cloud storage providers, local storage, and r...")
 
(Guides: Linux: Information: Pulsed Media)
 
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Rclone ==
+
= Rclone =
  
'''Rclone''' is a free and open-source command-line program used to manage, sync, and transfer files across various cloud storage providers, local storage, and remote servers. Often described as “''rsync for cloud storage,''” Rclone supports over 50 different backends including [[Google Drive]], [[Dropbox]], [[OneDrive]], [[Amazon S3]], [[Box]], [[FTP]], [[SFTP]], and many more.
+
[[File:rclone.jpeg|thumb|right|300px|Illustrative rclone image.]]
  
Rclone is widely used for backing up data, automating cloud syncs, managing seedboxes, and performing encrypted, high-performance file operations.
+
'''Rclone''' is a powerful, open-source, command-line program used for managing files on cloud storage and other remote storage systems. Often referred to as "The Swiss army knife of cloud storage," it provides features similar to Unix commands like rsync, cp, mv, mount, ls, and cat, but adapted for various cloud services.
  
=== Key Features ===
+
Rclone supports over 70 cloud storage providers, including popular services like Google Drive, Amazon S3, Dropbox, Backblaze B2, Microsoft OneDrive, and standard protocols like SFTP, FTP, and WebDAV. It's designed to be efficient, reliable, and versatile for tasks ranging from backup and synchronization to data migration and mounting cloud storage as local drives.
  
* **Sync files** – Between cloud, local, or remote storage
+
== Core Features ==
* **Mount cloud storage** – As a virtual drive on your local system
+
Rclone offers a wide array of functionalities for file management:
* **Copy/move operations** – With filters and advanced flags
 
* **Encryption support** – Encrypt filenames, file content, and metadata
 
* **Transfer optimization** – Multi-threaded uploads/downloads, chunked transfers
 
* **Scheduling & automation** – Cron-friendly CLI tool for scripts and backups
 
* **Cross-platform** – Available for Linux, Windows, macOS, BSD, Android, and Docker
 
  
=== Supported Storage Backends ===
+
  '''Copy:''' Copies new or changed files from source to destination.
  
* Cloud: Google Drive, OneDrive, Dropbox, Box, Mega, pCloud
+
  '''Sync:''' Makes a source and destination directory identical, typically in one direction.
* Object Storage: Amazon S3, Wasabi, Backblaze B2, DigitalOcean Spaces
 
* WebDAV-compatible: Nextcloud, ownCloud, seedboxes
 
* FTP/SFTP/HTTP remote servers
 
* Local file systems
 
  
See the full list: [https://rclone.org/overview/ rclone.org/overview]
+
  '''Move:''' Moves files from source to destination, deleting them from the source after verification.
  
=== Installation ===
+
  '''Mount:''' Mounts any local, cloud, or virtual filesystem as a standard disk on your operating system.
  
On Debian/Ubuntu:
+
  '''Crypt:''' Provides client-side encryption of files before they are uploaded to cloud storage, and decryption upon download.
  
<syntaxhighlight lang="bash">
+
  '''Check:''' Verifies file integrity by comparing hashes (e.g., MD5, SHA1) and checks for missing or extra files.
sudo apt install rclone
 
</syntaxhighlight>
 
  
Or use the official install script:
+
  '''List:''' Various commands (ls, lsl, lsd, lsjson) to list files and directories on remotes.
  
<syntaxhighlight lang="bash">
+
  '''Serve:''' Can serve remotes over protocols like SFTP, HTTP, WebDAV, and FTP.
curl https://rclone.org/install.sh | sudo bash
 
</syntaxhighlight>
 
  
=== Configuration ===
+
== Installation on Debian/Ubuntu ==
 +
Rclone is readily available in the package repositories for Debian-based systems.
  
To add a remote backend:
+
'''Update System Packages:'''
 +
It's good practice to update your package list and upgrade existing packages before installing new software.
  
<syntaxhighlight lang="bash">
+
  sudo apt update && sudo apt upgrade -y
rclone config
 
</syntaxhighlight>
 
  
Follow the interactive prompts to:
+
'''Install Rclone:'''
* Create a new remote
+
Install rclone directly from the Debian repositories.
* Select storage type (e.g., Google Drive, WebDAV)
 
* Authorize access using OAuth or credentials
 
  
Example configuration name: `gdrive`, `seedbox`, `dropbox`
+
  sudo apt install rclone
  
=== Basic Commands ===
+
'''Verify Installation:'''
 +
Check the installed version to confirm rclone is working correctly.
  
'''Copy a local folder to Google Drive:'''
+
  rclone version
  
<syntaxhighlight lang="bash">
+
This command will output the rclone version, OS/architecture, and Go language version it was compiled with.
rclone copy /local/folder gdrive:/remote/folder
 
</syntaxhighlight>
 
  
'''Sync two directories (deletes extra files):'''
+
== Configuration (Adding a Cloud Storage Remote) ==
 +
Before using rclone to transfer files, you need to configure it to connect to your specific cloud storage provider. This involves an interactive setup process.
  
<syntaxhighlight lang="bash">
+
'''Start Configuration:'''
rclone sync /local/folder remote:/folder
+
Initiate the interactive configuration by running:
</syntaxhighlight>
 
  
'''Mount remote storage as a filesystem:'''
+
  rclone config
  
<syntaxhighlight lang="bash">
+
If it's your first time, it will prompt: No remotes found, make a new one? (y/n)
rclone mount gdrive:/ ~/mnt/gdrive
 
</syntaxhighlight>
 
  
'''List files in a remote directory:'''
+
'''Create a New Remote:'''
 +
Type n (for new remote) and press Enter.
 +
You will then be asked for a name for your new remote. This name will be used to refer to your cloud storage in rclone commands.
  
<syntaxhighlight lang="bash">
+
  n) New remote
rclone ls remote:/path
+
  s) Set configuration password
</syntaxhighlight>
+
  q) Quit config
 +
  x) Advanced config
 +
  y) Same as New remote but skip name
 +
  e) Edit existing remote
 +
  r) Rename remote
 +
  c) Copy remote
 +
  rc) Show remote configs
 +
  d) Delete remote
 +
  p) Print existing remotes
  
'''Encrypt a remote:'''
+
Current remotes:
  
Use `rclone config` to create an encrypted remote layered on top of an existing remote.
+
  n) New remote
 +
  s) Set configuration password
 +
  q) Quit config
 +
  e) Edit existing remote
 +
  r) Rename remote
 +
  c) Copy remote
 +
  d) Delete remote
 +
  p) Print existing remotes
  
=== Scheduling and Automation ===
+
Enter name for new remote.
 +
name> pulsedmedia.com Server
  
Rclone is ideal for use in:
+
(Replace [[Pulsed Media]] Server with your preferred name).
  
* Shell scripts
+
'''Choose Storage Type:'''
* Cron jobs for daily backups
+
A list of supported cloud storage providers will appear. Type the number corresponding to your desired service.
* Seedbox automation
 
* Server-to-cloud file migrations
 
  
Example cron entry for daily sync:
+
Type of storage to configure.
 +
    ... (long list of options) ...
 +
    13 / pulsedmedia.com
 +
    \ "drive"
 +
  ... (rest of list) ...
 +
  Storage> 13
  
<syntaxhighlight lang="bash">
+
'''Follow Provider-Specific Prompts:'''
0 2 * * * /usr/bin/rclone sync /data remote:/backup
+
Rclone will then ask for details specific to your chosen cloud provider. Follow these prompts carefully. Many options can be left as default by pressing Enter.
</syntaxhighlight>
 
  
=== Use Cases ===
+
'''Save Remote:'''
 +
Once all details are entered and authentication (if required) is complete, rclone will ask to save the remote.
  
* Sync media from a [[seedbox]] to cloud storage
+
  y) Yes this is OK (default)
* Backup server files to Google Drive or Dropbox
+
  e) Edit this remote
* Mount remote drives for live access
+
  d) Delete this remote
* Transfer between two cloud providers directly
+
  y/e/d> y
* Encrypt data before uploading to untrusted storage
 
  
=== Rclone GUI Tools ===
+
Type y and press Enter to save the remote.
  
While Rclone is primarily CLI-based, GUI options include:
+
'''Quit Configuration:'''
 +
You can now quit the configuration session.
  
* **Rclone Web UI** – Built-in, browser-based dashboard (`rclone rcd --rc-web-gui`)
+
  q) Quit config
* **RcloneBrowser** – Community-developed graphical frontend
+
  e) Edit existing remote
 +
...
 +
  q) Quit config
 +
  > q
  
=== Security and Privacy ===
+
== Basic Usage Examples ==
 +
Here are some common rclone commands. Replace pulsedmedia.com with the name of your configured remote and /path/to/local/folder with your local directory.
  
* Offers optional **encryption** for file content, names, and directory structure
+
'''List Contents of a Remote Directory:'''
* Uses OAuth 2.0 or direct credentials (depending on backend)
+
To see the files and folders in the root of your remote:
* Supports `--delete-excluded`, `--dry-run`, `--backup-dir`, and more for safe sync
 
  
=== Comparison with rsync ===
+
rclone ls pulsedmedia.com:
  
{| class="wikitable"
+
To list contents of a specific folder on your remote:
! Feature !! Rclone !! rsync
 
|-
 
| Cloud backend support || Yes || No
 
| SSH/local support || Yes || Yes
 
| Encryption (built-in) || Yes || No
 
| Mount remote drive || Yes || No
 
| Bandwidth throttling || Yes || Limited
 
| GUI options || Yes (via Web UI) || No
 
| Delta syncing || No (uploads entire changed file) || Yes
 
| Scripting support || Excellent || Excellent
 
|}
 
  
=== Related Tools and Concepts ===
+
rclone ls pulsedmedia.com:pulsedmedia-server2.com/
  
* [[rsync]]
+
'''Copy Files from Local to Remote:'''
* [[SCP]]
+
Copies a single file from your local machine to the remote:
* [[SFTP]]
 
* [[Cloud storage]]
 
* [[Seedbox]]
 
* [[Nextcloud]]
 
* [[Cron]]
 
  
=== See Also ===
+
  rclone copy /path/to/local/file.txt pulsedmedia.com:
  
* [https://rclone.org/ Official Rclone Website]
+
Copies an entire folder from your local machine to a folder on the remote:
* [[Backup software]]
+
 
* [[Encryption]]
+
  rclone copy /path/to/local/folder pulsedmedia.com:pulsedmedia-server2.com
* [[SSH]]
+
 
* [[WebDAV]]
+
'''Copy Files from Remote to Local:'''
* [[Google Drive]]
+
Copies a file from the remote to your local machine:
 +
 
 +
  rclone copy MyGoogleDrive:MyRemoteFile.txt /path/to/local/downloads/
 +
 
 +
Copies an entire folder from the remote to your local machine:
 +
 
 +
  rclone copy pulsedmedia.com:pulsedmedia-server2.com
 +
  /path/to/local/downloads/
 +
 
 +
'''Synchronize a Local Folder with a Remote Folder (One-Way):'''
 +
This command makes the destination identical to the source. Files that exist in the destination but not in the source will be deleted from the destination. Use with caution!
 +
 
 +
  rclone sync /path/to/local/folder pulsedmedia.com:RemoteBackupFolder
 +
 
 +
'''Always test with --dry-run first:'''
 +
 
 +
  rclone sync --dry-run /path/to/local/folder pulsedmedia.com:RemoteBackupFolder
 +
 
 +
'''Move Files from Local to Remote:'''
 +
Moves (copies and then deletes from source) files from your local machine to the remote:
 +
 
 +
  rclone move /path/to/local/folder pulsedmedia.com:ProcessedFiles
 +
 
 +
'''Mount a Remote as a Local Drive:'''
 +
Requires fuse to be installed (sudo apt install fuse). This allows you to access your cloud storage as if it were a local folder.
 +
 
 +
  mkdir ~/my_cloud_mount
 +
  rclone mount pulsedmedia.com: ~/my_cloud_mount --vfs-cache-mode full &
 +
 
 +
The & puts the process in the background. To unmount:
 +
 
 +
  fusermount -u ~/my_cloud_mount
 +
 
 +
== See Also ==
 +
 
 +
*[[Command-line interface]]
 +
*[[rsync]]
 +
 
 +
[[Category:Information]]
 +
[[Category:Guides]]

Latest revision as of 11:58, 9 June 2025

Rclone

Illustrative rclone image.

Rclone is a powerful, open-source, command-line program used for managing files on cloud storage and other remote storage systems. Often referred to as "The Swiss army knife of cloud storage," it provides features similar to Unix commands like rsync, cp, mv, mount, ls, and cat, but adapted for various cloud services.

Rclone supports over 70 cloud storage providers, including popular services like Google Drive, Amazon S3, Dropbox, Backblaze B2, Microsoft OneDrive, and standard protocols like SFTP, FTP, and WebDAV. It's designed to be efficient, reliable, and versatile for tasks ranging from backup and synchronization to data migration and mounting cloud storage as local drives.

Core Features

Rclone offers a wide array of functionalities for file management:

 Copy: Copies new or changed files from source to destination.
 Sync: Makes a source and destination directory identical, typically in one direction.
 Move: Moves files from source to destination, deleting them from the source after verification.
 Mount: Mounts any local, cloud, or virtual filesystem as a standard disk on your operating system.
 Crypt: Provides client-side encryption of files before they are uploaded to cloud storage, and decryption upon download.
 Check: Verifies file integrity by comparing hashes (e.g., MD5, SHA1) and checks for missing or extra files.
 List: Various commands (ls, lsl, lsd, lsjson) to list files and directories on remotes.
 Serve: Can serve remotes over protocols like SFTP, HTTP, WebDAV, and FTP.

Installation on Debian/Ubuntu

Rclone is readily available in the package repositories for Debian-based systems.

Update System Packages: It's good practice to update your package list and upgrade existing packages before installing new software.

 sudo apt update && sudo apt upgrade -y

Install Rclone: Install rclone directly from the Debian repositories.

 sudo apt install rclone

Verify Installation: Check the installed version to confirm rclone is working correctly.

 rclone version

This command will output the rclone version, OS/architecture, and Go language version it was compiled with.

Configuration (Adding a Cloud Storage Remote)

Before using rclone to transfer files, you need to configure it to connect to your specific cloud storage provider. This involves an interactive setup process.

Start Configuration: Initiate the interactive configuration by running:

 rclone config

If it's your first time, it will prompt: No remotes found, make a new one? (y/n)

Create a New Remote: Type n (for new remote) and press Enter. You will then be asked for a name for your new remote. This name will be used to refer to your cloud storage in rclone commands.

 n) New remote
 s) Set configuration password
 q) Quit config
 x) Advanced config
 y) Same as New remote but skip name
 e) Edit existing remote
 r) Rename remote
 c) Copy remote
 rc) Show remote configs
 d) Delete remote
 p) Print existing remotes

Current remotes:

 n) New remote
 s) Set configuration password
 q) Quit config
 e) Edit existing remote
 r) Rename remote
 c) Copy remote
 d) Delete remote
 p) Print existing remotes

Enter name for new remote. name> pulsedmedia.com Server

(Replace Pulsed Media Server with your preferred name).

Choose Storage Type: A list of supported cloud storage providers will appear. Type the number corresponding to your desired service.

Type of storage to configure.

   ... (long list of options) ...
   13 / pulsedmedia.com
    \ "drive"
 ... (rest of list) ...
 Storage> 13

Follow Provider-Specific Prompts: Rclone will then ask for details specific to your chosen cloud provider. Follow these prompts carefully. Many options can be left as default by pressing Enter.

Save Remote: Once all details are entered and authentication (if required) is complete, rclone will ask to save the remote.

 y) Yes this is OK (default)
 e) Edit this remote
 d) Delete this remote
 y/e/d> y

Type y and press Enter to save the remote.

Quit Configuration: You can now quit the configuration session.

 q) Quit config
 e) Edit existing remote

...

 q) Quit config
 > q

Basic Usage Examples

Here are some common rclone commands. Replace pulsedmedia.com with the name of your configured remote and /path/to/local/folder with your local directory.

List Contents of a Remote Directory: To see the files and folders in the root of your remote:

rclone ls pulsedmedia.com:

To list contents of a specific folder on your remote:

rclone ls pulsedmedia.com:pulsedmedia-server2.com/

Copy Files from Local to Remote: Copies a single file from your local machine to the remote:

 rclone copy /path/to/local/file.txt pulsedmedia.com:

Copies an entire folder from your local machine to a folder on the remote:

 rclone copy /path/to/local/folder pulsedmedia.com:pulsedmedia-server2.com

Copy Files from Remote to Local: Copies a file from the remote to your local machine:

 rclone copy MyGoogleDrive:MyRemoteFile.txt /path/to/local/downloads/

Copies an entire folder from the remote to your local machine:

 rclone copy pulsedmedia.com:pulsedmedia-server2.com
 /path/to/local/downloads/

Synchronize a Local Folder with a Remote Folder (One-Way): This command makes the destination identical to the source. Files that exist in the destination but not in the source will be deleted from the destination. Use with caution!

 rclone sync /path/to/local/folder pulsedmedia.com:RemoteBackupFolder

Always test with --dry-run first:

 rclone sync --dry-run /path/to/local/folder pulsedmedia.com:RemoteBackupFolder

Move Files from Local to Remote: Moves (copies and then deletes from source) files from your local machine to the remote:

 rclone move /path/to/local/folder pulsedmedia.com:ProcessedFiles

Mount a Remote as a Local Drive: Requires fuse to be installed (sudo apt install fuse). This allows you to access your cloud storage as if it were a local folder.

 mkdir ~/my_cloud_mount
 rclone mount pulsedmedia.com: ~/my_cloud_mount --vfs-cache-mode full &

The & puts the process in the background. To unmount:

 fusermount -u ~/my_cloud_mount

See Also