Jump to content

Rclone serve s3 on PMSS

From Pulsed Media Wiki


rclone serve s3 is the simplest way to put an S3 API in front of a Pulsed Media storage box or seedbox: one command, a binary that is already installed on every server, and any S3 client can talk to your disk. It is among the lightest of the self-hosted S3 options — in a test on a Pulsed Media storage box the running server held about 51 MB of RAM under a light workload — and the files it stores stay as ordinary files you can still reach over SFTP and SSH. The endpoint runs on your box, the keys are yours, and the bytes stay in Pulsed Media's Finnish datacenters.

rclone serve s3 running on a Pulsed Media storage box: create a bucket, upload an object, list it, and read it back over the S3 API. The server held about 51 MB of RAM, and the same bytes remain ordinary files on disk.

When rclone serve s3 is the right tool

Reach for it when you want an S3 endpoint with the least possible ceremony:

  • A backup target for Restic or a Proxmox Backup Server S3 datastore — the common case, and what it does best.
  • Any application that only speaks S3 but writes little metadata: static-asset uploaders, object-store SDKs, a quick bucket for a script.
  • A throwaway or short-lived endpoint you want up in one command and gone just as fast.

It is deliberately simple. If you need object versioning, bucket policies, or a web console, use MinIO. If you need the endpoint to survive a whole box failing, cluster several boxes with Garage. For a single box and a plain bucket, nothing has less ceremony than rclone serve s3.

What one process needs

Almost nothing. rclone is a single Go binary, already installed fleet-wide, and the s3 server was added in rclone 1.65. On a Pulsed Media storage box (rclone v1.69.1, Debian 12) the running rclone serve s3 process held about 51 MB of resident memory while serving. There is no database and no separate store: objects are written straight to a directory under your home, so the disk cost is exactly the size of your data.

Verified behaviour

On a Pulsed Media storage box, rclone serve s3 was exercised end to end over the S3 API: start the server on a non-default port bound to localhost, create a bucket, upload an object, list it, and read it back. The object returned byte for byte, and the same file was visible on disk under the bucket directory — reachable over SFTP like any other file. The server held about 51 MB of RAM throughout. The screenshot shows that session.

Because the s3 server is marked experimental by the rclone project, it is deliberately narrow: it treats each top-level directory as a bucket and ignores loose files in the root, it does not support object versioning, and server-side multipart copies do not work. For a backup target or a plain bucket none of that matters; if you need those features, MinIO or Garage is the better fit.

Setting it up

Check the version first — the s3 server needs rclone 1.65 or newer, and rclone is already installed:

rclone version

If it is older than 1.65, drop the latest binary from rclone.org/downloads into ~/bin/. Then run the server. Pick your own port — not the default 9000, which automated scanners probe and which another user on a shared box may already hold. Bind it to localhost, or to a VPN or known client IP, rather than all interfaces. Run it inside a tmux or screen session so it survives a disconnect:

tmux new -s rclone-s3
rclone serve s3 --addr 127.0.0.1:PORT \
  --auth-key 'YOUR-ACCESS-KEY,YOUR-SECRET-KEY' \
  /home/USERNAME/s3-bucket
# Detach with Ctrl-b then d. Replace PORT with a high, uncommon number you choose.

Point any S3 client at http://127.0.0.1:PORT (or your chosen bind address) with the access key and secret. Files written to the bucket land as normal files under /home/USERNAME/s3-bucket, so you can still reach them over SFTP or SSH.

A stable hostname that survives a move

If you point a client at the server's own hostname and Pulsed Media later migrates your service, the address changes. Your service permalink avoids that: a permanent mcx.fi hostname, computed from your service ID, that always resolves to the server your service is on. Point your S3 client at <your-service-label>.mcx.fi:PORT and the endpoint keeps working across a migration.

TLS

Some clients, Proxmox Backup Server among them, refuse plain HTTP. A self-signed certificate is quick — set its common name to your service permalink so the trusted name survives a move:

openssl req -x509 -newkey rsa:2048 -days 365 -nodes \
  -keyout /home/USERNAME/s3-key.pem -out /home/USERNAME/s3-cert.pem \
  -subj "/CN=<your-service-label>.mcx.fi"

rclone serve s3 --addr 0.0.0.0:PORT \
  --cert /home/USERNAME/s3-cert.pem --key /home/USERNAME/s3-key.pem \
  --auth-key 'YOUR-ACCESS-KEY,YOUR-SECRET-KEY' /home/USERNAME/s3-bucket

Security

  • Use long, randomly generated values for the access key and secret. Short or default keys let anyone who reaches the port read your data.
  • Do not use the default port 9000. Pick your own, and bind to localhost, a VPN, or a known client IP unless you genuinely need public access.
  • Keep the bucket directory private at rest: chmod 700 /home/USERNAME/s3-bucket.
  • rclone serve s3 moves and stores bytes; it does not encrypt them for you. Combine it with client-side encryption when the data is sensitive.

See also