Jump to content

S3Proxy on PMSS

From Pulsed Media Wiki


S3Proxy puts an S3 API in front of a local filesystem — or in front of another storage backend entirely, bridging S3 to Backblaze B2, Azure Blob, Google Cloud Storage, or OpenStack Swift. On a Pulsed Media storage box or seedbox its most common use is the filesystem backend: an S3 endpoint whose objects are plain files on disk. Unlike the single-binary options, S3Proxy is a Java application, so it runs with a Java runtime in your home directory (or as a rootless container). In a test on a Pulsed Media storage box the JVM held about 141 MB of RAM. It is one of the self-hosted S3 options; the keys are yours and the data stays in Pulsed Media's Finnish datacenters.

S3Proxy on a Pulsed Media storage box, run with a portable Java runtime: bucket create, upload, list, and read-back over the S3 API, with the object stored as a plain file. The JVM held about 141 MB of RAM.

When S3Proxy is the right tool

Reach for S3Proxy when its bridging ability is what you need:

  • You want to front another storage backend with an S3 API — S3Proxy can proxy S3 requests to Backblaze B2, Azure, Google Cloud, or Swift, so an S3-only application can talk to a non-S3 store.
  • You want a simple S3 API over a local filesystem, where each object is a plain file — much like versitygw, with a different backend engine (Apache jclouds).
  • You need a throwaway S3 endpoint for local testing — S3Proxy is widely used as a test double for S3 in development.

If you just want a plain bucket on one box with the lightest footprint, versitygw (about 31 MB) or rclone serve s3 (about 51 MB) are lighter, being single Go binaries. S3Proxy earns its place when you want to bridge S3 onto a different backend.

What one process needs

S3Proxy runs on the JVM, so it needs a Java runtime — version 17 or newer. There is no system Java on a storage box by default, so you either unpack a portable JRE into your home directory or run S3Proxy as a rootless container. In a test on a Pulsed Media storage box (S3Proxy 4.0.0 on OpenJDK 21, Debian 12) the JVM held about 141 MB of resident memory while serving a bucket — heavier than the lightweight Go gateways (rclone at about 51 MB, versitygw at about 31 MB), which is the cost of the JVM. With the filesystem backend, objects are plain files on disk, so the disk cost is just your data.

Verified behaviour

On a Pulsed Media storage box, S3Proxy was exercised end to end: unpack a portable Java runtime, download the S3Proxy jar, start it with the filesystem backend on a non-default port bound to localhost, then over the S3 API create a bucket, upload an object, list it, and read it back byte for byte. A plain HTTP request to the port returned 403 — the gateway was up and correctly requiring an S3 signature. The uploaded object was stored as a plain file on disk under the backend directory. The JVM held about 141 MB of RAM. The screenshot shows that session.

Setting it up

You need a Java 17+ runtime. The simplest no-root path is a portable JRE unpacked into your home directory (an Adoptium/Temurin jre tarball works well); alternatively run S3Proxy as a rootless container — see Docker on PMSS. Then download the S3Proxy jar and write a small properties file:

# ~/s3proxy.conf
s3proxy.endpoint=http://127.0.0.1:PORT
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=YOUR-KEY
s3proxy.credential=YOUR-SECRET
jclouds.provider=filesystem
jclouds.filesystem.basedir=/home/USERNAME/s3proxy-data

Start it (with the portable JRE on your PATH), then point any S3 client at it. Pick your own port — not a default — and keep the endpoint bound to localhost, a VPN, or a known client IP:

export PATH=$HOME/jre/bin:$PATH
./s3proxy --properties ~/s3proxy.conf &

mc alias set s3p http://127.0.0.1:PORT YOUR-KEY YOUR-SECRET
mc mb s3p/mybucket
mc cp file.txt s3p/mybucket/

To bridge to another backend instead of the filesystem, change jclouds.provider (for example to b2, azureblob, or google-cloud-storage) and supply that backend's credentials.

A stable hostname

Point S3 clients at your service permalink<your-service-label>.mcx.fi:PORT — so the endpoint keeps working when Pulsed Media migrates your service.

Security

  • Configure a strong, random identity and credential, and use the aws-v2-or-v4 authorization mode so the gateway requires signed requests — an open, unauthenticated proxy is the wrong default to leave running.
  • Do not use a default port. Pick your own, and bind to localhost, a VPN, or a known client IP unless you need public access.
  • If you bridge to another cloud backend, that backend's credentials live in the config file — keep it private (chmod 600).
  • S3Proxy moves and stores bytes; it does not encrypt them for you. Combine it with client-side encryption for sensitive data.

See also