Tiered media storage with rclone and mergerfs
A Pulsed Media storage box turns your seedbox into something much larger than its own disk. Keep the fast seedbox as a hot tier for active downloads and streaming, keep the cheap storage box as a cold archive, and mount the two together so Jellyfin, Sonarr and Radarr (and, with mergerfs, your torrent client) see one library and never care which tier a file sits on. Traffic between a Pulsed Media seedbox and a Pulsed Media storage box in the same datacenter is unmetered at up to 10Gbps, so pulling a file back from the cold tier costs you nothing.
Everything on this page runs as a normal seedbox user. No root. The setup was built and tested by Pulsed Media community member StupidGenius on a live seedbox and storage box pair; the commands were re-checked on a test account (Debian 11, rclone 1.69.1). The pitfalls below are the things that actually went wrong along the way.
To play the library from a different machine, such as a home server or a Minidedi, see Seedbox Media Library on Separate Systems.
Two setups, in order of effort:
- Just streaming? Cache the storage box for streaming — one
rclone mount, five minutes. Point Jellyfin at it and watch. - Want a real hot/cold library? Build a tiered library — a union filesystem plus a mover script, so new downloads land on the fast disk and age out to the storage box automatically.
How it works
apps see ONE folder ─────────────► ~/media/{Movies,TV,Music}
▲
union = ~/media-local (hot tier: seedbox disk)
+ ~/mnt/storage (cold tier: rclone SFTP mount)
▲
storagebox:media (your storage box)
▲
mover ships aged files hot ──► cold on a schedule
New writes land on the hot tier and seed at full local speed. A scheduled mover ships older files to the storage box. Reads fall through to whichever tier holds the file. The apps only ever use the one merged path, so nothing they store ever has to change.
Before you start
You need an active seedbox and a storage box, and SSH access to the seedbox. Any seedbox tier works; the SSD and NVMe tiers gain the most, because the whole point is keeping hot data on fast disk.
Know your quota, not df. On a shared seedbox df -h shows the whole underlying filesystem, not your slice. Your real limit comes from:
quota -s
Always pass -s. Without it, quota can report raw kilobyte block counts that look like gigabyte numbers and quietly break any threshold script that reads them.
Replace these placeholders throughout: USERNAME (your seedbox username), STORAGE.pulsedmedia.com (your storage box hostname), PORT (its SSH port).
Cache a storage box for streaming
The simplest use: mount the storage box over SFTP with an rclone VFS cache on the seedbox's fast disk, and stream straight from it. The bare mount shown on the Jellyfin page works, but it reads directly from the remote with no local cache, so seeks stutter and a dropped read restarts the file. For a media server, use --vfs-cache-mode full plus the read-ahead and cache flags the short command leaves out. In full mode rclone buffers reads and writes to the cache directory as sparse files, keeps only the parts of each file you actually touch, retries failed reads, and lets the player seek anywhere.
mkdir -p ~/mnt/storage ~/.cache/rclone-storage rclone mount storagebox:/media ~/mnt/storage \ --vfs-cache-mode full \ --cache-dir ~/.cache/rclone-storage \ --vfs-cache-max-size 50G \ --vfs-cache-max-age 24h \ --vfs-cache-min-free-space 10G \ --vfs-read-ahead 1G \ --buffer-size 64M \ --dir-cache-time 1h \ --daemon
The flags the short command omits are the ones that matter for smooth playback:
| Flag | What it does |
|---|---|
--vfs-read-ahead 1G |
Reads ahead onto the cache disk beyond the in-memory buffer. The single biggest smoothness win for streaming — set it large (1–2G for 4K remuxes) and keep --buffer-size modest so RAM does not blow out on a 4GB box.
|
--vfs-cache-min-free-space 10G |
Stops the cache from filling the disk. Without it, only --vfs-cache-max-size applies, and the cache can crowd out everything else on a shared partition.
|
no --allow-other |
Leave it out on a seedbox. The media stack runs Jellyfin, Sonarr and Radarr as your own user, so they already read the mount, with no --uid or --gid needed. The flag only works after a root-level user_allow_other change, and on a shared server it could let other accounts read the mount.
|
Put the cache directory on an ext4, xfs or btrfs filesystem. full mode relies on sparse files; exFAT and FAT do not support them, and rclone will log an error and crawl.
Mount to watch, rclone copy to move. A mount is tuned to play one file smoothly, not to move a library fast. Copying a whole folder through the mountpoint with cp or rsync serialises everything. To move data in bulk, run rclone against the remote directly so it parallelises across files:
rclone copy storagebox:/media/Movies ~/local/Movies -P --transfers 8
Over a single stream, an SFTP transfer settles around 33–40 MB/s no matter how fast the link is — that is the nature of one stream, not a server fault. --transfers 8 runs eight files at once and fills the pipe; 4 to 8 is the sweet spot, higher rarely helps. If a cold file buffers while streaming, raise --vfs-read-ahead (and --buffer-size if RAM allows). Leave --sftp-chunk-size at its default on a mount and on a copy down: see Gotchas.
A tiered library
Caching is bounded and self-managing, but it does not decide what lives where. A tiered library goes further: the seedbox holds new and recently-watched content on fast disk, the storage box holds the deep archive, and a union filesystem presents both as one folder so the apps never see a file move between tiers.
Three rules make the design work:
- The media apps only ever use the union path (
~/media/…). Set root folders and library paths once; the union reshuffles what is underneath. With the rclone union the torrent client is the one exception (see Option A). - New writes always land on the hot tier. Both union tools below can guarantee this.
- With mergerfs, the torrent directory lives inside the union too, so Sonarr and Radarr imports are hardlinks (no extra disk), and it is excluded from the mover so active seeds never migrate. An rclone union cannot make hardlinks; Option A below says what to do instead.
Pick one union tool. Option B needs the storage box mounted first (the rclone mount from the section above works as the cold-tier branch); Option A reads the storage box itself and needs no separate mount.
Option A: rclone union (simplest, no extra software)
rclone is already installed on your seedbox, and its built-in union backend does exactly this job — no second program to install. Add a union remote whose upstreams are your local hot directory and the storage box, with the local branch marked writable so new files land there:
# ~/.config/rclone/rclone.conf [storagebox] type = sftp host = STORAGE.pulsedmedia.com port = PORT user = USERNAME key_file = ~/.ssh/id_ed25519_storagebox [media] type = union upstreams = /home/USERNAME/media-local storagebox:media action_policy = epall create_policy = ff
rclone mount media: ~/media --vfs-cache-mode full --dir-cache-time 1h --daemon
The create_policy = ff ("first found") sends every new file to the first upstream to respond — your local disk, which always answers before the storage box over SFTP — so new downloads land on the fast hot tier, not the cold one. (Do not use the epmfs default here: it picks the branch with the most free space, which is your big storage box, and would send new writes straight to the slow tier.) Reads fall through to the storage box.
An rclone union cannot hardlink: ln on it fails with "Function not implemented", so every Sonarr and Radarr import is a copy. Switch off "Use Hardlinks instead of Copy" in both apps and expect each import to cost the file twice until the torrent is removed. Point the torrent client at the plain local folder ~/media-local/torrents, never at the union, and use ~/media only for the library. A file written to the union is visible at once and lands in ~/media-local about five seconds later (the VFS write-back delay). This is the lowest-dependency path; use it when you have no hardlink imports or when mergerfs will not mount. If Sonarr or Radarr hardlink your imports, use Option B.
Option B: mergerfs (the Sonarr/Radarr power-user setup)
mergerfs is the union filesystem most of the self-hosted media community uses, because its hardlink behaviour and per-branch policies are battle-tested with the *arr apps. It is not part of the seedbox image. From the mergerfs releases page download the static tarball (the file name contains static-linux_amd64), check its sha256 against the digest the release lists, extract it and copy only usr/local/bin/mergerfs into ~/bin. Do not copy mergerfs-fusermount out of the tarball: as your own user it is not setuid and the mount fails; without it mergerfs uses the system fusermount3. Then union your local directory with the rclone mount:
~/bin/mergerfs \ -o category.create=ff,cache.files=partial,dropcacheonclose=true,minfreespace=5G \ /home/USERNAME/media-local:/home/USERNAME/mnt/storage /home/USERNAME/media
category.create=ff ("first found", local branch listed first) sends every new file to the hot tier; the storage box only ever receives files from the mover. minfreespace is the free space of the hot folder's filesystem below which new files go to the next branch, so keep it under what df ~ shows: with a value above the free space, new files stop landing on the hot tier. Verify both mounts came up:
mountpoint ~/mnt/storage && mountpoint ~/media && ls ~/media
To unmount either FUSE mount, always use fusermount3 -u <path>. Never Ctrl+C or kill a FUSE process — that leaves a kernel mount registration behind that ps cannot see but still occupies the mountpoint, producing confusing "it says mounted but shows nothing" states.
Route your torrent client through the union
This section is for Option B (mergerfs); on an rclone union imports copy, as Option A explains. Hardlinks cannot cross a mount boundary. If your torrent client saves to a plain directory while the library lives on the union, Sonarr and Radarr silently copy every import instead of hardlinking — doubling disk use per item. Put both ends inside the union and the hardlink passes straight through to the local branch. In qBittorrent (Options → Downloads):
- Default Save Path:
~/media/torrents - Keep incomplete torrents in:
~/media/torrents/incomplete
Verify after your first import that the seed file and the library file share an inode (link count 2):
stat -c '%h %n' ~/media/torrents/SomeFile.mkv ~/media/Movies/SomeFile.mkv
If the library copy shows link count 1, imports are copying — recheck the paths. Set seed goals in the *arr apps, set qBittorrent's seeding-limit action to pause (never "remove and delete"), and let the *arr apps be the single authority for removing finished torrents.
The quota-aware mover
A scheduled script ships aged files from the hot tier to the storage box when your quota fills. Two safety habits are baked in: it only lists what it would move unless you give it --run, and it moves only files that are not still hardlinked to an active seed (moving a still-seeding file frees no space and wastes upload bandwidth). It also moves nothing when it cannot read your quota or cannot reach the storage box. It is a PHP script (tested with PHP 7.4) and it keeps its list, lock and log in a private folder, ~/mover.
<?php
// php ~/scripts/media-mover.php [--run] [--force] (no --run: only list what would move; --force: ignore the quota threshold)
$home = getenv('HOME'); $hot = "$home/media-local"; $dest = 'storagebox:media'; $keep = ['torrents'];
$ageDays = 14; $percent = 85; $dir = "$home/mover"; $run = in_array('--run', $argv, true);
function note($m) { global $dir; file_put_contents("$dir/mover.log", gmdate('c') . " $m\n", FILE_APPEND); echo "$m\n"; }
@mkdir($dir, 0700, true);
if (is_file("$dir/mover.log") && filesize("$dir/mover.log") > 5000000) rename("$dir/mover.log", "$dir/mover.log.1");
$lock = fopen("$dir/mover.lock", 'c'); if (!flock($lock, LOCK_EX | LOCK_NB)) { note('another run is active'); exit(0); }
// the soft quota line of `quota -s -w`; when it cannot be read nothing moves, unless --force
if (!in_array('--force', $argv, true)) {
$q = shell_exec('quota -s -w 2>/dev/null'); $u = ['K' => 1 / 1048576, 'M' => 1 / 1024, 'G' => 1, 'T' => 1024, '' => 1 / 1048576];
if (!preg_match('~^\s*/\S+\s+([\d.]+)([KMGT]?)\*?\s+([\d.]+)([KMGT]?)~m', (string) $q, $m)) { note('quota unreadable, nothing moved'); exit(2); }
$used = $m[1] * $u[$m[2]]; $limit = $m[3] * $u[$m[4]];
if ($limit <= 0 || $used < $limit * $percent / 100) { note(sprintf('%.1f GiB used of %.1f: below %d%%, nothing moved', $used, $limit, $percent)); exit(0); }
}
exec('rclone lsf ' . escapeshellarg($dest) . ' --max-depth 1 >/dev/null 2>&1', $o, $rc); if ($rc !== 0) { note('cold side unreachable, nothing moved'); exit(3); }
$list = []; $cut = time() - $ageDays * 86400;
$it = new RecursiveIteratorIterator(new RecursiveDirectoryIterator($hot, FilesystemIterator::SKIP_DOTS));
foreach ($it as $f) {
$rel = substr($f->getPathname(), strlen($hot) + 1);
if (in_array(explode('/', $rel)[0], $keep, true) || !$f->isFile() || $f->isLink()) continue;
$s = lstat($f->getPathname());
if ($s && $s['mtime'] < $cut && $s['nlink'] === 1) $list[] = $rel;
}
note(count($list) . ' file(s) older than ' . $ageDays . ' days with one link' . ($run ? '' : ' (dry run, add --run to move)'));
if (!$list) exit(0);
$fl = "$dir/mover.files"; file_put_contents($fl, implode("\n", $list) . "\n"); chmod($fl, 0600);
$cmd = 'rclone move ' . escapeshellarg($hot) . ' ' . escapeshellarg($dest) . ' --files-from ' . escapeshellarg($fl)
. ' --transfers 4 --log-level INFO --log-file ' . escapeshellarg("$dir/mover.log") . ($run ? '' : ' --dry-run');
passthru($cmd, $rc); exit($rc);
The quota threshold is $percent, 85 percent of the soft limit that quota -s -w reports. The age limit is $ageDays. If your torrent client saves anywhere else under ~/media-local, add that folder to $keep first. A moved file exists once, on the storage box: RAID survives a failed drive and is not a backup, so move only what you are content to keep on the box alone.
Test it before trusting it: plant a throwaway file with a backdated timestamp (touch -d '30 days ago' ~/media-local/test-old.bin), run php ~/scripts/media-mover.php --force (a dry run that ignores the quota threshold) and read the list. Then run php ~/scripts/media-mover.php --run --force once and confirm the file is gone from ~/media-local, present on the storage box (rclone lsf storagebox:media), still visible at the same union path, and reads back with a matching checksum (compare sha256sum of the union path with rclone cat storagebox:media/test-old.bin | sha256sum). rclone move deletes each source file only after that file's transfer completes, so an interrupted run never leaves a half-file. Remove the test file afterwards with rm ~/media/test-old.bin.
Schedule it from your own crontab (crontab -e) once the test has passed; the first run above has created ~/mover, which the log redirect needs:
30 4 * * * php /home/USERNAME/scripts/media-mover.php --run >> /home/USERNAME/mover/cron.log 2>&1
Keeping the mounts alive
Mounts must survive a crash. A small watchdog on your crontab remounts anything that has dropped:
# ~/scripts/mount-watchdog.sh — re-run the rclone mount / mergerfs commands if their mountpoints are gone */5 * * * * /home/USERNAME/scripts/mount-watchdog.sh
Start the union only after the rclone mount is confirmed live (check with mountpoint -q) — a union built against a not-yet-ready mount comes up half-empty. A cron watchdog is the reliable choice here: it runs in the global mount namespace, so the mounts it creates are visible to your shell and your apps. Avoid mounting from systemd --user units without testing first — on a multi-tenant host the user manager can run in a private mount namespace, where the units report active and the mounts genuinely exist but nothing else can see them. Test with readlink /proc/self/ns/mnt versus systemd-run --user --pipe --wait readlink /proc/self/ns/mnt; if the two differ, do not mount from systemd user units.
Gotchas
- Jellyfin shows an empty list under
/home. Pulsed Media keeps/homenon-listable so accounts cannot see each other, and Jellyfin's folder picker cannot browse into it. Type the full path into the folder field instead, for example/home/USERNAME/media. fusermount3missing. Pulsed Media seedboxes already ship and holdfuse3, sorclone mountworks out of the box. If a mount fails with a missing-fusermount3error, that is a host-side package problem — open a support ticket rather than working around it, and support can restore it.- mergerfs says
fusermount: mount failed: Operation not permittedwhile an rclone mount works. The mergerfs download contains its own helper,mergerfs-fusermount. Extracted into~/binas your own user it is not setuid, and only a setuid helper can mount without root. Delete that copy and keep only themergerfsbinary in~/bin: mergerfs then falls through to the systemfusermount3, which is setuid, and the mount works. If it still fails, open a support ticket and name that error, or use Option A, the rclone union, which does not need it. - Leave
--sftp-chunk-sizeat its default on a mount and on a copy down. With rclone 1.69.1,--sftp-chunk-sizeset to 255 KiB made a cached mount of a 16 MiB file return wrong bytes from byte 65537 on, andrclone copyof the same file failed with "corrupted on transfer: sizes differ". Uploads were not affected. A mount or copy that sets it should drop the flag; after the first transfer of any large file, compare checksums (sha256sumon the mount path,rclone cat storagebox:media/path | sha256sumon the remote). - New files do not appear. Plain SFTP has no change notification, so a file added on the storage box shows up in the mount only after
--dir-cache-timeexpires. Raise it for a mostly-static library; send the mount processSIGHUPto refresh immediately. - Cache filling the disk. Almost always a missing
--vfs-cache-min-free-space, or a--vfs-cache-max-sizesmaller than the largest single file you stream — an open file cannot be evicted mid-playback, so size the cache to at least twice your biggest title. - Login loops on the rclone web GUI. It uses the same password as your seedbox web panel — there is no separate rclone login. A repeating prompt is usually a browser autofilling your billing email as the username.
- Do not
rm -rfthe cache to "fix" a stuck mount. Infullmode, writes land on the cache first and flush to the storage box a few seconds later. Files that have not flushed live only in the cache directory and are recovered on the next mount with the same--cache-dir. Delete the cache and you delete unflushed writes. - A
mvthrough the union is not always a local operation. It renames the file on whichever branch it currently lives on. Reorganising a cold file can quietly create directories on the storage box. To pull something hot on purpose, userclone move storagebox:path ~/media-local/pathexplicitly.
See also
- Rclone tutorial — install rclone, configure an SFTP remote, and the basic mount and sync commands.
- Jellyfin — run a self-hosted media server on your seedbox and point it at the mount.
- Storage Boxes — the cheap bulk-capacity boxes this guide uses as the cold tier.
- Seedbox for Plex and Jellyfin — choosing a plan and the media stack this design feeds.
- Seedbox access via FTP, SSH and SFTP — SSH key and connection setup.
A small fast box in front of a large cheap storage box gives you a media library bigger than either alone, with your active data on fast disk and your archive on RAID5. See the storage box plans.
Guide contributed by Pulsed Media community member StupidGenius.