Jump to content

Linux Kernel 6.1 vs 7.3 on HDD RAID

From Pulsed Media Wiki


With Linux 7.3, a seedbox account held to a disk read cap can truncate a 4 GiB file in a median 0.84 seconds; on Debian 12's stock 6.1 kernel the same truncate takes 24.1 seconds, and another account's worst write wait meanwhile is a median 17.1 seconds. Pulsed Media measured this on a bare-metal server with four 8 TB hard drives, 25 runs per test, together with what 7.3 costs on raw random reads under the bfq scheduler, where that cost comes from, and what it does to a seedbox-shaped load.

Under a 5 reads/s cap, 7.3 finishes the truncate in 0.84 s and the neighbour's worst fsync wait is 0.68 s. Median of 25 runs per kernel.

The problem

Every account on a shared seedbox server uses one filesystem on one RAID array. The kernel can cap an account's reads per second (cgroup v1 blkio.throttle.read_iops_device). Data reads behave. Metadata does not: when the capped account deletes or truncates a large file, ext4 reads block bitmaps while the truncate holds a journal handle. Those reads sit in the same capped queue as the account's data backlog, the journal commit waits for the handle, and every other account's fsync waits for the commit.

On 6.1 that happened in 25 of 25 runs. A run counts as stalled when the other account's worst fsync during the truncate takes longer than one second.

Test system

Item Value
Drives 4 × 8 TB HGST HUH728080ALE601, SATA, AHCI, queue depth 32
Array md RAID0, 2048 KiB chunk, about 29 TiB usable, ext4
Host bare metal, no other workload
Kernels Debian 12 6.1.0-54-amd64; mainline 7.3.4; Debian 10 4.19.0-27 for the old-kernel check
I/O scheduler bfq; none in the scheduler arms; cfq for the 4.19 legacy-block-layer run
Runs 25 per scenario per kernel, blocks in ABBA order, each block booted by kexec
Working set one 8 TiB file, about 28% of the array, caches dropped before each step

Before each block the harness checked that the arrays were healthy, that SMART reallocation and pending counters had not risen (tracked by drive serial, because device names change between boots), and that the kernel log was clean.

Unless a table header says otherwise, each ratio below is the 7.3 median divided by the 6.1 median. The bracket gives the two ABBA block-pair ratios, which show how far the answer moves between blocks; the median of all 25 runs can fall outside them. The results hold for this RAID0 array; RAID5 and RAID10 arrays were not measured in these runs.

Result: the stall

Cap of 5 reads/s, 64-deep read backlog, then a 4 GiB truncate 6.1 7.3
Runs that stalled 25 / 25 1 / 25
Truncate time, median 24.1 s 0.84 s
Other account's worst fsync, median 17.1 s 0.68 s
Other account's p99 fsync 946 ms 205 ms
Journal-wait samples 41 1

Without the backlog both kernels truncate the same file in about half a second (0.41 s and 0.51 s).

Why 7.3 does not stall

7.3's block/blk-throttle.c has a branch 6.1 does not: a bio flagged REQ_META or REQ_SWAP (bio_issue_as_root_blkg() in block/blk-cgroup.h) is dispatched at once even when its cgroup is over the limit, and is still charged to the cgroup, so that cgroup's later reads wait longer. ext4 flags block-bitmap reads REQ_META (fs/ext4/balloc.c) but reads extent-tree blocks with no flags (fs/ext4/extents.c).

The harness samples the truncating process's kernel stack. On the HDD array:

Where the truncate was waiting (25 runs each) 6.1 7.3
ext4_read_block_bitmap (flagged REQ_META) 25 0
__read_extent_tree_block (not flagged) 0 16
ext4_mb_load_buddy_gfp (allocator cache) 0 5
quota write 0 1
no sample 0 3

On 6.1 every sample caught the truncate waiting for a block bitmap. On 7.3 none did; the samples land on other reads, mostly the unflagged extent-tree read, and the truncate finishes in a median 0.84 seconds.

Result: raw reads and a seedbox mix

Test, 7.3 ÷ 6.1 IOPS MB/s
random read 4 KiB, queue depth 32 ×0.64 (0.60-0.71) ×0.64
random read 64 KiB ×0.61 (0.58-0.75) ×0.61
random read 256 KiB ×0.68 (0.64-0.84) ×0.68
random read 1 MiB ×0.56 (0.51-0.70) ×0.56
random read 4 MiB ×0.53 (0.52-0.58) ×0.53
seedbox mix: 95% reads, 512-1024 KiB, queue depth 32-100, background load ×0.97 (0.93-0.97) ×0.97

In the sweep each step uses one request size, so MB/s moves with IOPS. 7.3 serves a third to a half fewer raw random reads at a fixed queue depth of 32. In this run the seedbox-shaped mix loses 3%: a median 123.6 MB/s on 7.3 against 127.7 MB/s on 6.1. The two sections below show where the raw-read loss comes from.

Median IOPS of 25 runs per kernel, O_DIRECT random read at queue depth 32. 7.3 is lower at every request size.
7.3 as a share of 6.1. The raw sweep loses 32-47%; the seedbox-shaped mix keeps 97% of its read throughput.

Request size: count versus bytes

Counted per request, the median read the drives served in the seedbox test was 8 KiB on both kernels. fio itself requests 512-1024 KiB, so the many small reads come from elsewhere in the stack; which process sent them was not traced. Weighted by bytes it was 724 KiB on 6.1 and 512 KiB on 7.3, and requests of 64 KiB or less moved only 3-3.5% of the bytes. Over all 25 runs together, 6.1 sent 7.6% of its reads at 1 MiB or more (5.8% in the median run); 7.3 sent none, its large reads landing at 352, 512 and 672 KiB.

The two kernels also boot with different defaults on this hardware: transparent huge pages always on 6.1 and madvise on 7.3, member-disk max_sectors_kb 1280 against 4096.

Was the old kernel faster?

4.19 ÷ 6.1 blk-mq, bfq legacy block layer, cfq
random read 4 KiB ×1.25 (0.77-1.66) ×1.20 (1.27-1.38)
random read 64 KiB ×1.53 (1.49-1.53) ×1.42 (1.32-1.57)
random read 256 KiB ×1.27 (0.97-1.42) ×1.52 (1.42-1.51)
random read 1 MiB ×1.32 (1.23-1.30) ×1.20 (1.20-1.28)
random read 4 MiB ×1.69 (1.65-1.74) ×1.69 (1.66-1.73)
seedbox mix ×0.85 (0.85-0.90) ×0.98 (0.98-1.01)

Yes on raw random reads: Debian 10's 4.19 kernel serves 1.2 to 1.7 times the IOPS of 6.1, by about the same amount with the multi-queue block layer (blk-mq) and with the legacy single-queue layer, so the switch to blk-mq alone does not explain the gap. Where a bracket crosses 1.0 (4 KiB and 256 KiB on blk-mq) that size's gain is uncertain. No on the seedbox mix: 4.19 is at ×0.85 on blk-mq and ×0.98 on the legacy layer.

4.19 as a share of 6.1 on the same array, on both block layers. 1.0 = same as 6.1.

Same settings on both kernels

The two kernels boot with different THP and request-cap defaults on this hardware, so a second run of 25 runs per arm gave each kernel the other's: 6.1 with transparent huge pages at madvise and request caps of 4096 KiB on the members and 2048 KiB on the md device, and 7.3 with THP always and 1280 KiB caps. Writeback throttling needs no matching: bfq switches it off on both kernels. A fifth arm ran 7.3 with the member disks on the none scheduler instead of bfq.

Each kernel with the other's settings, and 7.3 on none, as a share of 6.1 with its own defaults. Median of 25 runs per arm.
Median of 25 runs, ratio to 6.1 default 6.1 with 7.3's settings 7.3 default 7.3 with 6.1's settings 7.3 on none
Runs that stalled 25 / 25 0 / 25 0 / 25 0 / 25
random read 4 KiB ×1.01 (0.86-1.19) ×0.56 (0.53-0.59) ×0.70 (0.58-0.74) ×1.32 (1.19-1.40)
random read 64 KiB ×1.01 (0.87-1.15) ×0.59 (0.52-0.71) ×0.55 (0.61-0.62) ×1.25 (1.09-1.38)
random read 256 KiB ×0.90 (0.84-0.97) ×0.61 (0.60-0.67) ×0.81 (0.67-0.84) ×1.32 (1.27-1.34)
random read 1 MiB ×0.92 (0.92-0.95) ×0.53 (0.48-0.58) ×0.66 (0.69-0.83) ×1.27 (1.27-1.29)
random read 4 MiB ×1.36 (1.35-1.35) ×0.52 (0.52-0.53) ×0.84 (0.78-0.91) ×1.65 (1.63-1.65)
seedbox mix, read MB/s ×1.14 (1.00-1.19) ×1.11 (0.97-1.13) ×1.17 (1.00-1.23) ×1.44 (1.23-1.53)
page-cache (mmap) read IOPS ×1.07 (1.06-1.07) ×1.15 (1.11-1.21) ×0.51 (0.50-0.51) ×1.37 (1.27-1.39)

The stall follows the kernel, not the settings: 6.1 stalls in every run with 7.3's settings, and 7.3 stalls in none with 6.1's.

The raw-read gap does not follow the settings either. 6.1 keeps its random-read rate with 7.3's settings, and gains 36% at 4 MiB, the only tested size above 6.1's own 1280 KiB request cap. 7.3 with 6.1's settings stays 16-45% below 6.1. With the none scheduler, 7.3 reads faster than 6.1 on bfq at every size, by 25-65%, and its seedbox-mix read latency at p99 drops by about a third. The cap that caused the stall is enforced above the scheduler, so 7.3 on none still never stalled.

On the seedbox mix the two block pairs disagree for 7.3 on bfq (0.97 and 1.13), so this run shows no clear change there, in line with the ×0.97 of the first run. Giving 7.3 THP always and 1280 KiB caps together halves its page-cache read rate.


bfq or the kernel?

A third run, again 25 runs per arm, put both kernels on bfq and on none, and gave 7.3 each of 6.1's two settings on its own.

Median of 25 runs, ratio to 6.1 on bfq 6.1 on none 7.3 on bfq 7.3 on none
random read 4 KiB ×1.27 (1.13-1.60) ×0.65 (0.55-0.90) ×1.49 (1.47-1.57)
random read 64 KiB ×1.28 (1.04-1.35) ×0.52 (0.50-0.51) ×1.19 (1.11-1.20)
random read 256 KiB ×1.52 (1.48-1.59) ×0.78 (0.70-0.85) ×1.31 (1.39-1.48)
random read 1 MiB ×1.25 (1.20-1.29) ×0.47 (0.42-0.64) ×1.33 (1.33-1.38)
random read 4 MiB ×1.65 (1.60-1.66) ×0.62 (0.54-0.63) ×1.57 (1.15-1.65)
seedbox mix, read MB/s ×1.36 (1.33-1.40) ×0.94 (0.93-0.97) ×1.25 (1.22-1.30)
seedbox mix, p99 read latency (lower is better) ×0.78 ×1.09 ×0.65
Both kernels on bfq and on none, as a share of 6.1 on bfq. Median of 25 runs per arm.

Without bfq the two kernels read within about 20% of each other, ahead at some sizes and behind at others. bfq costs 6.1 20-39% of its raw random-read rate on this array and costs 7.3 40-65%. So most of 7.3's raw-read loss comes from how bfq runs on 7.3; without bfq the remaining gap is under 20% in either direction. This test does not show which change inside bfq is responsible. bfq is the scheduler that divides disk time between processes or accounts by weight; none does no such division, so a shared server that relies on those weights keeps bfq.

The page-cache halving comes from the request cap, not from THP: 7.3 with only the 1280 KiB cap reads ×0.52 (0.47-0.54) of 6.1's page-cache rate, and with only THP always ×1.12 (1.11-1.17). On 6.1 that 1280 KiB cap is the default, so the halving is specific to 7.3. On the seedbox mix 7.3 on bfq sits between ×0.94 and ×1.11 of 6.1 across the three runs.

A benchmarking trap

fio's mmap engine sets MADV_RANDOM on a random job by default (fadvise_hint), which turns read-around off, so every page fault reads 4 KiB. On 6.1, 25 runs with the default gave a median of 6.6 IOPS at 17 KiB per disk request; with --fadvise_hint=0 the same job ran at a median 30.1 IOPS over 25 runs, at 495 KiB per disk request. Page-cache read benchmarks with fio need that flag.

At Pulsed Media

Pulsed Media seedboxes keep many accounts on one hard-drive RAID array, which is why this page measures a capped account next to an uncapped neighbour instead of one process alone. The 7.3 kernel on this page was measured on a dedicated hard-drive test server with no customer accounts before it ran on any seedbox server. The same harness, scenarios and ABBA block order are used for every kernel compared here, so a later kernel can be measured against these numbers.

Pulsed Media runs its seedboxes on hard-drive RAID arrays in datacenters it operates in Finland. See the seedbox plans.

See also