zfs - single disk setup: Where is the missing space? [1TB (931,5 GiB) disk: 32,5 GiB missing as usable space]

Short summary:

When creating a single zfs disk pool consisting of only one disk with a capacity of 1TB (= 931GiB) the file system only showed 899 GiB free space (df -h or zfs list; zpool list actually showed the partition size (931 GiB) minus some overhead (resulting in 928 GiB space).

Longer version:

I was trying to setup a zfs disk pool consisting of only one disk with a capacity of 1TB (= 931,53 GiB):

# fdisk -l /dev/sdb
Disk /dev/sdb: 931.53 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: xxxx
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: xxx
Device Start End Sectors Size Type
/dev/sdb1 2048 1953516976 1953514929 931.5G Linux filesystem

When setting up a zfs pool

# zpool create -f -o ashift=12 tank /dev/sdb1;

there are 32,5 GiB missing:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 360K 899G 96K /tank

What is causing a 32,5 GiB overhead on a single disk setup?

zpool list is reporting:

# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 928G 444K 928G - - 0% 0% 1.00x ONLINE -

but this isn't the actually useable space since also df -h is reporting:

# df -h
Filesystem Size Used Avail Use% Mounted on
tank 899G 128K 899G 1% /tank

There is no quota or reservation set:

# zfs get quota
NAME PROPERTY VALUE SOURCE
tank quota none default
# zfs get reservation
NAME PROPERTY VALUE SOURCE
tank reservation none default
# zfs get refquota
NAME PROPERTY VALUE SOURCE
tank refquota none default
# zfs get refreservation
NAME PROPERTY VALUE SOURCE
tank refreservation none default
# zfs get usedbyrefreservation
NAME PROPERTY VALUE SOURCE
tank usedbyrefreservation 0B -

Creating the zpool with ashift=9 doesn't make any difference.

I can live with an actual overhead of 3,5 GiB (= partition size vs. zpool reported size), but not with an overhead of 32,5 GiB or 29 GiB (= zfs reported size - zpool reported size) on such a small disk.

When trying the same with btrfs I get more free space:

# btrfs filesystem show
Label: 'tank' uuid: xxx Total devices 1 FS bytes used 128.00KiB devid 1 size 931.51GiB used 2.02GiB path /dev/sdb1
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 932G 3.8M 930G 1% /mnt

more details

# zfs --version
zfs-0.8.3-1ubuntu12.5
zfs-kmod-0.8.3-1ubuntu12.5
# uname -a
Linux xxx 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/os-release | grep VERSION
VERSION="20.04.1 LTS (Focal Fossa)"
VERSION_ID="20.04"
VERSION_CODENAME=focal
# zfs list -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
tank 899G 88.5K 0B 24K 0B 64.5K

Update: re-created with the command zpool create -oashift=12 tank /dev/sdb1.

No difference:

# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 928G 432K 928G - - 0% 0% 1.00x ONLINE -
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 336K 899G 96K /tank

Output of zdb tank | grep metaslab | tail -n 3:

# zdb tank | grep metaslab | tail -n 3
loading concrete vdev 0, metaslab 115 of 116 ... metaslab 114 offset e400000000 spacemap 0 free 8G metaslab 115 offset e600000000 spacemap 0 free 8G vdev 0 metaslabs 116 fragmentation 0%

Output of zdb | grep metaslab_shift:

# zdb | grep metaslab_shift metaslab_shift: 33

2 Answers

I cannot reproduce your problem.

root@banshee:/tmp# truncate -s 931G disk.bin
root@banshee:/tmp# zpool create -oashift=12 test /tmp/disk.bin
root@banshee:/tmp# zpool list test
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
test 928G 480K 928G - - 0% 0% 1.00x ONLINE -

In the above example, I begin with a 931GiB (1TB, roughly) block device, and create a pool on it using 4KiB sectors. The available capacity is 928GiB, as expected when you account for metaslab remainder.

root@banshee:/tmp# zdb test | grep metaslab | tail -n 3
loading concrete vdev 0, metaslab 115 of 116 ... metaslab 114 offset e400000000 spacemap 0 free 8G metaslab 115 offset e600000000 spacemap 0 free 8G vdev 0 metaslabs 116 fragmentation 0%

My 931GiB "disk" was divided into 116 8GiB metaslabs; this left a 0.44125 metaslab remainder.

0.44125 metaslabs * 8GiB/metaslab == 3.53GiB
931GiB disk - 3.53Gib metaslab remainder == ~~928GiB usable

... and Bob's your uncle. Why you are seeing roughly ten times that amount of overhead, I have no idea; I'm also on Focal, with the same ZFS version as you're reporting.

root@banshee:/tmp# apt policy zfsutils-linux
zfsutils-linux: Installed: 0.8.3-1ubuntu12.5 Candidate: 0.8.3-1ubuntu12.5 Version table: *** 0.8.3-1ubuntu12.5 500 500 focal-updates/main amd64 Packages 100 /var/lib/dpkg/status 0.8.3-1ubuntu12 500 500 focal/main amd64 Packages

It might help to see the actual creation of your pool itself command by command from scratch and/or the output of zdb | grep metaslab_shift on that pool.

As pointed out by the user grenkins in the subreddit /r/zfs ( - all credit goes to him) zfs is reserving 3,2% space by default. This reservation is done by the kernel module on the system and has nothing to do with the quota/reservation/refquota/refreservation/usedbyrefreservation zpool settings. See

The reservation is done by the spa_slop_shift setting of the zfs kernel module.

Summary:

Normally, the last 3.2% (1/(2^spa_slop_shift)) of pool space is reserved to ensure the pool doesn’t run completely out of space [...]
For large pools, when 3.2% may be too conservative and more usable space is desired, consider increasing spa_slop_shift

I increased the spa_slop_shift setting to 15 by creating the following file:

# printf "options zfs spa_slop_shift=15" > /etc/modprobe.d/zfs.conf

After a reboot the are 928 GiB available:

# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 928G 576K 928G - - 0% 0% 1.00x ONLINE -
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 408K 928G 96K /tank

which is actually the free space zpool list was reporting (931,5 partition size - some small overhead).

So this is fixed for me.