Improve mdadm RAID5 with SSD Cache

I have the following Setup: 3x 3 TB SATA HDD With software raid connected to a RAID5 On top of this LVM for snapshots (backup) And on top of LVM EXT4 formatted.

Actually around 400GB is used. I have one spare slot within the server.

The write performance of the webserver is quite slow for smaller files. But good/ok for larger ones

root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.93049 s, 120 MB/s
root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 25.2487 s, 42.5 MB/s
root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.0472 s, 97.2 MB/s
root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=100M count=1 oflag=direct
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 2.97071 s, 35.3 MB/s
root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=100M count=10 oflag=direct
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 15.7935 s, 66.4 MB/s
root@srv1:/home/srvadmin# dd if=/dev/zero of=/root/testfile bs=100M count=20 oflag=direct

Now i see different options: A: add bcache (don‘t know how) and use an SSD as cache B: Buy an additional disk and create a raid10 C: Buy an SSD, copy all files to it and use the ssd alone without any raid D: Buy two SSDs and use them mirrored

I guess solution A is the cheapest. I already have a 120GB SSD laying around and the filesystem is in place. But i dont know how to add the SSD for caching. Have you ever did something like this?

The Server is private used, running Ubuntu LTS, the server is also backuped regularely.

Thanks

Edit:

I found this:

It looks like that lvm offers caching. But im not sure whether i can add caching at this stage i am now.

Edit2:

I guess, i made it! I followed these two tutorials:

And ended up with these commands: Please check, if the outputs looks feasible: I have used cache type writethrough instead of writeback, cause i have read that writethrough is the more power failure safe one.

root@srv1:~# fdisk /dev/sdd
Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/sdd: 111.81 GiB, 120034123776 bytes, 234441648 sectors
Disk model: KINGSTON SA400S3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2BAB7835-0A17-43DE-83E1-57A2733E4CFD
Command (m for help): n
Partition number (1-128, default 1): 1
First sector (34-234441614, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-234441614, default 234441614):
Created a new partition 1 of type 'Linux filesystem' and of size 111.8 GiB.
Partition #1 contains a vfat signature.
Do you want to remove the signature? [Y]es/[N]o: y
The signature will be removed by a write command.
Command (m for help): p
Disk /dev/sdd: 111.81 GiB, 120034123776 bytes, 234441648 sectors
Disk model: KINGSTON SA400S3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2BAB7835-0A17-43DE-83E1-57A2733E4CFD
Device Start End Sectors Size Type
/dev/sdd1 2048 234441614 234439567 111.8G Linux filesystem
Filesystem/RAID signature on partition 1 will be wiped.
Command (m for help): t
Selected partition 1
Partition type (type L to list all types): 8e
Type of partition 1 is unchanged: Linux filesystem.
Command (m for help): p
Disk /dev/sdd: 111.81 GiB, 120034123776 bytes, 234441648 sectors
Disk model: KINGSTON SA400S3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2BAB7835-0A17-43DE-83E1-57A2733E4CFD
Device Start End Sectors Size Type
/dev/sdd1 2048 234441614 234439567 111.8G Linux filesystem
Filesystem/RAID signature on partition 1 will be wiped.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
root@srv1:~# pvcreate /dev/sdd1 Physical volume "/dev/sdd1" successfully created.
root@srv1:~# vgextend vg /dev/sdd1 Volume group "vg" successfully extended
root@srv1:~# lvcreate -n DataLVcache -L100G vg /dev/sdd1 Logical volume "DataLVcache" created.
root@srv1:~# lvcreate -n DataLVcacheMeta -L100M vg /dev/sdd1 Logical volume "DataLVcacheMeta" created.
root@srv1:~# lvconvert --type cache-pool --cachemode writethrough --poolmetadata vg/DataLVcacheMeta vg/DataLVcache Using 128.00 KiB chunk size instead of default 64.00 KiB, so cache pool has less than 1000000 chunks. WARNING: Converting vg/DataLVcache and vg/DataLVcacheMeta to cache pool's data and metadata volumes with metadata wiping. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert vg/DataLVcache and vg/DataLVcacheMeta? [y/n]: y Converted vg/DataLVcache and vg/DataLVcacheMeta to cache pool.
root@srv1:~# lvconvert --type cache --cachepool vg/DataLVcache vg/md128 Failed to find logical volume "vg/md128"
root@srv1:~# lvconvert --type cache --cachepool vg/DataLVcache vg/root
Do you want wipe existing metadata of cache pool vg/DataLVcache? [y/n]: y Logical volume vg/root is now cached.
root@srv1:~# lvs --all --options +devices vg LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [DataLVcache_cpool] vg Cwi---C--- 100.00g 0.61 6.47 0.00 [DataLVcache_cpool_cdata] vg Cwi-ao---- 100.00g [DataLVcache_cpool_cmeta] vg ewi-ao---- 100.00m data_snap_backblaze vg swi-a-s--- 2.28t root 2.99 [lvol0_pmspare] vg ewi------- 100.00m root vg owi-aoC--- 2.27t [DataLVcache_cpool] [root_corig] 0.61 6.47 0.00 [root_corig] vg owi-aoC--- 2.27t

If i now try to write to the disk, i get up to 2GB/s

root@srv1:~# dd if=/dev/zero of=/home/user12/testfile2 bs=100M count=2
2+0 records in
2+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.115231 s, 1.8 GB/s

I guess caching is working? If i execute the exact same command again, the performance is very poor, around 50MB/s. I guess this is because the file gets written to the RAID5 array, while i try to write to the file right?

Edit3:

Ok, something is going weird... Everytime i write the same file it gets more slow....

104857600 bytes (105 MB, 100 MiB) copied, 0.0622744 s, 1.7 GB/s
root@srv1:~# dd if=/dev/zero of=/home/user12/testfile1 bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.871399 s, 120 MB/s
root@srv1:~# dd if=/dev/zero of=/home/user12/testfile1 bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 4.3185 s, 24.3 MB/s
root@srv1:~# dd if=/dev/zero of=/home/user12/testfile1 bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 5.31411 s, 19.7 MB/s
root@srv1:~# dd if=/dev/zero of=/home/user12/testfile1 bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 8.95197 s, 11.7 MB/s
root@srv1:~#

Last one after waiting ~2mins

root@srv1:~# dd if=/dev/zero of=/home/user12/testfile1 bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 5.96534 s, 17.6 MB/s

Any clue about that? I waited around 20-50 seconds between executing the commands.

Without caching:

root@srv1:~# dd if=/dev/zero of=/home/user12/testfile2 bs=100M count=1 oflag=direct
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 11.1603 s, 9.4 MB/s

Know someone who can answer? Share a link to this question via email, Twitter, or Facebook.