System has 128 GB RAM, but 80 GB is Used Immediately After Reboot

My desktop system has 128 GB of RAM, which I expect most to be free after booting Ubuntu. However, free -m shows that 80 GB is in use, with only 40 GB free. Note that buff/cache is only under 1 GB, so this is not the cause of the high memory usage.

 total used free shared buff/cache available
Mem: 125G 80G 44G 20M 825M 42G
Swap: 8G 0B 8G

Output of ps aux | awk '{sum+=$6} END {print sum / 1024}'

1368.03

Based on another SO question, this means that my processes only use 1GB of memory but the system as a whole (excluding cache) uses 79 GB.

Question: What is really eating almost 80 GB of my memory? Launching a process that takes up around 40 GB will cause the system to crash, the 80 GB used initially on startup does not get released!

Any insights greatly appreciated!

Output of ps -A --sort -rss -o comm,pmem | head -n 5

COMMAND %MEM
mysqld 0.2
blueman-applet 0.0
caja 0.0
nm-applet 0.0

Output of cat /proc/meminfo

MemTotal: 131947364 kB
MemFree: 45513984 kB
MemAvailable: 45083132 kB
Buffers: 24900 kB
Cached: 299012 kB
SwapCached: 0 kB
Active: 1337380 kB
Inactive: 211000 kB
Active(anon): 1228248 kB
Inactive(anon): 20812 kB
Active(file): 109132 kB
Inactive(file): 190188 kB
Unevictable: 6864 kB
Mlocked: 6900 kB
SwapTotal: 34553848 kB
SwapFree: 34553848 kB
Dirty: 16 kB
Writeback: 0 kB
AnonPages: 1231920 kB
Mapped: 206100 kB
Shmem: 20896 kB
Slab: 521948 kB
SReclaimable: 77492 kB
SUnreclaim: 444456 kB
KernelStack: 20880 kB
PageTables: 23056 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 58584488 kB
Committed_AS: 4713068 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 40960
HugePages_Free: 40960
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 390128 kB
DirectMap2M: 8921088 kB
DirectMap1G: 124780544 kB

Output of sudo lshw -C memory

 *-firmware description: BIOS vendor: American Megatrends Inc. physical id: 0 version: P3.50 date: 12/24/2018 size: 64KiB capacity: 15MiB capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer acpi usb biosbootspecification uefi *-memory description: System Memory physical id: 8 slot: System board or motherboard size: 128GiB *-bank:0 description: DIMM Synchronous 2134 MHz (0.5 ns) product: CMK128GX4M8X3800C19 vendor: Unknown physical id: 0 serial: 00000000 slot: DIMM 0 size: 16GiB width: 64 bits clock: 2134MHz (0.5ns) *-bank:1 description: DIMM Synchronous 2134 MHz (0.5 ns) product: CMK128GX4M8X3800C19 vendor: Unknown physical id: 1 serial: 00000000 slot: DIMM 1 size: 16GiB width: 64 bits clock: 2134MHz (0.5ns) *-bank:2 description: DIMM Synchronous 2134 MHz (0.5 ns) product: CMK128GX4M8X3800C19 vendor: Unknown physical id: 2 serial: 00000000 slot: DIMM 0 size: 16GiB width: 64 bits clock: 2134MHz (0.5ns) *-bank:3 description: DIMM Synchronous 2134 MHz (0.5 ns) product: CMK128GX4M8X3800C19 vendor: Unknown physical id: 3 serial: 00000000 slot: DIMM 1 size: 16GiB width: 64 bits clock: 2134MHz (0.5ns) *-bank:4 description: DIMM Synchronous 2134 MHz (0.5 ns) product: CMK128GX4M8X3800C19 vendor: Unknown physical id: 4 serial: 00000000 slot: DIMM 0 size: 16GiB width: 64 bits clock: 2134MHz (0.5ns) *-bank:5 description: DIMM Synchronous 2134 MHz (0.5 ns) product: CMK128GX4M8X3800C19 vendor: Unknown physical id: 5 serial: 00000000 slot: DIMM 1 size: 16GiB width: 64 bits clock: 2134MHz (0.5ns) *-bank:6 description: DIMM Synchronous 2134 MHz (0.5 ns) product: CMK128GX4M8X3800C19 vendor: Unknown physical id: 6 serial: 00000000 slot: DIMM 0 size: 16GiB width: 64 bits clock: 2134MHz (0.5ns) *-bank:7 description: DIMM Synchronous 2134 MHz (0.5 ns) product: CMK128GX4M8X3800C19 vendor: Unknown physical id: 7 serial: 00000000 slot: DIMM 1 size: 16GiB width: 64 bits clock: 2134MHz (0.5ns) *-cache:0 description: L1 cache physical id: b slot: L1 - Cache size: 3MiB capacity: 3MiB clock: 1GHz (1.0ns) capabilities: pipeline-burst internal write-back unified configuration: level=1 *-cache:1 description: L2 cache physical id: c slot: L2 - Cache size: 16MiB capacity: 16MiB clock: 1GHz (1.0ns) capabilities: pipeline-burst internal write-back unified configuration: level=2 *-cache:2 description: L3 cache physical id: d slot: L3 - Cache size: 64MiB capacity: 64MiB clock: 1GHz (1.0ns) capabilities: pipeline-burst internal write-back unified configuration: level=3

Output of sudo grep Memory /var/log/kern.log | grep reserved

Jan 14 16:53:53 server2 kernel: [ 0.000000] Memory: 131807432K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283932K reserved, 0K cma-reserved)
Jan 14 17:47:08 server2 kernel: [ 0.000000] Memory: 131807432K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283932K reserved, 0K cma-reserved)
Jan 14 17:52:04 server2 kernel: [ 0.000000] Memory: 131807436K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283928K reserved, 0K cma-reserved)
Jan 14 18:09:33 server2 kernel: [ 0.000000] Memory: 131807436K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283928K reserved, 0K cma-reserved)
Jan 14 18:25:01 server2 kernel: [ 0.000000] Memory: 131807436K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283928K reserved, 0K cma-reserved)
Jan 14 18:43:40 server2 kernel: [ 0.000000] Memory: 131820396K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2270968K reserved, 0K cma-reserved)
Jan 14 18:48:07 server2 kernel: [ 0.000000] Memory: 131807428K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283936K reserved, 0K cma-reserved)

1 Answer

Partial answer...

CPU: AMD Threadripper 2990WX
Motherboard: ASRock X399 Professional Gaming sTR4
128GB of RAM

1. Update your motherboard BIOS

   • You're currently at BIOS version P3.50
   • BIOS 3.80, from 1/8/2020, can be downloaded here
   • Confirm that this is the correct web page for your model #
   • Backup important data before installing new BIOS
   • view the free -h command to confirm memory usage

2. Go to and download/run their free memtest to test your memory. Get at least one complete pass of all the 4/4 tests to confirm good memory. This may take many hours to complete.

3. Using proper ESD and power procedures, re-seat all RAM dimms into their sockets. (Wait with doing this until I tell you more about this.)