NVIDIA Kernel Driver in use on only 4 of 6 GPUs

I have 6 GPUs attached via PCI slots on a machine. They are all recognized being recognized:

$ lspci -v | grep 'VGA'
01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
02:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
04:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
05:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
07:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
08:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])

However, the NVIDIA kernel driver is only being used on 4 of them:

$ lspci -v | grep -A 10 'VGA' | grep 'Kernel driver in use:'
Kernel driver in use: nvidia
Kernel driver in use: nvidia
Kernel driver in use: nvidia
Kernel driver in use: nvidia

On the other 2, no kernel driver is even listed:

$ lspci -v | ...
07:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GP104 [GeForce GTX 1070]
Flags: fast devsel, IRQ 18
Memory at d6000000 (32-bit, non-prefetchable) [size=16M]
Memory at <ignored> (64-bit, prefetchable)
Memory at <ignored> (64-bit, prefetchable)
I/O ports at a000 [size=128]
Expansion ROM at d7000000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

A full listing for a GPU that has a kernel driver listed is presented for comparison:

05:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GP104 [GeForce GTX 1070]
Flags: bus master, fast devsel, latency 0, IRQ 325
Memory at d8000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at b8000000 (64-bit, prefetchable) [size=32M]
I/O ports at b000 [size=128]
[virtual] Expansion ROM at d9000000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

The 5th and 6th gpus are not recognized in nvidia-settings, either.

How can I get the 5th and 6th GPUs to start using the NVIDIA kernel driver?

Using: Ubuntu 17.04, NVIDIA driver version 375.66.

1 Answer

Bios update fixed it for me. Appears the system had an issue allocating address space. Look at dmesg output for NVIDIA see if there are any issues there on boot.