How to achieve fast transfers over 10Gbe

I'm trying to transfer files over 10 gigabit ethernet. However the maximum speed I can achieve between 2 RAM-Disks (mounted on /mnt on each node) is around 200MB/s:

root@odin$ scp /mnt/largeFile marc@10.0.0.12:/mnt
marc@10.0.0.12's password:
largeFile 86% 6904MB 192.4MB/s 00:05 ETA

When running Iperf I can achieve 9.89 Gb/s:

root@odin$ iperf -c 10.0.0.12
------------------------------------------------------------
Client connecting to 10.0.0.12, TCP port 5001
TCP window size: 325 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.11 port 59028 connected with 10.0.0.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 11.5 GBytes 9.89 Gbits/sec

The RAM-Disks achieve write speeds of 1.2 GB/s.

The network config on the 2 machines:

Server:

enp1s0 Link encap:Ethernet HWaddr 00:02:c9:4d:a3:12 inet addr:10.0.0.12 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::202:c9ff:fe4d:a312/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1

Client:

ens1 Link encap:Ethernet HWaddr 00:02:c9:4f:98:14 inet addr:10.0.0.11 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::202:c9ff:fe4f:9814/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1

I'm wondering how I can achieve speeds similar to the iperf ones for file Transfers?

1 Answer

It's due to the transfer limits caused by various aspects of scp/OpenSSH.

If you look at your destination machines resources while doing lperf and scp tests you should notice that even at 5x the speed the CPU will be under a lower load with lperf than with scp , this solely due to encryption.

There is a great set of resources and patches available from the Pittsburgh Supercomputing Center,