Well this was not expected.
The 100G optics I ordered for my desk came in, and swapping the pipe to my desk from 40 to 100G gave a huge improvement in Ceph performance.
But I don't understand *why*.
Baseline config from earlier in the year: client on 40G, cluster nodes on dual 10G, 1558 MB/s on linear reads
Moving cluster nodes to 40G, client on 40G: 1864 MB/s
Moving client to 100G, cluster nodes on 40G: 3787 MB/s.
The confusing thing is, even 3787 MB/s is only about 30 Gbps, so after protocol overhead I would expect it to fit comfortably in 40G. Why can I get this performance with the client on 100G, but not on 40?
The 100G optics I ordered for my desk came in, and swapping the pipe to my desk from 40 to 100G gave a huge improvement in Ceph performance.
But I don't understand *why*.
Baseline config from earlier in the year: client on 40G, cluster nodes on dual 10G, 1558 MB/s on linear reads
Moving cluster nodes to 40G, client on 40G: 1864 MB/s
Moving client to 100G, cluster nodes on 40G: 3787 MB/s.
The confusing thing is, even 3787 MB/s is only about 30 Gbps, so after protocol overhead I would expect it to fit comfortably in 40G. Why can I get this performance with the client on 100G, but not on 40?
1
