Well this was not expected.
The 100G optics I ordered for my desk came in, and swapping the pipe to my desk from 40 to 100G gave a huge improvement in Ceph performance.
But I don't understand *why*....
@nprofile1q... are the new nics more pcie lanes wide or are they negotiating a faster bus speed?
1