Damus
Open in Damus
Andrew Zonenberg profile picture
Andrew Zonenberg
@Andrew Zonenberg

Security and open source at the hardware/software interface. Embedded sec @ IOActive. Lead dev of ngscopeclient/libscopehal. GHz probe designer. Open source networking hardware. "So others may live"

Toots searchable on tootfinder.

npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpdnpub
[email protected]nip05
Relays (1)
  • wss://relay.ditto.pub – read & write

Recent Notes

Manawyrm | Sarah · 5d
nostr:nprofile1qy2hwumn8ghj7un9d3shjtnyd968gmewwp6kyqpqqv5atqz9k9c54q8c28kra6sfata0wk7w7x5gkrnde8vmxe5gt00q8mmv7s nostr:nprofile1qy2hwumn8ghj7un9d3shjtnyd968gmewwp6kyqpqhsdpcz27r5nfnw5xy72yenpyuxwe0h0pqzsaevj9sfra80jd6tfqfurues my usual rule of thumb is: "don't use anything other than mellanox for a...
Andrew Zonenberg profile picture
Well this was not expected.

The 100G optics I ordered for my desk came in, and swapping the pipe to my desk from 40 to 100G gave a huge improvement in Ceph performance.

But I don't understand *why*.

Baseline config from earlier in the year: client on 40G, cluster nodes on dual 10G, 1558 MB/s on linear reads

Moving cluster nodes to 40G, client on 40G: 1864 MB/s

Moving client to 100G, cluster nodes on 40G: 3787 MB/s.

The confusing thing is, even 3787 MB/s is only about 30 Gbps, so after protocol overhead I would expect it to fit comfortably in 40G. Why can I get this performance with the client on 100G, but not on 40?

1
✧✦Catherine✦✧ · 5d
nostr:nprofile1qy2hwumn8ghj7un9d3shjtnyd968gmewwp6kyqpqqv5atqz9k9c54q8c28kra6sfata0wk7w7x5gkrnde8vmxe5gt00q8mmv7s latency?
Andrew Zonenberg · 6d
Hmm, I misremembered the hardware config. I thought all the Ceph nodes were 6x 8GB. alexandria-1 had 6x 8GB sticks (48 GB) now has 6x 16GB (96 GB). But alexandria-2 had 12x 4GB stick (48 GB) now has...
 Andrew Zonenberg profile picture
Andrew Zonenberg @Andrew Zonenberg
It seems like the best I can probably do with the current hardware is to put 4x 16GB in there, bumping capacity from 48 to 64 GB, but with two channels empty. That will mean a slight degradation in RAM performance from 6 to 4 channel.

Then I'll put all of the 8 and 4GB sticks in the giveaway pile, and maybe see if I can eBay two more 16GB sticks at some point to get all of the cluster nodes back to a balanced configuration.
Andrew Zonenberg · 6d
Second order hardware upgrade time! Trickle down may not work in economics but it's very real in the hardware world. All of those 16GB DDR4 DIMMs I pulled from my office desktop are going into the st...
 Andrew Zonenberg profile picture
Andrew Zonenberg @Andrew Zonenberg
Hmm, I misremembered the hardware config. I thought all the Ceph nodes were 6x 8GB.

alexandria-1 had 6x 8GB sticks (48 GB) now has 6x 16GB (96 GB).

But alexandria-2 had 12x 4GB stick (48 GB) now has 6x 16GB (96 GB).

Alexandria-3 currently has 12x 4GB too.

I have 4x 16GB, 6x 8GB, 12x 4GB sticks out on the bench. Wut do?
1
Andrew Zonenberg · 6d
It seems like the best I can probably do with the current hardware is to put 4x 16GB in there, bumping capacity from 48 to 64 GB, but with two channels empty. That will mean a slight degradation in RAM performance from 6 to 4 channel. Then I'll put all of the 8 and 4GB sticks in the giveaway pile, ...
Andrew Zonenberg profile picture
Second order hardware upgrade time! Trickle down may not work in economics but it's very real in the hardware world.

All of those 16GB DDR4 DIMMs I pulled from my office desktop are going into the storage cluster to replace the 8GB sticks currently in those nodes.

1
Andrew Zonenberg · 6d
Hmm, I misremembered the hardware config. I thought all the Ceph nodes were 6x 8GB. alexandria-1 had 6x 8GB sticks (48 GB) now has 6x 16GB (96 GB). But alexandria-2 had 12x 4GB stick (48 GB) now has 6x 16GB (96 GB). Alexandria-3 currently has 12x 4GB too. I have 4x 16GB, 6x 8GB, 12x 4GB sticks o...
🏳️‍⚧️ Chloé: ADhdSL 512Kb/s · 6d
nostr:nprofile1qy2hwumn8ghj7un9d3shjtnyd968gmewwp6kyqpqqv5atqz9k9c54q8c28kra6sfata0wk7w7x5gkrnde8vmxe5gt00q8mmv7s Just a quick question (that other ngscopeclient builders can have), I just cloned ng...
 Andrew Zonenberg profile picture
Andrew Zonenberg @Andrew Zonenberg
@nprofile1q... The error bounds for the tests cases are somewhat arbitrary and i'm still collecting data on how big an error bound we want to allow. It should be safe to ignore for the near term but file a ticket against ngscopeclient/scopehal so we have a record of this and can revisit
1
🏳️‍⚧️ Chloé: ADhdSL 512Kb/s · 6d
nostr:nprofile1qy2hwumn8ghj7un9d3shjtnyd968gmewwp6kyqpqqv5atqz9k9c54q8c28kra6sfata0wk7w7x5gkrnde8vmxe5gt00q8mmv7s Do you need some more insights for the ticket?
Andrew Zonenberg · 6d
Everything is set up but for reasons I don't currently understand the fan on *one* of the CPUs is running at what sounds like max RPM for no reason. I can't figure out why, seeing as both CPUs are sit...
 Andrew Zonenberg profile picture
Andrew Zonenberg @Andrew Zonenberg
But it's all set up and working great. A full build of ngscopeclient from an empty checkout took only 185 sec, down significantly from the 294 it took on the dual 6144s, but not a huge speedup over the 200 sec it took on my lab box with a single 8362.

I'm probably hitting Amdahl's law here, there's not enough source files for a "make -j128" to be much of a benefit over -j64.

Andrew Zonenberg · 6d
Had an initial post failure but that seems to have been cleared by either jiggling cards or swapping the bios battery (unsure which fixed it i had to move cards to reach said battery). Guess that's t...
 Andrew Zonenberg profile picture
Andrew Zonenberg @Andrew Zonenberg
Everything is set up but for reasons I don't currently understand the fan on *one* of the CPUs is running at what sounds like max RPM for no reason. I can't figure out why, seeing as both CPUs are sitting at like 25C. Guess I've got some digging to do.
1
Andrew Zonenberg · 6d
But it's all set up and working great. A full build of ngscopeclient from an empty checkout took only 185 sec, down significantly from the 294 it took on the dual 6144s, but not a huge speedup over the 200 sec it took on my lab box with a single 8362. I'm probably hitting Amdahl's law here, there's...
Andrew Zonenberg · 6d
Ports you don't think about until you look and they're not there: 3.5mm audio lol. Luckily i have a usb dongle https://files.ioc.exchange/iocexchange/media_attachments/files/116/481/153/853/826/530/o...
 Andrew Zonenberg profile picture
Andrew Zonenberg @Andrew Zonenberg
Had an initial post failure but that seems to have been cleared by either jiggling cards or swapping the bios battery (unsure which fixed it i had to move cards to reach said battery).

Guess that's the risk you take buying NOS boards thst have been sitting around for a while.

Anyway it's booted and now i need to remove the Nvidia driver and set up the new gpu
1
Andrew Zonenberg · 6d
Everything is set up but for reasons I don't currently understand the fan on *one* of the CPUs is running at what sounds like max RPM for no reason. I can't figure out why, seeing as both CPUs are sitting at like 25C. Guess I've got some digging to do.