Post Mortem:
The server I run Flotilla, zooid, and other things on got hacked this morning at about 5 AM Pacific. I had disabled the gitea action runner for pull requests, but not for users' own repositories. The attacker triggered two workflow runs (called "exploit" and "probe") in an action runner container which had the docker socket mounted.
At about 8 AM this morning I purged the attacker's two accounts (which was a mistake because it destroyed forensics), and gave Claude ssh access and asked it to clean things up. Gitea has been moved to a dedicated host with the action runner container configuration fixed and registrations disabled. The rest of the services have been manually moved to a fresh host.
There's no evidence that the attacker exfiltrated any sensitive data, installed any software, or executed any supply chain attacks (no packages were published, no updates released, and I keep signing keys elsewhere). This appears to have been an automated attack targeting poorly-secured gitea instances.
I'm treating the server as completely compromised, and have rotated all credentials and moved everything to new servers.
The main things for you to know:
- If you host a relay with me, it's possible the attacker had access to your space's chat messages.
- Hosted relays' secret keys were compromised. These were never shared with tenants, have been rotated, and never published any metadata. They were only used to map self-published events to a relay url.
This attack has made me reflect on the security characteristics of two things in particular:
- Nostr private keys can't be rotated. I stored low-value keys on disk rather than remotely, let alone using multisig. This is a very different threat model than people are used to. I am going to look into a developer-oriented way to store private keys for production use.
- NWC connection strings can't revoke themselves. One of my users had entered their connection string into caravel to enable automatic payments. I've contacted them and encouraged them to revoke the connection, but it's not something I can do myself. I am going to look into adding a self-revoke protocol to NWC (and NIP 46).
The server I run Flotilla, zooid, and other things on got hacked this morning at about 5 AM Pacific. I had disabled the gitea action runner for pull requests, but not for users' own repositories. The attacker triggered two workflow runs (called "exploit" and "probe") in an action runner container which had the docker socket mounted.
At about 8 AM this morning I purged the attacker's two accounts (which was a mistake because it destroyed forensics), and gave Claude ssh access and asked it to clean things up. Gitea has been moved to a dedicated host with the action runner container configuration fixed and registrations disabled. The rest of the services have been manually moved to a fresh host.
There's no evidence that the attacker exfiltrated any sensitive data, installed any software, or executed any supply chain attacks (no packages were published, no updates released, and I keep signing keys elsewhere). This appears to have been an automated attack targeting poorly-secured gitea instances.
I'm treating the server as completely compromised, and have rotated all credentials and moved everything to new servers.
The main things for you to know:
- If you host a relay with me, it's possible the attacker had access to your space's chat messages.
- Hosted relays' secret keys were compromised. These were never shared with tenants, have been rotated, and never published any metadata. They were only used to map self-published events to a relay url.
This attack has made me reflect on the security characteristics of two things in particular:
- Nostr private keys can't be rotated. I stored low-value keys on disk rather than remotely, let alone using multisig. This is a very different threat model than people are used to. I am going to look into a developer-oriented way to store private keys for production use.
- NWC connection strings can't revoke themselves. One of my users had entered their connection string into caravel to enable automatic payments. I've contacted them and encouraged them to revoke the connection, but it's not something I can do myself. I am going to look into adding a self-revoke protocol to NWC (and NIP 46).
1711❤️6👀3💜2🤙2👍1🔥1
