You just pointed a finger at the reason anarchism doesn't work. It can't work if there are bad actors, within or outside. And there are always bad actors.

You can have a rights respecting society with a government. If that government is focused on preserving rights. Or in other words is focused on preventing people, from within or form outside, from taking others rights.

So what do we have in this government?

Army
Secretary of state
Courts
Police
Treasury

Now the question is how do you finance this government without force. That is possible and I can go into it if you want.