Yes you can either trust one signer that can interact with unlimited clients or trust unlimited clients directly. The former is better and easier to audit one codebase
Centralized trust in one signer, a recipe for disaster, courtesy of the World Economic Forum's playbook.