A person only has the rights they can hold on to.
— Kumo Crew

I feel like this quote from Elite: Dangerous sums up well, the lower bound.

In the comic, the other animals that freely decided to abide by the non-aggression principle lacked a way to ensure compliance by those, outside of the contract.

Not amassing power is a good way to prevent internal threats, but in case of external threats, which will amass power, this makes it too weak.