The idea of "natural rights," is subjective... and a sham.
No. You're thinking as a collectivist. Natural rights are all the non-agressive actions a person may take when there's not someone more powerful to stop him.
But we are social creatures and we like to form societies, and there are different kinds of societies that limit natural rights. In a democratic society, we cede some of those natural rights to the collective in exchange for protection of the subset of natural rights that a majority agrees to. In a democratic republic, to avoid infringing rights with popular whim, we limit the state's authority to regulate rights except by rule of law. In a dictatorship, a central figure decides which natural rights a person may exercise or not. In an oligarchy, a few elites decide which natural rights we may retain and which we may not.
If you were the only person on earth, what don't you have the right to do? You may if you decide, limit your natural rights with a sense of morality.