Voters didnt go out because the only human in the fellowship group was a paladin. Sauron proceedes to own the humans for the next few centuries.