The shop keeper is just prejudiced. Charlie’s four grandparents are in a long-term polycule together. Charlie is the result of an insane two-generation long plan to get all members of the polycule to have a single genetic child together. The polycule has been completely open to the whole town about this plan, and everyone knows it.

The shopkeeper isn’t prejudiced against the poor. He just has sexual hangups. And he’s projecting his bigotry onto an innocent young child that had no choice but to be part of this whole situation.