I even have my phone upside down in my pocket so the charging port is facing up.

This is probably the difference, people who put their phone in with the port facing down in their pocket have lots of opportunity to pickup lint as the phone is going into the pocket on the way down, and then also as you’re moving, any lint collected in the bottom of your pocket can enter the port. Unless you have super baggy pants, debris won’t be randomly entering your pocket from the top and getting into the charging port if you have the phone upside down with the charging port facing up.