Maybe you did pass too close to the womenfolk. Freaky polygamist dude may have correctly identified your gender, on the grounds that women don't have short hair, and certainly not shaved hair.

Which, if that is what happened... score one for the freaky polygamist dude, I guess. It would be kind of interesting if his view of the world led him to the correct conclusion when so many 'modern' people want to ignore presentation in favor of biology.