I think part of the problem stems in the fact that when faced with knowing nothing firsthand about an organization, people tend to believe whatever someone else tells them. And one thing I've run into - often - among people of all faiths is the myth that everyone else is being deluded, somehow. For example, if you are pagan, it's often said that you /think/ you are worshipping some gentle goddess or whatever, but really, you're worshipping the devil, and you either a) are an idiot fooled by the devil's trappings, or b) know the 'truth' but are lying to trap other people.

I've heard quite a few... rumors about the Mormon faith. I know very little firsthand. I wish I knew more.