A significant number of people think that parents being polyamorous, or swingers, or in an open relationship, is somehow harmful to children. When parents are divorced, this is often used to keep children away from a parent involved in completely ethical non-monogamy. In a recent /r/polyamory post, someone asked what grounds people have to maintain this prejudice.
This is the kind of question for which it's difficult to find a rational answer. One must resort to asking people who actually have these opinions, and figure out their thought process based on their evasions and non-replies.
To many of us practicing some of these lifestyles, it's obvious there's no danger at all. But try to explain that to people whose range of opinions might include that homosexuality is an abomination, and that gay people should not adopt children - presumably because they'll raise them to be homosexual, or tolerant of homosexuality, which they basically consider just as bad.
If you ask them why they believe this sort of thing, they won't be able to tell you. These aren't people who form opinions based on arguments, they form them based on the first emotion that comes to mind. This emotion might be a feeling of disgust, or jealousy, or some other kind of generalized wrongness.
Last time I had an argument with a person like that, they pretty much equated feelings of disgust and jealousy with conscience. By that, this person explicitly meant that if one overcomes jealousy in a poly setup, or does not feel disgust for a homosexual act, then one either (1) does not have a conscience, or (2) is repressing their conscience, and the only thing that's left of a person on this track is sociopathy.
What we have here is a majority of people who lack introspection in any meaningful sense; who don't differentiate between disgust, jealousy, and conscience; who think that any instinctual negative feelings are God-given and right, and trying to overcome them is immoral.
The way these people see it, it's immoral to expose kids to social structures where they might receive insight that would encourage tolerance, and hinder instinctual negativity. They want the child to be like them, which is to experience the same destructive emotions which they think constitute "conscience".