Several years ago, my sister Theresa posed the question, "How large does a group have to be before you are assured of having at least one objectionable person?" I don't think any definitive answer has ever been arrived at, but this assumes that you are not inviting the members of the group.
Another question that I do know the answer to is, "How many people does it take to ruin a discussion?"
It takes one.
Back about fifteen years ago, The Foundation For Economic Education sponsored discussion groups around the country that anybody could start with their assistance. I started one and it went well for quite a while. FEE sent some speakers and they were all well received, but it was usually just us locals discussing various things, usually from The Freeman.
The whole thing was very informal, and members could invite others to "join", even though there was no organization and no dues. One day, one of the regulars brought somebody who proved to be the death of the whole group. The person was very nice, polite and not stupid, but incredibly shallow. No matter what was being discussed, from the origin or money, to the French Revolution or the types of igneous rocks in Iceland, somehow this person would always work the conversation around to some movie.
"Igneous rocks? Oh, that reminds me of the movie Journey to the Center of the Earth where these guys go down into a volcano....etc." The conversation is wrecked from that point. Unless you are prepared to be very rude and say "Shut up, you're a bore. Nobody wants to hear about that movie." The whole discussion seems to be unsalvageable. I don't know exactly why this happens, but it seems to be a kind of lowest common denominator conversation.
If I am ever so foolish as to start another discussion club it will have the provision that nobody can bring a guest without the prior approval of the others, and the guest can only attend once. After the initial encounter, the regulars can decide if the guest would make a good addition. If so he can be voted in; if not, he can be left in the outer darkness where there will be weeping and gnashing of teeth.