My wife the psychologist struggled a lot with her survey, which was part of her thesis on parenting attitudes. It took months to get it together, and it was reviewed multiple times by multiple people who were evaluating her experimental design, and it was also reviewed for ethics by the human studies review board. I acquired a healthy respect for the skills required!
So when I saw the Conflict In the Secular Movement survey, I had to gag; even I could see the huge flaws. Leading questions, overt biases in the choices of answers…what amateur put that thing together? It was clearly somebody with no knowledge of survey design.
Both Major Mike at Secular Woman and Amanda Marcotte have scathingly sarcastic responses to this awful survey. Go read them.
I just knew the survey was terrible when I saw the first question was what is the cause of conflict in the secular movement?
, and Rebecca Watson wasn’t one of the choices. Come on, crappy survey designer, you knew you wanted to make her name one of the choices for every question!
1. Is PZ Meyers* a big, mean poopyhead?
0 Yes
0 Si
0 Oui
*Have to spell your name wrong for authenticity’s sake.
A Likert scale with appropriate modifications to their questions would have improved that survey 100 fold.
My brain leaked out my ears when “I do not think diversity is a problem in the secular movement” was one of the possible answers to a question which began “If you do not think diversity is a problem in the secular movement…”
Yes, I do know that surveys are hard to design. I did survey research for 30 years, and one of my pet peeves is people who think any literate person can design a survey. That survey is a textbook example of what can go wrong when amateurs try to do it.
Even if we ignore all the leading questions about evil divisive feminists…
my bolding.
Did someone write this 5 minutes before due date and then whoever was overseeing the whole thing just published it without looking?
*sigh*
I should really hit refresh more often. Sorry, Mellow Monkey.
The second question doesn’t even make sense: “2. Do you believe feminists in the secular movement create conflict because of their emphasis on feminism?”
Do they not realize that conflict requires two parties? That what the words means. Two things, two views, two approaches, two people, etc, are in conflict. You cannot create conflict on your own.
*Ahem*. If I could speak on behalf of true amateurs everywhere, plenty of (literal) “professionals” have no business, no pun intended, charging for their incompetent services.
That is an idiotic survey in the extreme, though the tip-off should probably be seen in the assertion in the first sentence.(“recently”, eh?) and the grammar of the second (among other things, what community?).
Jacob Schmidt @ 7
Even apart from that, what do they even think they’re implying? I mean, yeah the emphasis on feminism is a source of conflict. Because misogynists don’t like it when you call them misogynists. Why would you even answer “no” to it? I guess you could argue that misogynists cause conflict because of their emphasis on misogyny but it amounts to the same thing.
That isn’t a survey; it’s just a piece of drek that John D of sl;ymepit infamy linked to and almost certainly put up himself that, before Major Mike’s piece, no one had noticed, even in the slymepit. It had, as far as I can tell, literally zero publicity or acknowledgement outside of that single slymepit post before Major Mike’s piece arrived. In that context this “survey” doesn’t even rank as one of the hundred most cringeworthy things said in the slymepit that day, much less does it rank as something worthy of Amanda Marcotte and PZ Myers singling it out for special treatment.
Seriously guys, this barely rises to the level of someone being wrong on the internet.
duh! Some people fabricate “surveys” to camouflage their bold assertions. Isn’t that why we ‘pharyngulate’ odious faux-polls?
I really don’t know. Usually the emphasis on questions like that is the final clause (i.e. “because of their emphasis on feminism”), so if one answers “no,” it might be taken as “they create conflict, just not because of feminism.” It really should have been broken up into 2 questions: “Do you think feminist activists are at the centre of the conflict,” and “Do you think their emphasis on feminism is central to the conflict,” and maybe a third question letting people answer some variation of “It’s their man hating b!tchiness that’s the source of conflict.” We’d probably get a lot of the 3rd one, but at least it would be a clear answer, not that vague, useless hogwash.
Oh yes, they are.
You need to formulate your abstract constructs in advance, you need to check whether your question actually asks for what you think it asks, best do a trial with interviews alongside, reevaluate your questions, rinse and repeat. Be aware of biases and tendencies, and so fucking on
Feminism: threat or menace?
R Johnston @10
Your concern or lack thereof is noted.
In my younger years I was asked a very loaded question by a local TV reporter out to score some anti-atheist point. Instead of backing up and tip-toeing through his minefield, I just said “Fuck you, that’s why,” knowing they’d have to cut that part out of the broadcast news.
They did! “Fuck you, that’s why,” should really be an answer for all of those. It is implied, after all, for many of them.
The survey was circulating around Facebook before “Major Mike” got involved, and some were trying to determine its prominence. Not sure what the threshold is supposed to be before something deserves comment.
Meh. I didn’t see it on Facebook–although I’m admittedly not much of a facebooker–and google and other searches turned up literally nothing other than a single slymepit post linking to it. I could certainly be wrong, but it seems to me that there’s no there there, or at least there wasn’t much until people on the right side of the rift started publicizing the “survey.” Even by slymepit standards this wasn’t a serious attempt at a survey.
R Johnston, you’re seriously concern trolling right now. I’m gonna go ahead and venture that PZ, Amanda Marcotte and Major Mike are all better judges than you are of what’s worth their time to write about. Nobody is claiming it’s a harbinger of the end times. It’s just been offered up as an example of how not to construct a survey.
Al Dente: @15:
That’s radically unfair. Fuck off with your accusation of concern trolling. I may not post often, but it’s plenty clear what side I’m on and it’s PZ’s side and yours. PZ can write about whatever he feels like writing about. It’s just that this “survey” seems like a completely random dropping to be drawing such notice from prominent people on the right side of things. I’d get it if people on the other side were clearly hyping the “survey” and any “results” it produced, but they don’t appear to be.
Wait, wait, wait. Wait.
When did “outreach” get to be a f*cking verb, as in question 5, option 2?
“…work harder to outreach minorities”
I srsly don’t even. That’s the weirdest weirding of language I’ve seen in a loooong time. (and these jerks just need to GTF offa my damned lawn!)
Another big question is exactly what the survey means by the “secular community.” Perhaps it should ask — oh, whether the survey respondent has been to any conventions, maybe?
I mean, since I’ve become an atheist, about 98% of my contact with other atheists has been right here on FTB and Skepchick. So as to the question of whether the “secular community” has been welcoming to minorities — sure, the part that I’ve seen has been. Is that part a majority or a minority of the community as a whole? Well that depends on whose side you listen to, doesn’t it?
@ R Johnston
When you start saying things like “nobody was paying attention to this until you brought it up” that’s concern trolling. I give no fucks which side you’re on or how much it hurts your feefees to have it pointed out. However, I do know what side you’re on which is why I’m rather dismayed that you think wondering aloud why anyone cares is a useful thing to do.
The survey designers wanted a pre-determined outcome, and they worked very hard to design a survey that would give them that outcome. I think they have a brilliant future ahead of them as Faux Noize pollsters.
R Johnson @20
If you really, truly didn’t care about the survey then you wouldn’t have said anything about it. Instead you complained bitterly how people were paying attention and even commenting on “a piece of drek that John D of sl;ymepit infamy linked to and almost certainly put up himself” and ” ven by slymepit standards this wasn’t a serious attempt at a survey.” That’s concern trolling. “Oh noes, people are paying attention to something I don’t think warrants attention. Stop paying attention to this, you mean attention payers.”
I’m not annoyed that you told me to fuck off because that would mean that I considered you to be someone worthy of annoyance. You don’t pass the threshold of annoyance. Better luck next time. (And yes, that’s me concern trolling you.)
A bit rich, for a survey designed to be passed around via social media.
Also, what is this “the” secular movement of which they speak. There has to be at least two, ’cause I certainly don’t wish to share the one this twit’s a part of.
(Also ? Here; this might help.)
And how would they work to achieve this without encroaching on their sacred ? You can tell you don’t actually care about freedom of speech when you are asking yourselves the same question authoritarian govts tend to ask.
It’s election time and I got a polling call the other day. They asked about the minimum wage hike then sited a report saying jobs would be lost if it happened but it gave no details about who did the study. My first thought was “who did this study, the Heritage Foundation?” but since it was automated I couldn’t get any details to determine if their study was legit or not. Either the poll was poorly designed or it was intentionally designed to push people a certain way. The joke was on them because I already voted thanks to early voting options.
Well, I do think there’s a lot of unnecessary name calling online; to pick only two examples, witness how Rebecca Watson and Stephanie Zvan have been described. I wish there was any real chance that’s what that question was about.
It’s always intentional to try to influence your vote. I used to get those kind of calls all the time before I stopped having a land line. I hung up on the robo-calls (my time is far too valuable to waste listing to one of those), and typically I ended up frustrating the hell out of the live calls since I always answered a “how would it influence the way you plan to vote if I told you that candidate X is in favor of kicking puppies and stealing candy from orphans” with a request for evidence.
I learned how difficult surveys are to design when I took my political science research methods course. A cursory look at this survey has me questioning if they even operationalized the relevant terms.
Your first clue that it’s a bad survey should be that it was on surveymonkey.
The RNC does ‘surveys’ like this in their fundraising letters…
“Do you agree with the Democrat plan to give away all your hard-earned money to diseased illegal immigrants?”
xD
How do you even answer a question like”do you believe that the responses over an argument are generally appropriate or antagonistic?”
Do you mean arguments in general or some specific argument ?
@32 coreyhammer: You got that right!
The second clue is the poor grammar in the introduction, “Recently, there has been conflict in the secular movement. This survey is collect information that give insight to the particular issues the community is facing.” “…is collect…”? And the third clue is there is no indication of who posted the survey, nor what use (if any) they will make of it.
When I was the Forms Designer at the U of Arizona Medical Center, I often worked with a biostatistician on his surveys. We had lots of ding-dong conversations about what a particular question/statement was *really* saying. Sometimes it was really difficult to tease out meanings and reword things, not to mention the actual form design itself for ease of use. Fun times.
that did not seem to be a survey to me really. I started to hear it as a speech with questions inserted for response from the audience to help rally them. It was just disguised as a survey a little bit but still slanted as hell.
uncle frogy
Having taken a research methods course in a social science discipline, yes, I did know that surveys are actually quite difficult to design well. I’m going to guess whoever made this survey has never even heard the term “survey design”, let alone studied it in any capacity. Conditional questions without a response option for those who don’t meet the condition, a tautological response, positioning items that are not mutually exclusive as though they are, several matters of fact framed as opinion: the thing is a mess. It wasn’t even checked for conformity to standard grammatical conventions – there are missing and misused words. I’m not even sure what the author is trying to ask in some of the questions.
Okay, yeah, that last question just about fried my brain. It was utterly incoherent, in a way that would make Salvador Dali shake his head at its obtuseness. Some of the earlier questions, as noted, were clearly designed to try to score rhetorical points without getting called out on the bullshit, but “appropriate or antagonistic” as some sort of binary option to a vague, almost illiterate sentence doesn’t even manage to rise to the level of being shit. It has no meaning, no point, no purpose.