How to make specious claims that research is drawing specious conclusions
UPDATE: The author of post I discuss below let me know that my comments were not blocked (i’m glad). The essence of this post, however, is more about what they say about the scientific study in question and not about my comment being blocked.
Like many science bloggers, I don’t like it when people use their blogs to try and shoot down scientific studies for unfounded reasons.
Let me give you an example.
Yesterday, I came across a blog on the WordPress home page, which highlights various categories of other WordPress blogs. Under the “science” category was this post: How to Draw Specious Conclusions From Research. How could I not click on that?!
You might remember a few weeks back, the media outlets and bloggers picked up on a study published in Nature Neuroscience titled “Neurocognitive correlates of liberalism and conservatism” (if you have access, you can see full-text of the paper here). There was some discussion of the study over at ScienceBlogs (see here and here). But, I don’t want to talk about the study in detail…i’m not a neuroscientist and will let others deal with the nitty gritty…what I do want to talk about is sloppy or, worse, purposefully misleading commentary about the study by bloggers.
One of the great things about being a scientist is that you learn how to read in detail and appreciate minutia. Remember, the whole point of their post is to claim that this particular study makes specious conclusions. So, I commented on the blog that perhaps it was the media’s interpretation and reporting of the study that is specious (since they link to an LA Times story and not the paper). My comment is paraphrased here:
Typically, the media report will try and make the big claims (i.e., headlines) to attract readers, whereas the study may offer some speculations, but the actual conclusions will be not that exciting.
The blog author immediately replied to my comment:
Trust me, I checked the paper. The authors are the ones who draw the specious conclusions, not the media. If you’d like to check the paper out, here is the Bibliography:
DM Amodio, JT Jost, SL Master, CM Yee, “Neurocognitive correlates of liberalism and conservatism” Nat Neurosci, 2007
Fair enough. I took a little time to read the actual paper. The first thing I wanted to do was check and see if what the blog author claims are their conclusions actually are. They start off their post with this:
Among their list of conclusions were:
- “conservatives tend to be more structured and persistent in their judgments”
- conservatives tend to ignore information
- “liberals are more open to new experiences”
- liberals can be expected to accept new scientific and social ideals faster
- the results “provided an elegant demonstration that individual differences on a conservative-liberal dimension are strongly related to brain activity.”
Firstly, only three of these bullet points even appear in the paper at all. Secondly, and much more importantly, the conclusions listed above aren’t Amodio et al’s conclusions. They address these attributes as hypotheses to test and cite the previous work that made the conclusions. This is very clearly stated in the abstract and the first paragraph of the paper! The point of their study (i.e., the point of a lot of scientific studies) was to take previous work and results and test them in a very specific way. So, the blog author is simply wrong when stating that “the authors are the ones who draw the specious conclusions”. The blog author asked me to check the real paper…I did. Amodio et al.’s actual conclusions go like this:
Taken together, our results are consistent with the view that political orientation, in part, reflects individual differences in the functioning of a general mechanism related to cognitive control and self-regulation.
Like most conclusions from very specific studies, what they actually say is worded carefully and within the specific context of the previous work they were addressing.
So, I submitted a comment to the post similar to what I just said above to point this out. Apparently my comment was thrown into the spam filter and never appeared (it happens). Nevertheless, my statements in this post about their deceitful (or simply sloppy) reporting of the study in question stand.
I’m not gonna take the time to try and comment again on their thread and deal with the spam issue. They can respond to what I say about their post on this thread if they like.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Clastic Detritus 
We don’t censor/block comments. It must have been eaten by the spam filter. Feel free to repost it if you like, I’ll check the spam filter before emptying it for the next few days.
no need to repost on your blog…my post here is linked to through the comment thread on your post…you can reply to my comments above on this thread if you’d like
My main comment is this: You’re nitpicking vocabulary. As someone who has written a fair number of scholarly articles, I agree with you that the wording in articles is very deliberate and very concise. While you are correct about the conclusion section of the paper, the authors deliberately tie these results with specious conclusions in the rest of their paper. They have an agenda and they are pushing it with a questionable at best study which did not control for other variables which could effect performance.
As a study of response times, it functions reasonably, but because they introduce a political agenda, which they tie to their results deliberately in sections preceding their conclusions, I have to disagree with you and I stick to my conclusion.
You and I both agree papers are written deliberately with an eye to meaning and implication. We both can see clearly the implications they are waving in everyone’s face clear as day, and that the results don’t support them.
What exactly is their ‘political agenda’ (as it’s laid out in the paper)?
How could they not discuss politics when their study was aimed at examining the relationship of political leanings with respect to neurocognitive measures?
I would agree that how the story was picked up by various and numerous news reports and blogs misrepresents and overhypes their paper…but that is my point, that is different from their paper.
Their political agenda is spelled out in the abstract and introduction. They’re trying to attribute certain qualitative aspects to Liberals and Conservatives with quantitative data. Unfortunately they never build any link (and their study is flawed. Only 8/43 were conservatives? Thats an obscenely low number, and I challenge their results on conservatives on the grounds that they did not provide adequate samples to draw meaningful results).
They could have avoided political strawmanning by sticking to the quantitative facts.
I still don’t understand…is their ‘agenda’ deemed political by you because they are doing a study about political leanings?
Or, you are claiming it’s political because you perceive their study as biased towards one way or the other (i.e., the sampling problem you mention)?
As for their agenda being spelled out in the abstract/intro…mabye you can say I’m parsing words too much, but I think it’s significant. They are addressing previous conclusions and testing them…perhaps those studies that they cite are specious (I don’t know, I didn’t read them)…that would require some more digging.
How should they have gone about addressing those previous conclusions without devising a test, doing it, and then discussing results with their peers?
If you think that studying political psychology is inherently biased, then, that is something deeper and fundamental to discuss…science is done by people after all. Although good researchers try and minimize those effects as best they can. But, I get the feeling you simply don’t trust these researchers for one reason or another right from the start (e.g., your putting quotes around the word researchers in the first sentence of your post; I assume that is commentary implying they are not “real” researchers).
This is one study…they are not claiming (in this paper) that their results are some grand unifying theory or description of Americans’ political psychology. Others are interpreting their paper as this and getting fussy over it.
How should they have gone about addressing those previous conclusions without devising a test, doing it, and then discussing results with their peers?
You’re missing the entire point. Their tests did not address those previous conclusions. They only addressed response times in a group of 43 students, for a specific test. They did *nothing* to tie these to the conclusions the mention earlier, except for throw them in the same paper haphazardly.
If it’s a WordPress blog, then it should still be in the spam folder and can be “de-spammed”
Only if the spam folder still contains the messages. Our spam folder gets hit with hundreds of messages a day. We empty it on every update.
“They only addressed response times in a group of 43 students, for a specific test.”
So, you don’t like the sample size…that’s fine. More data is always desirable, I agree.
Let’s get something straight…I’m not here to defend every aspect of their study. Of course it has problems…all science does, that’s why we continue to do it; so we can improve it. And we keep building upon previous results and studies. No one study proves/disproves anything…and they aren’t claiming that.
If you remember, my initial beef w/ your post was with your bullet points of conclusions. As I said above, firstly, some of those statements are nowhere to be found in their paper. And, your implication that this single study is what concluded those points is misleading at best.
I am not a neuroscientist or psychologist…and have neither the time nor the energy to look up and read the references they cite. You should dig up those papers and evaluate those studies. They may very well be flawed.
1. Jost, J.T., Glaser, J., Kruglanski, A.W. & Sulloway, F.J. Psychol. Bull. 129, 339–375
(2003).
2. Alford, J.R., Funk, C.L. & Hibbing, J.R. Am. Polit. Sci. Rev. 99, 153–167 (2005).
3. Block, J. & Block, J.H. J. Res. Pers. 40, 734–749 (2006).
4. Miller, E.K. & Cohen, J.D. Annu. Rev. Neurosci. 24, 167–202 (2001).
5. Botvinick, M.M., Braver, T.S., Barch, D.M., Carter, C.S. & Cohen, J.D. Psychol. Rev. 108,
624–652 (2001).
Finally…the LA Times story that you link to…I don’t think they did a very good job of portraying this study. That reporter is hypeing it up, I hate that crap.
As I said above, firstly, some of those statements are nowhere to be found in their paper.
Not in so many words, no, but thats the difference between quoted and unquoted.
And, your implication that this single study is what concluded those points is misleading at best.
The study proposes itself as something which validates old results, which gave those conclusions, so, no, I’m sorry, this study does claim they support those conclusions.
“Not in so many words, no, but thats the difference between quoted and unquoted.”
Okay…so, the unquoted conclusions are your interpretations…not their statements. Big difference. That by itself is misleading.
“The study proposes itself as something which validates old results…”
That’s right…the results of their study are consistent with other findings. That is what they concluded.
I think we’ve beat this dead horse enough…essentially, I think what may seem like subtle or seemingly insignificant wording is extremely important in scientific writing.
Okay…so, the unquoted conclusions are your interpretations…not their statements. Big difference. That by itself is misleading.
Its called a summary. Do you find opinion articles in papers misleading too?
That’s right…the results of their study are consistent with other findings. That is what they concluded.
No, they aren’t. Their study proves nothing about the way liberals and conservatives think. Their study shows only a (weak) correlation between liberals and scoring well on identifying M’s from W’s.
I think what may seem like subtle or seemingly insignificant wording is extremely important in scientific writing.
I agree, which is why I think the paper is junk.
“Its called a summary.”
One of your bullet points (which you said they stated was one of their conclusions) was: ‘conservatives tend to ignore information’
Another one, also labeled in your post as a conclusion by Amodio et al. was: ‘liberals can be expected to accept new scientific and social ideals faster’
How are these summaries? Where do they come from? Sounds like your interpretation to me. And the last bullet point is some other researcher’s (not even connected to the study) comment on the study.
“Do you find opinion articles in papers misleading too?”
I don’t understand the relevance of that in this conversation.
“Their study proves nothing about the way liberals and conservatives think.”
Again…Amodio et al. conclude: ‘Taken together, our results are consistent with the view that political orientation, in part, reflects individual differences in the functioning of a general mechanism related to cognitive control and self-regulation.’
I don’t read that as proving the way someone thinks.
Like I said…dead horse, beaten. Got it…you think this study is junk. You don’t like the researchers, you don’t like the institutions involved, you don’t like the methods, you don’t like the results. In fact, based on your comments, it seems you don’t trust the entire field of study. It all fits. Again, my point is not to say that their study proves or disproves of anything…my point was that you misrepresented what they actually state in their paper. That’s all.
You don’t like the researchers, you don’t like the institutions involved
Ah, yet another person drawing specious conclusions.
Okay then…it seems we’ve reached the end of this conversation. I made my point, the majority (3 out of 5) of your bullet points that you claim are Amodio et al.’s conclusions aren’t even in their paper.
Seeing your last comment, I guess you just want the last word in a downward-spiraling discussion…have at it. Cheers.
Seeing your last comment, I guess you just want the last word in a downward-spiraling discussion…have at it. Cheers.
Nope, just commenting on your resorting to ad hominem. I thought we were having a pretty good discussion and you decided to take it elsewhere. I could care less if you deleted my “last words”. Just trying to point out why I’m no longer going to bother with this entry, unless you’d like to try and start over
“Just trying to point out why I’m no longer going to bother with this entry”
Your previous comment (at 12:06am) pointed that out clear enough. I’m sorry you perceived the suggestion that you wanted the last word as ‘resorting to ad hominem’ (bloggers just love that term)…since your 12:06 comment did not address my main point, I figured you were done.
The ad hominem I was referring to is:
You don’t like the researchers, you don’t like the institutions involved
You’re drawing conclusions about my opinions with no facts to back up your assertions, you’ve decided to just characterize what I say as you would prefer it to be. If you’re going to straw man me in such a way, I don’t know how we can have a discussion.
Sorry…the real straw-manning is listing conclusions as if they were in the paper and then subsequently tearing them down.
Regarding my comment about you not liking the researchers, I got that impression from the first sentence of your post: “Today’s article is brought to you by “researchers” at NYU and UCLA.”
Why is the word researchers in quotes? I may be interpreting it wrong, but the usage of quotes in this way is typically negative commentary and that the term shouldn’t be applied. I really don’t care…i’m not defending them, but this is why I interpreted it this way.
You accused me of being nit-picky about the wording in Amodio et al.’s paper…I think you may be a little nit-picky about accusing me of ad hom. Ad hominem statements are typically much more personal than what I said.
Nevertheless, I apologize and retract those statements.
But, I do think this conversation has reached a limit of ‘diminishing returns’ so to speak. I really don’t see what else can come of it.
Sorry…the real straw-manning is listing conclusions as if they were in the paper and then subsequently tearing them down.
They are in the paper. They just aren’t in the conclusion section. However, they are claimed to be directly linked to the conclusions drawn. As such, the paper claims their results lead directly to those conclusions, which they don’t.
Why is the word researchers in quotes? I may be interpreting it wrong, but the usage of quotes in this way is typically negative commentary and that the term shouldn’t be applied. I really don’t care…i’m not defending them, but this is why I interpreted it this way.
Because the work in the paper doesn’t actually prove what they claim it does. I’m questioning their motives in writing it and suggestion that for the purposes of this paper, they are playing another role.
As for NYU and UCLA, both are perfectly respectable institutions, and the authors may be perfectly fine researchers when wearing that hat. I wear a research hat often myself. My writing style is very different then, and I don’t include opinion or personal politics. They confused their roles.
You accused me of being nit-picky about the wording in Amodio et al.’s paper
Not about wording, but about sections. Everything is in that paper that I claim, the section it appears in doesn’t not automagically make it not a conclusion.
Nevertheless, I apologize and retract those statements.
I apologize for any statements I may have made which you found insulting, and admit to my statement on your reaction to op-ed’s as being to far, and retract it as well.
But, I do think this conversation has reached a limit of ‘diminishing returns’ so to speak. I really don’t see what else can come of it.
Maybe, up to you.
Hmmm…so, how is my comment about your statement about the researchers ad hom? I thought ad hom was unrelated to the subject being discussed…if I was referring to how you referred to the researchers, which was to question their motives about this study (i.e., what we’ve been discussing), then is my comment really ad hom?
Just reading what I wrote above tells me we (or at least I) have reached a limit where anything constructive comes out of continuing this…I made a nit-picky point about how you presented the study’s conclusions and now we’re starting to discuss how we’ve been discussing it.
Your opinion is that the researchers were politically motivated and/or biased, my opinion is that you misled your readers with the conclusions list — yes these aspects are connected to the paper, yet also distinct points. We could go on forever trying to explain what we’re actually discussing to each other. We have both made our points. Diminished returns.
I’m not in it for the thrill of debating or ‘winning’…I don’t gather you are either…but, just in case you are, I have no problem ‘conceding’ as this is getting rather silly. In other words, I don’t really care that much.
It’s an insignificant piece of work that looks like an undergraduate research project. This sort of note just adds to the authors’ citation list. If work like this is to be published at all, it should be part of a bigger study that draws it all together in a proper comprehensive paper, so maybe such a paper is forthcoming. Far more valuable scientifically is the response of the press and conservatives. The press show they are scientifically illiterate, and conservatives reacted just as RWAs would.