Friday, January 18, 2013

why academic research gets a bad reputation: Bad data, disingenuous representation, and muddying the waters in an Ivy League journal

This post probably does not belong here, but it is directly associated with good academic writing and academic integrity, so I am sticking it here.  [mostly not to punish my facebook friends with it :) ]

----- ----- -----

Scott's link:
And then there is this:
http://www.law.harvard.edu/students/orgs/jlpp/Vol30_No2_KatesMauseronline.pdf

----- ----- -----

My response:

(sorry for hijacking your thread) Hey Scott, a couple of points on the link you posted.

1. I should be honest, I find this the most specious, misleading, and disingenuous bit of academic "research" since the tobacco companies were paying folks to fabricate misleading reports on the links between tobacco and cancer. Rather than post my own treatise here which would likely devolve into the sort of ad hominem attacks found in the article, let me focus on just two points: Which countries they are comparing the US to in international data, and critiques of the correlational research that underpins the "more guns, less crime" argument.

2. Their major argument in the international data is something like, "Well those guys said America has the worst murder rate in the world, but really Russia is worse." That is an accurate statement.  Ummm, and we are not as bad as Somalia and Rwanda and Pakistan, either.  The argument from the "gun-control lobby" has always been compared to other industrialized nations.  Russia (and the CCCP) was always excluded form that argument.  I was living in a former Soviet republic in 2001--it was not industrialized, or western, or an all "civil" even in the capital city.
  But a bit more on the evidence they use for comparison -- my data here comes from gun policy.org which seemed to be using CDC and WHO data for the most part. If you look at the chart on page 652 and mentally put in the us figure which was 6.13 according to the CDC, we are in third place for the highest murder rate. Do you see that Luxembourg is number 2?  Weird, right? The authors goes on to make a big point of it in the chart on page 664: Luxembourg is European! interestingly, they don't even bother to do a correlation, or show years of data to indicate a trend.  But here is the funny thing, Luxembourg's murder rate in 2002 was not 9.01.  I checked because Luxembourg's population is tiny, so their rates can fluctuate more, right?  Turns out the rate was 2.1 according to the WHO (see gunpolicy.org). In fact, in that data between 1995 and 2008 Luxembourg's rate fluctuated between 0.5 and 2.5.  So these guys' data is off by a factor that would average to around 10! But maybe the data I have is wrong, so I checked their data. They used the 2002 annual report from the Canadian Center for Justice Studies.  And sure enough there it is in the report 9.01 for Luxembourg.  So they are not lying. But what about the trend? I went three years either way, and NO mention of Luxembourg in the Canadian reports. So you can choose which data to trust: a one off Canadian report, or a consistent sample from the WHO's European commission.  If you choose the WHO then you have all of the data that these author's have presented for international comparison showing, rightly, that the US murder rate was not as bad as some (even many) of the former soviet republics ten years after all of the policing mechanisms of their "civil" (I mean totalitarian) society had fallen apart. Go USA!

3. RE more guns, less crime: First, the Lott/Mustard study is correlational, not experimental.  it is not even structural equation modeling.  It relies heavily on their models and ultimately, there is no causation. Moreover, regardless of the title, it is not about "more guns: less crime" it is about "more concealed weapons: less crime."  So when Kate/Mauser ("these guys") argue that it the Lott/Mustard study refutes the "less guns: less crime" contention it does not and cannot.
   But I want to direct you to the footnotes on page 658. The author's write: "Several critics have now replicated Lott’s work ... the replications all confirm Lott’s general conclusions"  Wow! impressive. Take a look at some of the critics listed.  One of them is Lott himself! Another critic is mustard.  Wow, they are critics of their own work.
   Moreover, hadn't I read somewhere that different statistical analysis had produced different results? Wasn't there an Ayres & Donohue (2003) study that showed not statistical significance, or even an increase in assaults using the same data set, and adding in subsequent years data (Lott/Mustard was 1997)? Yes there was.  I read that paper. It says the Lott/Mustard data falls apart.  How is it that they missed this study when they say "the replications ALL confirm" (my emphasis)?  And how is that they can say that when the Ayres & Donohue (2003) is listed in the same footnote? [note the ad hominem attack in their footnote].  But maybe that was the only one . . . an oversight? Nope, turns out that another reference in the same footnote is and empirical study that replicated the Lott/Mustard study and found that the Florida data in the study accounted for 80% of the variance, and if you remove the Florida data the differences are no longer significant.  These dudes are intentionally lying when they say "the replications all confirm."

I am pretty sure I would find more stuff if I looked harder.  But let me ask you this:  If a moron non-expert like me can find all of these critical errors in their arguments/paper then what would someone who was an expert in the field find? [and also ask yourself, where the hell was Harvard in the peer review process?]

cheers,


No comments:

Post a Comment