28 September 2007

A paper on Lauria cylindracea that should have been rejected

I occasionally review on this blog papers that are several notches below the standard quality of a scientific paper, including some that should never have been published. There is one example in this post.

This edition is about a paper by Carsten Renker, titled "Genetic break in Lauria cylindracea..." and published this year in the Archiv für Molluskenkunde, vol. 136, pp. 1-7.

Lauria cylindracea is a rather small land snail. It appears to be somewhat variable in the dimensions and morphology of its shell. So, there is a possibility that there may be cryptic species within the group of snails loosely identified as Lauria cylindracea. To evaluate that possibility, which was a great idea, Renker studied 20 samples of L. cylindracea from various locations in Europe, including, Great Britain, Germany and Greece. He compared the sequences of nuclear ribosomal DNA and shell dimensions of the samples.

I am not qualified to judge the DNA part of the study. I am, however, very critical of the way shell dimensions were analyzed and reported in the paper.

The fatal flaw is that Renker compared the means of the 20 samples using multiple t-tests. Even the most elementary statistical courses and books warn the students that they should not use the t-test for multiple comparisons of more than 2 samples. That's because, without going into details, multiple t-tests increase the chances of getting a statistically significant result just by coincidence. The accepted and standard procedure for comparing more than 2 samples is the method known as the analysis of variance (ANOVA). But apparently, neither Renker nor the 2 anonymous reviewers were aware of this fact. (Here is a brief discussion of the t-test versus ANOVA and here is another one.)

All Renker says about the results of the tests is this: "...shells collected in German populations were generally slightly taller (p<0.001) than shells from other countries (Fig. 1)."

First, this is not how one reports the results of multiple comparisons. I would like to know which German samples were "slightly taller" than which of the other samples. Figure 1 from the paper reproduced below shows that there was quite a bit of scatter and overlap in the distributions of the sample means and 2 German samples (marked by the letter D) are positioned among the samples from other countries. Second, Renker's one sentence summary is a meaningless statement. What does "generally slightly taller" mean? It means nothing. There may indeed be statistically significant differences between the means of some of the samples, but no reliable evidence is given in this paper that they are.

renkerFig1

The above figure from the paper is a bivariate plot of shell length versus shell width. The markers denote the positions of sample means and the bars are the standard deviations. In a plot like this it would be more meaningful to show, not the standard deviation, but the extent of the standard error of each mean. Since Renker does not give sample sizes either, there is no way to estimate the confidence intervals of sample means from the standard deviation values.

Another problem: Renker states that "Morphological data were tested for normal distribution with the Kolmogorov-Smirnov test." But he does not say if the data were normally distributed. Stating that a test was done is not helpful unless the results are also given. Since the t-test was used, we can only assume that the distributions were normal.

I am disappointed that Archiv für Molluskenkunde published this paper. If I had been a reviewer, I would have rejected it.

No comments: