25 January 2008

Some insignificant thoughts on statistical versus biological significance

Statistics books often remind us that the statistically significant differences between sets of data may not necessarily have practical significance. The emphasis on practical significance is probably due to the heavy use of statistics in comparisons of different procedures with practical applications in medical and industrial research.

I know of only one book, David Heath's An Introduction to Experimental Design and Statistics for Biology (UCL Press, 1995), that discusses, albeit briefly, biological significance as opposed to statistical significance:

…a statistically significant difference is not necessarily biologically significant. By this we mean that it may not be interesting from a biological point of view or useful from a practical point of view. The reason is that in actual fact the null hypothesis [that the samples being compared are identical] is probably never likely to be true! Animals in two different places are almost certain to be slightly different in size and seeds treated in two different ways are almost certain to have slight differences in germination times. If you took big enough samples, you could probably find statistically significant differences in almost any experiment or investigation, but this would not necessarily mean that the differences were biologically meaningful.
In the late 1940s and the early 1950s, Harold W. Harry (1921-1995) worked on the tiny land snail Carychium exiguum for his Ph.D. thesis. But for reasons unknown to me, the bulk of his thesis was not published until after he died*. There are a lot of shell measurements in Harry's thesis that were not analyzed with proper statistical methods, probably because of the way most biological research was done back then and also because computers were not available.

Recently, I started analyzing Harry's shell measurements of C. exiguum. I don't have access to Harry's raw data, which may not exist any more anyway. Luckily, however, Harry gave (in Table 11 of his thesis) the number of shells, range, mean and the standard deviation for 38 lots he had measured. For starters, I looked at the shell dimensions of 4 samples of C. exiguum collected at one station in Cheboygan County, Michigan between July 1949 and July 1950.

HarrysData

I thought the means for these 4 samples, all being from one station collected over a span of about a year, could be averaged into one set of data. But to make sure, I compared the mean shell heights using one-way analysis of variance (ANOVA). (Even when the individual measurements are not available, ANOVA can still be applied to the summary data.)

To my surprise, ANOVA revealed that there was a statistically significant difference (p=0.003) between the mean shell heights of the 4 Cheboygan samples. This is when I started thinking about biological significance as opposed to statistical significance.

The absolute difference between the mean shell heights of sample R1 (the largest mean) and sample R3 (the smallest mean) was only 0.041 mm! This is not only small in absolute terms, but also in a relative sense, corresponding to a difference of only 2.1%. The absolute and mean differences between the means of the other lots were even smaller. Could a mean shell height difference of 2.1% have any biological significance for the occupant snails under any circumstance? The answer depends on what we mean by biological significance.

What exactly is biological significance?

1. A variable (and measurable) phenotypic trait that confers a selective advantage (or a disadvantage) at certain quantities will be biologically significant when the same quantitative differences between individuals are statistically significant. Therefore, biological significance implies evolutionary significance.

2. Biological significance is not a fixed attribute of certain individuals. Biological significance is spatially and temporally variable; differences between traits within a population that are biologically significant under one set of external conditions may not be significant under another set of conditions.

3. Biological significance is a probabilistic attribute. For example, a predator may selectively kill smaller prey most of the time, but it may also kill larger prey occasionally.

4. Biological significance is a continuous attribute, for it can't be defined with respect to a strict cut-off point or an absolute limit in the quantity of a phenotypic trait. For example, in the case of the shell size of C. exiguum, it would be meaningless to claim that only the size differences larger than 0.1 mm were biologically significant. We can only hypothesize that biological significance increases as the size difference increases.

5. Biological significance does not necessarily imply an evolutionary advantage for larger, stronger, faster individuals. Partly due to the trade-offs associated with many phenotypic traits, being large, strong, fast, could be disadvantageous under some circumstances and, in contrast, being small, weak, slow could be advantageous under other circumstances.

6. The trait differences between individuals within a population could be as biologically significant as those between individuals from different populations of a species. Statistical tests compare population (actually, sample) means, but in real life it is the individuals that compete against each other.

As for Harry's C. exiguum shells from Cheboygan County, I don't think that the mean shell height difference of 2.2% could have had any biological significance for the occupant snails.

Furthermore, the shell height ranges of all 4 samples of C. exiguum completely overlapped (see the table above). It is difficult to find biological significance between the dimensions of shells in 2 or more samples when not only the means but also the minimums and maximums of all the lots are very close to each other or identical. In fact, the shell heights of the largest and the smallest individuals in sample R1 differed by 0.56 mm. If there was any biological significance between shell dimensions, it was probably between the largest and smallest snails within lots. Those would have been the individuals that made up the fractions (~4.6% of the population) that were about 2 or more standard deviation units away from the sample means.

So, why do we get differences between samples collected at a given location at different times? I can think of 2 explanations:

1. What we are seeing are the results of sampling errors.

2. The mean shell dimensions of a population of a snail (or any other animal) species may indeed fluctuate randomly or may go thru regular or irregular cycles. Such fluctuating or cycling quantities may or may not be biologically significant depending on the circumstances. To establish their biological significance it would be helpful to consider: (1) long term temporal trends in the data; (2) short term drastic changes in the environment.

Part 2 in this series is here.


*Harry, H.W. 1997-1998. Carychium exiguum (Say) of Lower Michigan. Walkerana 9(21):1-104.

3 comments:

pascal said...

Also keep in mind that shell height is a two dimensional measure. Snails are three dimensional creatures. I wonder how much difference there is in shell volume between these populations.

Interesting that there appears to be more variance in the R4 population than the R2 group, despite having a narrower max-min range. Just goes to show you how important a good sample size is when it comes to morphometrics.

AYDIN ÖRSTAN said...

Right. The problem is shell volume is not easy to calculate. I could, however, calculate the usual approximation of it as HxDxD.

Anonymous said...

Dear Aydin,
I'm uncomforatble with your conclusion, but I'm not entirely sure why. Maybe Fred Schueler's comment gets at part of my discomfort, that biological significance matters only when you are testing a hypothesis.
Anyway, I disagree with the first point in your Blog that biological significance implies evolutionary significance. I prefer a definition of biological significance that gets at the fundamentally basic tenet of biology (passing your genes to the next generation). If a difference influences an organism's reproductive success, then I would call it biologically significant. Since phenotypic differences can definitely be non-genetic, and phenotypic differences can definitely influence reproductive success (look at boob jobs and toupees), I disagree that biological significance implies evolutionary significance.
If you are doubting biologically meaning in the shell length differences, per se, you might be right (although I'm not entirely convinced). But if you look at the causes of the shell length differences, whatever they might be (e.g., a different amount of rainfall or warmer temperatures during the month before he took the sample), then they could still be biologically meaningful in the reproductive fitness sense. A snail with a larger shell can produce more eggs (references exist, but I don't have them at my fingertips) and therefore have greater reproductive fitness. (And even if genetics is not involved in shell size, genetics could be involved in finding microhabitats that would have the optimum combination of conditions for greater fitness, and might incidentally influence shell size.) Or on the other hand, a larger shell might be more cumbersome to carry, maneuver, or fit into smaller hidey holes for protection from predators or desiccation. Even a 1% difference can definitley be meaningful in an evolutionary context (references exist).
Best wishes, -Tim-