Heterogeneity
When we combine the results of 2 or more studies in a meta-analysis we need to assess the amount of difference or variability between the study outcomes. If 2 apparently similar studies give opposite outcomes there must be a reason (eg the interventions were not exactly the same, the study populations were different). So if you see in a Forest plot that the confidence intervals surrounding the point estimates do not overlap (this is visual heterogeneity) you may wonder about the heterogeneity of the studies. One method of establishing the presence of heterogeneity statistically is by using I2 which measures the variability in point estimates that is due to true differences in the effect rather than differences due to chance. So when I2 is 0% any differences in the point estimates is due to true differences not chance. On the contrary if I2 is 75%, so 75% of the variability in outcomes is due to true differences and not just chance, one would question whether it is appropriate to combine these studies. In this meta-analysis both the forest plots show I2 is 0% so any variability is due to chance and it is appropriate to combine the studies and come up with a pooled estimate of the treatment effect.
Statistical Heterogeneity
The authors do a formal assessment of the statistical heterogeneity in the study. Statistical heterogeneity can be less intuitive than clinical and methodological heterogeneity, but it’s easier to quantify. Statistical heterogeneity, again, refers to the differences in the results found in the different studies. The authors use statistical software to calculate a Cochran’s Q statistic. The Cochran’s Q (no relation to the Cochrane collaboration—they are just friends) depends on the differences between the results of the individual studies and the overall results of the meta-analysis. Other references exist for those who are very interested in learning how to calculate and work with Cochran’s Q.
Using the Cochran’s Q, the authors can then calculate an I2 statistic. The I2 is the current darling of heterogeneity measurements. It can be helpful in letting you know if there is a lot of statistical heterogeneity, or just a little bit—but it does not tell you anything about the other kinds of heterogeneity. It does not tell you whether the authors did a good job of focusing their question well. Only you, as a careful and informed reader, can decide that.
If the Cochran’s Q and the I2 show significant statistical heterogeneity, look to make sure the authors have performed a type of analysis called a random effects analysis. This analysis will help account for the statistical heterogeneity. Less statistical heterogeneity can be dealt with through a fixed effect analysis.