^{a}

^{b}

^{a}

^{b}

^{a}

^{b}

^{c}

^{a}

^{a}

^{b}

Meta-analytic datasets can be large, especially when in primary studies multiple effect sizes are reported. The visualization of meta-analytic data is therefore useful to summarize data and understand information reported in primary studies. The gold standard figures in meta-analysis are forest and funnel plots. However, none of these plots can yet account for the existence of multiple effect sizes within primary studies. This manuscript describes extensions to the funnel plot, forest plot and caterpillar plot to adapt them to three-level meta-analyses. For forest plots, we propose to plot the study-specific effects and their precision, and to add additional confidence intervals that reflect the sampling variance of individual effect sizes. For caterpillar plots and funnel plots, we recommend to plot individual effect sizes and averaged study-effect sizes in two separate graphs. For the funnel plot, plotting separate graphs might improve the detection of both publication bias and/or selective outcome reporting bias.

The visual representation of quantitative data is valuable for any scientific discipline. The added value of using graphical displays is twofold. It can help researchers to understand better the behavior of their data (

In the field of meta-analysis, defined as the statistical technique that combines evidence from different studies to reach more accurate conclusions (

There are two figures that are the gold standard in meta-analysis: forest plots and funnel plots. Forest plots (

The conventions for displaying forest plots and funnel plots are widely known for the scenario where a unique ES is reported per study (see, for instance,

Before detailing these extensions, we first give an overview of the available statistical models to perform meta-analyses, because some of the elements of the forest and funnel plots depend on the model fitted. The traditional random-effects model assumes that observed differences between ESs are not only due to random sampling variation, but also due to systematic differences across studies (e.g., each study uses its specific procedures, protocols, etc.). Under this model, it is assumed that each study has its own population value and that study-specific values are normally distributed around a ‘grand’ mean population effect (

where

When studies report multiple ESs, it is important to apply a statistical model that takes into account that ESs from the same study are related to each other, that is, that dependency among ESs exists. Ignoring this dependency can lead to biased estimates of the standard error and of the variance components, and hence to flawed inferences (

The three-level approach consists of extending the random-effects model of

where

Each of the following sections begins with a description of traditional plots, and continues with suggestions of extensions. In the ^{1}

Part of the code for building forest plots has been taken from the R code supplied by

Forest plots give information about the precision and the weight of each study. Each ES is depicted with a dot (usually a square) on the horizontal axis, and each row in the graph represents a different study. In a meta-analysis in which just one ES per study is reported, the precision of each study is represented with the (typically) 95% CI depicted along the sides of each ES (i.e., dots). This CI is built using the sampling variance,

where

where

At the bottom of the graph, a diamond representing the pooled ES is typically depicted, with its horizontal diagonal representing its 95% CI. Additionally, it is recommended to plot a vertical line that indicates the value of the overall effect estimate, and also a ‘no effect’ vertical line that serves as an indicator of whether ESs are negative and positive, or whether studies report statistically significant ESs.

In meta-analysis, there is a direct relationship between the precision of a study and the extent to which that study contributes to the final estimation of the combined ES. This relationship is visually evident in forest plots: The width of the CI of a study (i.e., its precision) is related to the area of the dot of that study (i.e., its weight), that is, less precise studies contribute to a minor extent to the estimation of the pooled ES. It is important to mention that the study-weights depend on the model fitted. When a fixed-effect model is fitted, the weight equals the inverse of the sampling variance, whereas when a random-effects model is fitted (

The weight of a primary study that reports multiple ESs depends on more parameters than solely the sampling variance and the between-studies variance. This can be clearly understood by looking at the variance of the mean ES estimate obtained when we apply a random-effects model to each study

The precision (or inverse variance) of the effect estimate for a study

Moving on to the weights, let us remember that in a traditional forest plot of a random-effects model, the weight of each study depends on both the sampling variance and the between-studies variance (^{2}

When performing a three-level meta-analysis, this between-outcome variance is typically assumed to be the same for all studies, and therefore is estimated across all studies.

) and on the between-studies variance. For obtaining the actual weight of studyThe matrix

More information about how to obtain

In order to appropriately visualize the different factors that affect the precision of individual ESs and studies, we propose to draw a modified version of the ‘summary forest plots’ (

The summary forest plot shown in

where

As shown before, the total precision of the studies included in

The first element is an additional grey CI, that is based on the sampling variance of individual observed ESs of the study. The formula for obtaining the upper and lower limit for these grey CIs is the following:

where median

The weight assigned to each study is represented by the area of the central dot, and in a three-level meta-analysis the study weight can be calculated with

The final conclusion of the forest plot in

Caterpillar plots are nothing but rotated forest plots. The main differences are that ESs are sorted and plotted by their magnitude (from negative to positive) and that ESs are depicted closer to each other, making it possible to get a general view of the distribution of all ESs. An advantage of plotting ESs closer to each other is that many of them can be plotted at the same time, whereas including too many ESs in the forest plot would make it difficult to interpret.

Because caterpillar plots allow the inclusion of many ESs, we have used the complete dataset of

This figure is commonly used in meta-analysis to investigate the presence of selection bias. The funnel plot is a scatter plot in which ESs (horizontal axis) are plotted against a measure of their precision (vertical axis). This measure of precision can be the sampling variance, the standard error, or their inverses. In this paper, we will use the standard error for the vertical axis, as recommended by

Besides publication bias, other types of bias might also be present, as for instance selective outcome reporting bias (

In a three-level meta-analysis, we recommend to first plot all ESs in the funnel plot to check for both selective outcome reporting bias and publication bias. In

In order to further clarify which of these kinds of bias took place, we propose to additionally plot study effects against their meta-analytic standard errors in another study-funnel plot (

An additional suggestion for the study-funnel plot is to depict the number of outcomes reported in each study next to the dots representing the study effects and/or to plot the size of the study-effect proportionally to the number of effects included in that study (as in ^{3}

In the functions included in the R code in the

Another special characteristic of the funnel plots in

where

In

In

Visualization of data is very important to understand, interpret, and summarize results, regardless of the research area of interest (

An advantage of the proposed forest plot is that a lot of information is included in just one image: the study-effects and their uncertainty, the uncertainty due to sampling variation, the number of outcomes reported in each study, the importance of each study in the estimation of the pooled ES, the extent to which each study contributes to the between-outcomes variance estimate, the value of the overall effect, and its precision. Regarding caterpillar plots, by plotting separate caterpillar plots for the observed ESs and for the study-effects, it is possible to get information of the pattern of the effects at the lowest level and at the upper-study level. Other existing graphs, such as Galbraith plots (

In sum, we believe that the incorporation of these extensions to forest plots and funnel plots in three-level meta-analysis can greatly help to accurately summarize and understand these (often large) datasets, as well as to improve initial inferences about the presence of selection bias.

The authors have no funding to report.

The authors have declared that no competing interests exist.

The authors have no support to report.

This article was written within the scope of the Doctoral dissertation of the first author (