Comment on "Resource Footprints are Good Proxies of Environmental Damage".

Steinmann et al.[1] argue that “resource footprints are good proxies of environmental damage”. They do so by using multiple regression analysis of four independent variables (energy, material, land and water footprint) to model two dependent variables (damage to human health and damage to biodiversity). Their model employs logarithmic transformations “because the footprints varied up to 10 orders of magnitude”, as it is based on no fewer than 976 products, each with 1 kg of product as the unit of analysis. Indeed, their results show that “the four resource footprints accounted for more than 90% of the variation in human health damage” and that “the four resource footprints also accounted for at least 90% of the damage to biodiversity”. In the end, the authors “conclude that energy and land footprints provide valuable proxies for the overall environmental damage produced by a particular entity”.

While indeed an R2 of 90% or higher is often seen as representing a model with a high explanatory power, it is questionable if that implies a high predictive power as well, as the term “proxy” is supposed to reflect: by doing a quick resource footprint analysis, the much more complex damage indicators can be simulated by proxy. It is this connection that we challenge in the first place. To do so, we proceeded as follows:

we used the Excel file with the raw data from the article’s SI and the regression coefficients from Table S2 of the article’s SI;

we calculated for each of the 976 products the estimated human health damage, applying the antilog to the predicted values from the regression analysis;

we calculated and analyzed the degree of correspondence between the estimated and the “observed” values of the human health damage.

The ratio between estimated and “observed” human health damage is found to vary between 0.03 and 28, with a mean value of 1.7. This means that on average, the model overshoots the “observed” value by a factor of 1.7, but that overshoots by a factor 28 also occur, as do undershoots by a factor . Seen in the perspective of the many orders of magnitude, such factors may seem negligible. However, they are not negligible, because LCA results are typically used for decision-making in a comparative context. For instance, an LCA study might address the question if product A is environmentally superior to product B. To study the effects of using a footprint proxy instead of the detailed damage model, we calculated the ranks of all 976 products using the two systems. Results are shown as a scatterplot in Figure . In general, products that rank high with one system also rank high with the other system. This is not surprising, because the 976 ranks cover so many orders of magnitude. But within 1 order of magnitude, there are quite a few differences, as will be clear by the large number of points that are not on the diagonal line. Every such point indicates a possible rank reversal, i.e. a case where one system would indicate a preference for product A while the other system favors product B. As Figure shows, this is far from uncommon.

Figure 1

Every point represents the rank of the estimated human health damage (through a regression analysis of four resource footprints) versus the rank of the “observed” human health damage.

A second point is the equal unit (1 kg) of all 976 products analyzed. This might seem as a fair basis of comparison, but it is responsible for the many orders of magnitude difference in impacts. As an example, the human health damage ranges between for 2 × 10–9 yr for 1 kg of water to 6 × 10–2 yr for 1 kg of rhodium. As such, it distorts the results. Compare this to the case of using GDP to predict CO2-emissions. Of course a small country like Honduras is low on both, while a big country such as Brazil is high on both. So, GDP might be regarded as a good proxy for CO2-emissions. But if we rescale the data per capita, a very different story will be told, and differences in the structures of the economies, which were first overshadowed by the scale effect, will start to emerge. In the case of the 976 products, no obvious rescaling principles suggests itself. So, we rescaled all products to an amount such that the energy footprint is equal to 1 MJ. Next, we observed what this implies for the human health damage. We find a vector of numbers that fluctuate around the mean of 2.8 × 10–7 yr, with a standard deviation of 9.4 × 10–7 yr, yielding a coefficient of variation of 336%. So, products with an equal energy footprint of 1 MJ can display a huge variation of human health damage scores, while according to Steinmann et al. (2017) they have a similar estimated proxy value. We interpret this as a bad proxy.

A final issue is the use of statistics for the purpose of this paper. The selection of 976 products is anything but random. It is based on all products in a certain database, to which some additional filtering criteria were applied. Finding the coefficients of best fit and computing R2 does not need a random sample, but further statistics (standard errors, VIF, AIC, etc.) require random samples. As such most of the statistical analyses by Steinmann et al. (2017) are not valid.

In conclusion, the analysis by Steinmann et al. (2017) is far from convincing. It demonstrates that footprints can explain a substantial part of the variation in damage scores across products, but it does not answer the question if footprints provide good proxies for such damage scores in the context of comparative decision-making. Further, if we remove the orders of magnitude, a much poorer explanatory power remains, suggesting that the predictive power is not at all good.