Burgess COMMENTARY

Peter Burgess

What sustainability metrics can learn from school reform

Editor's note: This is the first part in a multi-part series examining the pitfalls of sustainability measurements that draws on examples from outside the business world. Part 2 will offer examples from criminal justice and financial services, and future installments will bring in other fields.

It’s a great irony that the best-intended measurement programs can create perverse incentives or unintended consequences, like cheating scandals born from school testing initiatives.

Deloitte recently reported that two-thirds of CFOs now are driving sustainability activities, putting quantifiable metrics reporting front and center. While we (mostly) support expanding sustainable business measures and work like GRI’s new G4 framework guidelines, we also caution against increased risks of reaching incomplete or only partially correct conclusions.

Why the “hazards ahead” sign? Mainly because recent history shows us that rational, objective measurement designs are significantly challenged by pervasive human idiosyncrasies, biases and blind spots.

In this series, we’ll look at some of these measurement pitfalls from largely outside the business world. By taking a meta-look at the “metrics” of metrics, we can learn from these examples to avoid or course-correct problems on the way to improved sustainable business measurements.

We start off with a major pitfall from education.

Pitfall 1: Counting what’s easy to count rather than what’s important

Einstein famously said, “Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.” This pitfall is the realm of overly emphasized, inappropriate numbers, or numbers without sufficient context and explanation to make them meaningful.

Further, the very act of establishing a metric focuses attention spans on it, and can divert attention away from other aspects of a topic that have not been “metric-fied.” It’s true that creating metrics for these other aspects, or “intangibles,” sometimes may be difficult or feel flat-out impossible. But much like the smaller, quieter sibling in the bird nest, there are consequences when non-quantifiables are out-competed and left off the measurement dashboard. Since we don’t yet fully understand which elements will prove to be significant for sustainability measurement, disregarding “difficult to measure” markers creates risk.

Here’s a real-world illustration: Billions of dollars have been spent in the last two decades to measure teacher effectiveness and student achievement in U.S. public schools, with the unintended consequences of cheating scandals, testing failure cover-ups and a disheartening lack of convincing evidence that all that money has done any good.

High-stakes standardized testing, as it exists today in the wake of the 2001 No Child Left Behind Act (NCLB), hasn’t worked as it was intended. The Atlanta Journal-Constitution does a great job of laying this out in its 2012 “Cheating Our Kids” investigative series. Under the pressure to produce improving scores, with money and jobs at stake, the cheating scandals and cover-ups we keep seeing were practically inevitable.

Image of a classroom provided by dotshock via Shutterstock.

Next page: Important but less tangible indicators

More to the point, according to many critics, is that the way metrics have been applied in the pursuit of NCLB goals doesn't necessarily measure educational achievement. Important subjects like art are “narrowed out.” Moreover, the dynamic that has been created may in fact be only revealing the results of (and even causing) thinking-deficient “teaching to the test” -- the opposite of the intended goal.

Another problematic result, has been a simplistic assignment of blame to one factor -- teachers -- as opposed to more complex, difficult, and appropriate analyses that looks at social, financial and health factors. (See the New York Times’ Mar. 2011 and Feb. 2012 articles for more on the unintended consequences of faulty teacher-assessment methods.)

Recently, we’ve seen a few attempts to use metrics in more nuanced ways that may actually help show and improve educationally meaningful student outcomes. Writing for the Times in May 2012, Joe Nocera reported that Bill Gates is funding research for teacher evaluation and improvement systems that include student test scores but don’t rely fully on them. Perhaps this is will prove to be a legitimate and less controversial step forward towards improved, classroom achievement.

After so many years of largely unproductive debate, there’s understandable temptation to avoid some difficult thinking about what “education” means and just get something done. But acceding to a simpler, superficial, testing mindset isn’t the right path. There’s hard, messy, more nuanced, and more cooperative work still to be done.

Corporate Responsibility practitioner Kevin Moss helps brings the idea here back to business sustainability in his July 2012 blog post: “We need to be sure that our focus is on what we can measure doesn’t distract us from other important but less tangible indicators.”

We’re hopeful that sustainable business professionals, who deal with similarly complicated systems, can find lots to relate to in the above example, avoid the traps, and take notice of the emergence of new “less tangible” or “less familiar” metrics indicators in their realm. Two of these are ecosystem services, being piloted by Dow and PPR, and the “Context” concept being introduced by Baue and McElroy. Now that these concepts, actually critical to our business’ long term futures, have been introduced into our field, the stakes are too high to leave them in the “Rather than what’s important” category.

We don’t think you can truly measure sustainability without them.