Metric of Performance
What is performance measurement?
In order to measure performance there is a need to relate inputs with outputs.
Inputs can be measured in terms of resources used. Output should be
measured in terms of incremental value created.
It is fairly clear that there is a lot of failure in the relief and development
sector over the past 40 years, but why has their been such failure? My
experience suggests that the fundamental reason is that the output value has
been small while the input use of resources has been large ... almost $3
trillion over these years.
There is not much measurement of actual performance. While there is a lot of
dialog and analysis in the process of preparation and justification of projects
and programs, there is rather less during implementation and in evaluation.
At any rate it seems that way. Either there is a level of secrecy about
performance, or there is little information about performance. It is not clear
which of these is the true explanation ... but neither is good.
Where performance is measured.
Most implementation performance is measured, if it is measured at all, either
within the project or program or within an organization. This is not a good
place to do this measurement. Most socio-economic performance is measured
at the national level, and again, this is not an effective way of getting useful
management information.
Almost all organizations are required to publish some form of financial report
every year. These reports usually provide a modicum of analysis, usually the
allocation of their spending between “program” and administrative overhead.
Almost no organizations publish anything which shows how effective their
programs have been ... there is nothing numerical about the value arising as a
result of the programs.
But it is rare for organizations to provide much information about their
projects and programs. There are some tentative steps, like, for example, the
World Bank's AIDA database that is accessible on the web ... but this is
difficult to use and tends to move the user rapidly from very basic information
about the project to a lot of data that buries the user in material, almost all of
which adds little to the understanding of the project performance.
And while there is tentative progress, there is regression. The UNDP which
for years published the Development Cooperation Report has discontinued
the report, probably under pressure from donors who intensely dislike the
very useful information that these reports used to present.
There is little good to say about national level economic information. Its only
use is to prove or disprove economic theory ... and to the extent that it
disproves the theory what on earth do you do next.
The best place to measure relief and development performance is at the
community level. I have used community level analysis in my own work over
the years, not intentionally, but really by default ... and eventually concluded
that community is the natural level of the measurement of performance.
Value analysis
Value analysis is is a technique to understand in a systematic way whether or
not activities are value creating ... good ... or value destroying ... bad.
When this technique is applied to relief and development activities it becomes
clear that there is a lot of value destruction. Value destruction happens when
money is disbursed, activities take place, and at the end there is nothing much
of tangible value remaining.
Development ought to be about building value. In fact relief and development
spending does more of value destruction. What is apparent is that much of
what goes on in the relief and development sector ends up being value
destruction. Money is consumed, but with minimal results. While it is
apparent that a lot of relief and development expenditure ends up being
“value destruction” ... there is very little or no analysis of this going in the
major relief and development sector organizations. To the extent that anyone
knows, it seems to be a secret.
Value analysis shows a lot about the impact of investment, corruption and
favored treatment on the economy, and especially on the economy in the host
community.
Value analysis can also address the issue of debt. There has been value
destruction directly caused by bad projects, but worse the bad lending / bad
borrowing has resulted in an enormous third world debt. The only thing “for
sure” in a World Bank project, and all other loan financed initiatives, is that
the disbursements will result in debt, but nothing is for sure about the
benefits that will arise.
Value analysis also throws light on the impact of the policies of the rich
“north” on the poor “south”. While some argue that globalization is making
the world a richer and better place, the facts seem to be that for some, the
rich, this is true for for most, the poor, globalization is a disaster. What is
unacceptable is that much of this is brought about be decisions and policy
choices that ensure that these are the results.
Value Destroying Activities
Projects and programs
The relief and development sector operates largely through projects. Not
many of them are value adding, rather they a value destroying. Though the
project form of organization works well for the construction of major civil
works such as dams, roads, ports, etc., it works very badly as suppport for
work in the social sectors like health and education.
The performance of projects and programs is one of the best kept secrets in
the relief and development sector. There is a lot of focus on how much
money is disbursed, but very little about the value that is accomplished using
the money. Without measurement and effective oversight, then project
performance becomes unimportant.
Value Destroying Projects
At one point in my career I was retained by UNDP to evaluate (desk review) part of
their project portfolio. I was not asked to repeat the exercise, and my work was
probably totally ignored. But the overriding conclusions was that there was much
disbursement of money for work that sounded worth doing, but the activities and the
results rarely did anything like what was projected.
All the projects were worth doing, based on the project proposals, but feedback of
performance information to the head office showed that while the money had been disbursed,
and most of the project activities completed, rarely anything of substance had been achieved. The
costs were there, the durable value was not.
Though it was very clear that disbursement of money had been done for work that sounded worth doing, it was equally clear that real development progress was not being achieved.
Value Destruction in South Sudan
For years and years UNDP funded a small capacity building project in South Sudan.
The project consisted of a single CTA working in an advisory capacity with the local
government. The expenditures were a generous UN rate salary together with normal
living and transport allowances ... perhaps 50 times what he would have earned in a
government job in his home country. The CTA had an office in the government
administration building and by all accounts was “on seat” around 10 hours per week.
The project goals were never achieved, but every year someone in the local
government wrote glowing letters asking that the project be continued, and it was,
over and over again, despite considerable staff objections.
Nobody knows who was paying who, but it is almost certain that this was the underlying problem.
The problem of project performance has been systemic. Why was it that a
project with almost 100% value destruction in South Sudan kept on going
year after year ... the funding agency had official government letters saying
the project was important, and that was all it took for the funds to be
reallocated year after year. Not a great outcome.
A World Bank / DANIDA funded shrimp project in Yemen (YAR) at
Hodieda is another example of value destruction going on because of a bad
project. Almost everything about this project was wrong, but the
disbursement kept on going ... even after a supervision mission had
recommended an immediate end to the funding.
Shrimp Project in Yemen (YAR)
I worked with a World Bank mission in Yemen (YAR) to help assess progress on a
shrimp project based in Hodieda. As a former CFO with an international company in
fisheries (shrimp) I know something about the fisheries business and I was shocked by
almost everything I learned about the project .
The World Bank and the Yemen authorities had taken perfectly good resources and
converted them into into a operational liability and created a big debt repayable to the
World Bank. This was an almost 100% value destruction performance.
Similar project performance is relatively common. It is something the World
Bank and others in the relief and development sector should be ashamed of.
Having a failure once in a while is understandable, but failing almost all the
time is unforgivable.
Why on earth did the World Bank, UNDP and the broader relief and
development community do little or nothing for years about pathetic project
and program performance.
Foreign direct investment (FDI)
How has foreign direct investment (FDI) turned into a source of value
destruction for the “south”. FDI should be a winner for the beneficiary
country, the communities hosting the investment activity and the people. But
more often than not, FDI is a disaster for local people, the host communities
and even the country.
It is apparent that experts from the “north” have been able to negotiate and
get agreements with the “south” leadership that have been very beneficial to
the corporate “north” and have done little or nothing for the people of the
“south” and arguably have created nasty long term environmental liabilities.
Extractive industries
The value chain for the extractive industries end to end from the natural
resources to the consumer is not usually presented for publication and easy
access. In spite of big investment and great profits, it seems that there are
huge durable value problems in many of the host communities while the
“north” corporations and their stockholders are enjoying stupendous profits.
Value Destruction in a Mining Enterprise
An investor comes into the country to open a mine. The host community has an area
that is rich in ore. The investment is estimated to be $500 million. The host country
does everything to welcome the investor, and the investor arrives.
A large amount of mining equipment arrives ... the reported cost is around (say) $400
million and work commences. Some local staff are employed and trained, but the
local payroll is tiny relative to the expatriate payroll. The mine does well and soon
over a of ore has been removed, processed and exported. The investor is well
pleased.
The mine is now ready to be transfered to local ownership. In reality the mine comprises nothing more than a
huge hole in the ground with the best ore already gone, together with worn out mining equipment that is most likely
almost fully depreciated. Keeping this mine profitable is going to be a challenge no matter how low the transfer price.
While the community has had some economic activity for a few years, it now has a dirty polluted area with not much future. Most of the mineral value has been extracted and the community
has not a great lot to show for it.In this typical scenario the investors probably make $100 of profit for every $1 of value that is realized in the community. Worse, in many cases, mining causes the pollution of ground water and turns FDI into a disaster for the local community ... with absolutely no effective recourse for community people.
This should not be possible, but money talks. There is usually an elite group
in the “south” that is also able to profit mightily from foreign direct
investment. Rigorous value analysis shows that, more often than not, FDI
extracts value from the host country rather than putting value into the local
economy ... and combining this with the pattern of bribery and corruption,
there are big value distortions when corruption is out of control.
Infrastructure construction
Infrastructure investment is very much needed ... but does infrastructure
building in the “south” need to be as expensive as it is. Where there is
international FDI the driver is the maximum of corporate profit, rather than
being the the maximization of durable socio-economic benefit.
There are hundreds ... no, thousands ... of PhD level economists at the
World Bank and around the relief and development sector organizations yet
they do not seem interested or capable of doing the analysis to show that there
is an optimum way of doing construction in the “south” to get the maximum
of socio-economic benefit. Instead for many years now the “south” has been
getting infrastructure projects that are designed in a way that makes profit at
the expense of tangible socio-economic improvement. Maybe Bechtel
benefits, but the “south” does not.
Coffee, cocoa, tea, cashew, palm oil, etc., etc.
There is something fundamentally wrong with the value chain. At retail in the
“north” the prices have gone up along with everything else. At the farmgate,
however, the prices are more or less the same as they were years and years
ago ... and costs have risen for farm inputs and everything else a family has to
buy. Net net the farmers doing coffee, cocoa and tea are increasingly poor.
What is the problem? In large part the World Bank got it terribly wrong 30
years ago when they were predicting rapidly increasing demand and prices for
all these items.
World Bank's Rotten Projections
Over and over again in the late 1970s and early 1980s I objected to the World Bank's
rotten commodity projections. They had a department that occupied itself with price
projections and a new-fangled computer program that gave them the definitive
answer about the future ... but it got it wrong ALL THE TIME.
I had recently moved on from being a corporate CFO who needed to get price
projections for our products right in order to have profits ... and in the areas I knew
very well the World Bank was getting it very wrong.
Over a period of years, I worked in a variety of agricultural project appraisals and supervisions for the World Bank and IFC ... and
during this work most every projection made by World Bank staff was considered wrong by the
expert independent consultants involved. Using World Bank projections made with their staff projections, these
projects were beneficial enough for approval ... while using the independent consultants' expert views, the projects were not good enough. In every case the World Bank view of the
world was the one that got used with perfectly predictable results.
Parastatals, FDI and privatization
Parastatals, FDI and privatization provide another set of sad stories of value
destruction for the “south”. A lot of productive foreign corporate investment
was nationalized in the early days of independence, which was
understandable. Much of it turned out to be a mistake, and parastatal
performance deteriorated over the years. Subsequent privatization of these
entities with foreign investors has recreated some of the same exploitive value
chains that existed in the colonial era ... surely this could have been
anticipated by the relief and development sector experts. Not surprisingly the
good parastatals were sold off easily ... and the dogs remained for a long time.
Almost all aspects of these programs ended up doing value destruction for the
“south”
Trade
In the global mercantile era of the 18th and 19th centuries, trade was bigger,
fairer and more free than it is today.
For some, trade is easier now than in years past. Mainly it is the corporate
“north” that has easier trade, but the “south” is stuck. Terms of trade are
almost universally worse than they were years ago, especially for agricultural
commodities. International agreements are long and complicated and rarely
do anything of substance that benefits the “south” while having huge value for
the corporate “north” ... worse, with bad results for the “south”.
Workshops and conferences
All workshops and conferences have a cost, but not all of them have value. It
takes a substantial expenditure to run a workshop or conference both on the
part of the organizers and the participants.
Some workshops and conferences are popular because the participants are
paid to attend, and receive travel and per diem allowances as well. These
payments can be substantial relative to the pitifully low salaries paid to civil
servants in most of the countries in the “south”, including quite high positions
in the civil service system. For these people there is a money value that is
quite tangible, but is an inappropriate justification for the workshop or
conference.
Workshops and conferences may have a high professional value in facilitating
networking and sharing professional experience. Some professionals may get
significant value from this, and where this is going on, the costs may well be
justified. But more often than not, workshops and conferences do little more
than cost money and do not very much. Even so, donors seem to be willing to
support them because they are an easy and quite visible way of showing that
there is some tangible activity going on, and they can use this in public
relations efforts to feed the media and other stakeholders.
Without metrics that measure relief and development performance, the
majority of workshops and conferences are never evaluated to see how big the
costs are in relation to any tangible relief and development benefits.
Research and studies
All research and studies have a cost, bit not all of them have value. They
represents a very substantial investment that is almost total value destruction.
There is a huge archive of research and studies which is very difficult to access
and use.
A Study of Studies in Lesotho
I was part of a team doing research on the rural sector in Lesotho in
1987. The TOR called for us to review the studies that had already been
completed and not carry out any “new” work. When we arrived in
Maseru to start the work, UNDP had collected for us most of the reports
that we should review. The collection was stacked against a wall of the
conference room and measured some 3 feet high and some 12 feet long.
The cost of preparing these reports was probably upwards of $20 million,
a substantial amount in the context of Lesotho. The practical “on the
ground” investment in rural sector development was tiny. Much less than
the cost of all the studies. Value destruction in progress.
Research and study have no value in themselves. Their value is only realized
when the results of the research and study are incorporated into some
practical activity that improves socio-economic conditions in some tangible
and durable way.
The academic community
The academic community has a significant involvement in the relief and
development sector, but it is not clear that it is value positive. Certainly it
ought to be, but looking at the performance of the relief and development
sector it raises questions about the role of the academic sector in this
performance.
It looks very much as if what has been “taught” about relief and development
and the operation of all the institutions and activities might not be right. In
fact it might be significantly wrong. This is a serious problem and one that
does not seem to be getting much attention, if any.
Teaching Relief and Development Methodology
I first realized that academia was part of the relief and development performance
problem when a USAID staff member retired and soon afterwards was teaching a
course at a well known university on USAID's Logical Framework. This happened
soon after I had been working on a USAID project and had been faced with a logical
framework that ensured the project would fail, unless the project, based on this
logical framework was redesigned.
I also ran into a similar problem where the World Bank was recruiting just graduating
students and then training them on how the World Bank did things ... ensuring that
the World Bank methodology was applied, and ensuring that the World Bank
methodology would never be the best that it could be and should be.
It also looks as if the “research” being done in the academic arena is of very
little importance. Most research is done as a “requirement” for some academic
credential, and though this is important in the academic context for the
individual, it is not of much importance in the mainstream of relief and
development. To the extent that this research is often funded using grants that
come from budget lines that ought to be used for important work in relief and
development, this is an inappropriate diversion of scarce relief and
development resources.
Academia should be an important and valuable component of the relief and
development arena, but it does not appear that they are adding value. As in
most things, there are exceptions, but the role of academia in the “north”
putting some really valuable into the mix seems still-born. From my
perspective as someone with an interest in management information, the
typical academic and statistical approach to information is dangerous, and not
an appropriate way of proceeding. Rather it would be better if a little bit of
academic effort that costs a lot were to be replaced by a lot of good relief and
development information that would cost rather little.
Valueless political rhetoric and press releases
There is a lot of political posturing and press releases, mainly about plans, or
reports that have been prepared, but never about moneys disbursed, activities
undertaken and results achieved.
This is not surprising because many activities do not achieve much in the way
of result. In order to have relief and development success, resources have to
reach the intended beneficiaries, and be funding activities that are useful for
the beneficiaries.
Value Destruction Aggravated with Debt
Debt is a result
Debt is a result of decisions and actions. Unsustainable debt has come about in
the “south” because the lending has been bad and the use of resources has been
bad.
The “south” suffers from a shortage of investment ... too little infrastructure
... too little of public services ... so how come there is so much debt? There
are some simple answers: (1) money has been lent badly by the World Bank
and other lenders; (2) the money has been used badly by the borrowers,
primarily borrowing governments and their crony's; and, (3) the structuring
of the loans so that they are denominated and repayable in a basket of “hard”
currencies.
Back in the 1970s when the international financial system was in turmoil after
the oil shocks, Walter Wriston, the CEO of Citibank from that era summed
up the financial opportunity saying something along the lines that developing
countries “needed the money, paid high interest, and could not go bankrupt”
and the international financial community was happy to oblige. Many
commercial banks eventually lost some in the eventual restructuring and debt
forgiveness that subsequently has taken place, but overall they did very, very
well. The World Bank participated in the flurry of lending, but in the case of
the World Bank, did not do much to forgive its debt when the “projects”
failed to deliver much “development”.
The whole debt discussion is a cruel hoax on the world's poor and vulnerable.
Without other interventions these people remain poor and vulnerable. The
“south” needs to get financing for the things that it needs so that it can do
critical things on a timely basis and on a meaningful scale.
Some of the relief and development experts in the “south” understand the
issues, but do not often have a forum for their views to be heard and taken
into consideration.
Minister in Namibia Had It Right
When I was working in Namibia I was present at a meeting of the donors with the
Minister of Finance of Namibia. The purpose of the meeting was to move forward the
country's strategy for development and how it would be funded. As is the custom in
these meetings all the donors had an opportunity to say their piece.
When it came to the turn of the Commonwealth Secretariat, represented as I recall
by a Canadian, Namibia was encouraged to gain membership of the Breton Woods
organizations (World Bank, IMF et al) so that they could borrow for development.
The Minister asked some pertinent questions including about rates or interest and the
currencies of the loans, and was told that the interest rates would be low, and that the
currency would be a basket of currencies like the US dollar, the Canadian dollar, the
Swiss franc, and so on.
The Minister then asked about project performance from such financing, and
specifically what rate of return and what cash flow would be projected. The donor
representatives present started to be a little uncomfortable, and no real answer was
provided.
The Minister went on saying more or less the following: “I think you know, our
currency is the South African Rand. This is a currency that is being rapidly devalued
because of the economic sanctions against South Africa ... which we support ... but it
does mean that our currency is devaluing at a rate of more than 20% per annum. If
we borrow, we have a real interest cost of 20% plus whatever the low interest that is
being charged by the lenders. And up to now, I have not seen any development
project that is going to be able to support debt financed development in Namibia”.
This was one of the best summaries of a critical development dilemma. The Minister
was absolutely on target with his analysis. Sadly, I am not sure the donor
representatives really understood. Most were not as well educated and experienced as
this Minister.
Debt cancellation, restructuring ... whatever
There has been around 20 years of dialog about debt forgiveness and debt
restructuring or rescheduling or whatever ... but much less discussion of how
and why the debt came into existence in the first place, and specifically who
are the responsible people involved. The leadership of the relief and
development sector has never taken any meaningful action to get at the
culpability of the various actors and to hold them accountable.
There are many advocates for debt cancellation or some form of radical
restructuring. There have been “agreements” about this over and over again,
but also conditionality. There has been some debt cancellation, but only after
the beneficiary country has jumped through hoops to satisfy the terms and
conditions. The way the process works is typical of the relief and
development sector, a huge amount of study and reports and meetings and
negotiations before any agreement and any useful action.
The media and the advocates for debt reduction have periodic press bonanzas,
but for poor people in poor countries it is all but invisible. The 2005 G8
Summit hosted by Prime Minister Tony Blair in Scotland highlighted
commitments about debt reduction, but for all practical purposes, not very
much can be seen from the perspective of the poor communities. There are
headlines in the “north” nothing to be seen in the “south”. The reorganization
of the “debt” is not more more than a bookkeeping exercise with far too little
of financial substance.
No Sustainability
Value destruction is unsustainable
The relief and development sector has done very little that is sustainable in the
past few decades. After years of relief and development sector
mismanagement none of the poor countries in the “south” can operate
independent of its donors. Very little can be sustainable without a complete
rethink of how relief and development operates.
There are many areas where sustainability should be accomplished: (1) in the
family; (2) in the community; (3) in the organization; and, (4) in the country.
The idea of sustainability is clear in the corporate for profit organization,
because if the company fails and goes bankrupt, it will be liquidated. In the
relief and development sector, there is no equivalent.
Environmental sustainability
Environmental sustainability is talked about, but rather little is being done
about it. There is increasing concern in the public dialog, but this is not
translating much into actions that make a difference.
Over the past 30 years plus I have taken flights in small planes over various
parts of the “south” ... and places that used to be covered in trees are now, for
all practical purposes, desert. Trees go, and soil erosion increases
dramatically. Less trees, more global warming. Less trees, and a fuelwood
crisis for poor communities.
The rich “north” has cleaned up its rivers over the past 50 years. The air looks
cleaner, but colorless gases increase as energy consumption increases and
more and more industrial materials are produced. Air pollution is notorious
in urban areas of the “south” and especially the richer countries in the “south”.
Why is this? In the north of England in my youth there was a saying “Where
there's muck, there's money”. This is still very much true today, except that it
is also unsustainable ... and will get much worse.
Sustainability of the community
While the countries of the “south” may not be sustainable without donor
involvement, communities in the “south” have had no choice. They may not
have progressed, but they have kept going with little or no external support
from either their own government or the international community.
From time to time, events have impacted communities in terrible ways.
Natural disasters like earthquakes and tsunami waves have wiped out
communities, as well as drought and famine. Violence in its many forms also
has a big affect on communities. It is in the community that the impact of
failed relief and development is at its most visible. It is not pretty.
Organizational sustainability
The biggest challenge for organizations in the relief and development sector is
to ensure that their budgets get funded, and that the staff remuneration is
covered without too much staff attrition or too much challenge around
pensions and other long term benefits. Keeping the organization sustainable is
“Job 1” in any organization, and impact on the relief and development process
is subservient to this.
Keeping donors happy is an essential aspect of organizational sustainability,
and making reports to donors is very important for this reason. This has,
however, little to do with relief and development performance, and the use of
funds to get the most relief and development benefit.
Some of the ways in which donors are “kept happy” are of a questionable
nature. The revolving door between donor organizations and the community
of primary recipients is wide open.
An organization expecting to be funded by USAID needs to structure itself to
suit USAID. One that gets its funding from World Bank projects has a
structure that suits the World Bank. The same goes for organizations looking
to get funding from the various philanthropic foundations.
And USAID must organize itself to get its funding from the Congress of the
USA, and similarly for other bilateral aid agencies, and the World Bank from
its funding stakeholders. Mostly this means that administrative resources go
into study of the funding sources ... and it can be said that the organizations
that are getting a lot of funding are taking care of this part of the business.
Organizations do not have a priority to look “down” towards the intended
beneficiaries. There is much less imperative to have a strong relationship with
the downstream organizations, and little or no incentive to invest money and
effort to be strong on the ground. Clearly there has to be enough of a field
presence to be able to stay in the front rank of donor interest, but best to have
data and evidence that spins well rather than hard cost and performance
information that could prove embarrassing.
Project sustainability
Most projects play lip service to sustainability. It is almost always talked about
in the project documentation, but the reality of sustainability is not given
much consideration.
Almost by definition a typical World Bank, UN or donor funded project
cannot be sustainable. Almost everything that makes up the project mode is
artificial relative to the host economy, and it is rare for the project to be able
to transition into the general economy.
National sustainability
The problem of national sustainability has not been taken seriously by the
relief and development community. The arguments about relief and
development performance tend to ignore the crisis of national accounts and
the failure of the national economy to be viable in its relationship with the
global society.
This is a silent crisis that has been permitted because it really has had little
impact on anyone except the people of the “south”. Why is it that the
exchange rate for almost every “south” currency is a fraction of what is was 40
years ago?
It should matter. But when the financing institutions of the relief and
development community denominate their loans in external hard currencies,
what goes on with the local currency does not matter very much. If you are an
ordinary person in the society, however, it can have a terrible impact.
|