I am beginning to understand that the chief issues with evaluating projects and interventions in complex human adaptive systems is not whether it can be done well but whether government and non-government providers of funds accept such evaluations.
For many of our government agencies and foundations on this side of the Atlantic evaluation is another management tool to assure compliance and accountability. Of funding sources of which I am familiar linearity is a necessary given for programs in education, some health, substance abuse treatment and prevention, welfare and other human services, urban development. This attitude was captured several years ago when Susan Sclanfani, assistant secretary for vocational and adult education at ED, spoke of her aims for a math and science initiative she had taken from NSF, “We hope to be able to tell [teachers] that, if you do a, b, and c, you’ll be sure to see results.” (P. 1863, Science, Vol 307, 25 March 2005.)
Ben Ramalingam and Harry Jones in their monograph for the Overseas Development Institute discuss this split between complexity of human systems such as development and the aid, or funding, system. Here’s two paragraphs from their section, “ The challenge of applying complexity science to development and humanitarian work,” that I think are particularly relevant and balanced:
Complexity, for many, is seen as an indulgence. This has led to organisations that are increasingly rigid, risk-averse and bureaucratic; it has meant led to the prevalence of tools and techniques that are linear and simplistic in their scope and outlook (Bakewell and Garbutt, 2005), it has meant change at the level of ideas is much more likely than at the level of ground-level practices (Killick, 2005); it means that one can always predict who will do what, when and how (Seaman, cited in Kent, 2004); and it means that the adaptive learning of particular actors are short-circuited, or worse, suppressed. It means that, when mistakes are made in this complex system of governments, NGOs, UN agencies and donors, everyone points the finger at everyone else (Smillie and Minnear, 2004).
Aside from the political will, there are other more practical issues to consider. Organisations may not have the resources to bear the analytical burden of examining the systems they operate in; most reporting frameworks are geared to a linear mindset; they may not have the scope to incorporate a realistic understanding of the uncertainties of their efforts when planning and implementing projects and programmes; they may not have it in their hands to ensure that those acting to solve a problem address it in a coherent manner; the pressures of accountability to donors or the public may not allow for such uncertainties to be honestly and openly addressed. The following is a summary from a 2004 report on future dynamics of crisis in humanitarian agencies addressing the take-up of complex conceptual frameworks:
‘Coherent analytical frameworks, while a step in the direction of more effective programs, are no panacea … The challenges for agencies in adopting wider conceptual frameworks are many. Slow uptake may be associated with the burden of more complex analysis and the problems of perceived reliance on outside expertise. A framework requires an elaborate combination of qualitative and quantitative indicators and therefore agencies need to create, teach, develop, sustain, and monitor complex analysis at the national and program levels. Since not all agencies have access to expertise, affiliations with academic or training institutions may be useful. The reluctance to adopt complex frameworks may also reflect an awareness of the difficulties in using these tools for better programming … Agencies need to use these frameworks to advocate and justify more innovative programming. Similarly, donors need to be more flexible and open to tailored responses’ (Feinstein International Famine Center, 2004).
You can get the references that are cited from the document itself. It is a very good read. It is written with Toussaint Reba and John Young (October 2008). Exploring the science of complexity: Ideas and implications for development and humanitarian efforts, 2nd Ed, Working Paper 285. London: Overseas Development Institute (ODI). Go to their website, http://www.odi.org.uk/, and click on the “publications” link.
Did you catch the “uk” extension. See what I mean. It seems that most of the effort at getting us in the social change endeavors to recognize complexity are people in the UK and in Europe. Here in the States the ones calling our attention to complexity are in DOD and in public health! One of my earlier post dealt with an engineering perspective on public heath in the context of CAS. I recently found two articles tackling evaluating community situations described as complex adaptive systems:
- Olney, C. A. (2005). Using evaluation to adapt health information outreach to the complex environments of community-based organizations. Journal of the Medical Library Association, 93(4) supplement
- Hargreaves, M. B. (2007). Using complexity science to improve the effectiveness of public health coalitions. International Conference on Complex Systems, New England Complex Systems Institute, Quincy, MA.