Tema: Styring og evaluering

Transkript

1 Nr. 13 Januar 2007 Tema: Styring og evaluering

2 Side 1 Leder - Nye styringsformer og evaluering Af Steffen Bohni Nielsen, Rambøll Management, medlem af bestyrelsen i Dansk Evalueringsselskab Dansk Evalueringsselskabs bestyrelse ønsker vores medlemmer et godt nytår. Det gør vi med et nyt tiltag. Evalueringsnyt vil fremover have en leder fra et medlem i bestyrelsen. Såvel leder som indhold i denne udgave af DES Nyt beskæftiger sig med samspillet mellem nye styringsformer og evaluering. Den største reform af den offentlige sektor i over 30 år er nu en prekær virkelighed. Det er i den virkelighed, at mere eller mindre skrøbelige organisatoriske og menneskelige konstellationer skal bevise deres værd. Valg af styringsform og anvendelse af evaluering er helt centrale elementer i forsøget på at skabe nye institutioner, der skal levere produkter og services til borgerne hurtigst og bedst muligt. I dette nummer af Evalueringsnyt præsenterer vi papers fra tre spændende plenumoplæg fra evalueringskonferencen 2006 som på forskellig vis søger at finde evalueringens rolle i styring, i den offentlige debat og i den politiske beslutningsproces. Hanne Foss Hansen beskriver hvordan evaluering anvendes i styring og styring sker igennem evaluering. David Hunter argumenterer for, at en nødvendig forudsætning for, at offentlige organisationer kan levere værdi til borgerne, er at evaluering er et snævert integreret redskab i styringen. Dette forudsætter, skriver han, at organisationerne har den nødvendige kapacitet til at fortolke, udvikle og indsamle evalueringsviden. Ray Rist reflekterer over evalueringens rolle når dens resultater kommunikeres til og anvendes i det offentlige liv. Han påpeger, at evaluering som disciplin er dårligt rustet til den simplicitet, med hvilken evalueringens resultater omtales i den offentlige debat. En udfordring evaluerings fællesskabet ikke kan lade overhørig. Husk, redaktøren modtager gerne nyheder inden for evalueringsverdenen, fx. om nye bøger, metoder, kurser, uddannelser, konferencer mm.

3 Side 2 Redaktion Redaktør: Jesper Pedersen, jp@discus.dk, telefon Baggrundsgruppe: Dorte Kabell, medlem af bestyrelsen i Dansk Evalueringsselskab Bianca Albers, medlem af bestyrelsen i Dansk Evalueringsselskab Marie Louise Møller, DES sekretær Nyheder, debatindlæg og artikler sendes elektronisk til Jesper Pedersen Deadline Deadline til det næste nyhedsbrev er d. 31. januar Forside Billeder fra årets evalueringskonference: Øverst til venstre: Ray C. Rist, Verdensbanken Øverst til højre: Hanne Foss Hansen, Københavns Universitet Nederst fra venstre: David E.K. Hunter, Hunter Consulting og Steffen Bohni Nielsen, Rambøll Management

4 Side 3 Indholdsfortegnelse Leder - Nye styringsformer og evaluering...1 Redaktion...2 Deadline...2 Forside...2 Nyheder...4 Ny sekretær i DES...4 Udskiftning i bestyrelsen...4 Ny baggrundsgruppe til nyhedsbrevet...4 Nyt fra universitetsverdenen...6 Artikler...7 Evalueringsstyring...7 Bringing Evaluation into Public Life...13 Daniel, Nebuchadnezzar, and the Birth of Government Program Performance Management...18 Konferenceoversigt...23 Om Dansk EvalueringsSelskab...24 Medlemskab...24 Bestyrelsen...25

5 Side 4 Nyheder Ny sekretær i DES Dansk Evalueringsselskab har pr. den 29. november ansat en ny student til sekretariatet. Hun hedder Marie Louise Møller og læser HA på Roskilde Universitetscenter. des@eva.dk Udskiftning i bestyrelsen Hanne Kathrine Krogstrup har valgt at træde ud af bestyrelsen i DES. Nyt medlem er Tim Jeppesen. Han er direktør i Det Kommunale og Regionale Evalueringsinstitut (KREVI). Ny baggrundsgruppe til nyhedsbrevet Evalueringsnyt har fået en baggrundsgruppe, der har til opgave at fungere som støttegruppe for redaktøren. Baggrundsgruppen består af Dorte Kabell, Kabell Konsulting APS Bianca Albers, Styrelsen for Social Service Marie Louise Møller, DES sekretær Jesper Pedersen, Redaktør Gruppen modtager gerne forslag til forbedringer af Evalueringsnyt. Følgende initiativer for 2007 er besluttet: De næste 4 numre i 2007 kommer til at omhandle følgende: o Nr. 14: Temanummer om Performance Management, Audit og Evaluering. o Nr. 15: Temanummer om arbejdet med indikatorer i evaluering. o Nr. 16: Temanummer som præsenterer årets evalueringskonference. Nummeret skal fungere som appetitvækker til konferencen. o Nr. 17: Rapportage fra årets evalueringskonference. Til hvert temanummer udarbejder et bestyrelsesmedlem en leder. Der udarbejdes en konferenceoversigt over internationale evalueringskonferencer. Der tages kontakt til universiteterne med henblik på at indhente evalueringsnyheder fra universitetsverdenen. Fremover skal der være mere synergi mellem aktiviteter i DES og indholdet i Evalueringsnyt. F.eks. kan et tema i Evalueringsnyt være grundlag for et gå-hjem-møde eller et halvdagsseminar i DES.

6 Side 5 Nyt fra universitetsverdenen Kursus i evaluering - modeller og metoder Syddansk Universitet udbyder et kursus i evaluering som finder sted fra d. 29. januar til d. 1. februar 2007 på Hotel Koldingfjord i Kolding. Målet med kurset er: at give deltagerne en forståelse for hvad evaluering er, hvilke muligheder det giver og hvordan evaluering anvendes, at give deltagerne mulighed for at vælge, hvilke model der er anvendelig til en given problemstilling, at give deltagerne kompetencer og værktøjer til planlægning, fokusering og udførelse af evalueringer, og at give deltagerne systematisk viden om, hvordan man metodisk arbejder med spørgeskemaundersøgelser og interviews. Kurset henvender sig til alle, der skal planlægge og gennemføre evaluering i deres organisationer. Link: Konsekvenser af evaluering i folkeskolen Ved Institut for Statskundskab, Syddansk Universitet er Carsten Strømbæk Pedersen den 1. december 2006 blevet ansat som kandidatstipendiat. Carsten skal forske i konsekvenserne af evaluering i folkeskolen. Ph.D.-afhandlingen vil indgå som en del af det internationale forskningsprojekt "Fabricating Quality in European Education: Quality Assurance and Evaluation in Danish, English, Finnish, Scottish and Swedish compulsory Schooling".

7 Side 6 Artikler Artikler sendes elekronisk til jp@discus.dk 4 Retningslinjer: Artikler skal være kortfattede, det vil sige max 5 sider (linjeafstand 1,3) Artikler skal omhandle evaluering Artikler må ikke være anstødelige og diskriminerende Artikler må ikke have et markedsføringssigte Om nødvendigt bedes bestyrelsen foretage en vurdering af disse forhold Evalueringsstyring Af Hanne Foss Hansen, professor på Institut for Statskundskab på Københavns Universitet. Begrebet styring kan defineres som påvirkning, der fører til ændret adfærd. I styringsteorien er der en lang debat om styreformer. Der er udviklet typologier, der skelner mellem forskellige styreformer på basis af variationer i virkemåde, styringsretning og sanktionstype. Styring kan f. eks. ske via bureaukrati (hierarki og regler), marked, demokrati (fælles meningsdannelse), normer (værdier, kultur) samt information. Der er ligeledes udviklet typologier, der skelner mellem forskellige styreformer på basis af hvilken genstand styringen rettes mod. Der tales f. eks. om inputstyring, rammestyring, målstyring, resultatstyring, kvalitetsstyring, kontraktstyring, økonomistyring, personalestyring, IT-styring m.v. Udviklingen i disse år på det teoretiske såvel som det praktiske plan gør det relevant at tilføje endnu en styreform evalueringsstyring. Evalueringsstyring kan defineres på to måder og derved indskrives i begge de ovenfor nævnte typologier. For det første kan evalueringsstyring defineres som styring via evaluering. Fokus er i denne sammenhæng på den form for styring, hvori evaluering indgår som et betydningsfuldt element. Fokus er samtidig på, hvordan adfærd ændres som følge af evaluering. Evalueringsstyring i denne betydning vil være i fokus nedenfor. Det skal imidlertid også nævnes, at evalueringsstyring kan defineres som styring af genstanden evaluering. I denne sammenhæng sættes der fokus på, hvordan evalueringsprocesser og -systemer kan designes og implementeres. Da evalueringslitteraturen er righoldig i denne sammenhæng, og pladsen her knap, skal dette tema ikke forfølges her. Evalueringsstyring defineret som styring via evaluering er under kraftig udbredelse i disse år. Tænk for eksempel på uddannelses- og sundhedssektoren. I uddannelsessektoren evalueringsstyres der på mange niveauer. Folkeskolen evalueringsstyres via krav om ud-

8 Side 7 vikling af evalueringskultur. Kommunerne skal f.eks. rapportere om kvaliteten hvert år. Samtidig gennemføres der evalueringsprojekter lokalt såvel som sektorielt bl.a. i regi af Danmarks Evalueringsinstitut (EVA). Senest er der også oprettet en styrelse - Styrelsen for Evaluering og Kvalitetsudvikling i Grundskolen. Styrelsen har blandt andet til opgave at medvirke til at udvikle evalueringskultur samt dokumentere resultater, herunder udarbejde og administrere de nye obligatoriske nationale tests. På CVU området gennemfører EVA akkrediteringer af både uddannelser og institutioner. For at blive akkrediteret skal institutionerne kunne dokumentere, at de lever op til en lang række kriterier vedrørende blandt andet faglighed, organisation, undervisere og kvalitetsudvikling. På universitetsområdet gennemfører EVA auditeringer. Her skal institutionerne kunne dokumentere, at de har udviklet kvalitetssikringssystemer. På universitetsområdet er der også planer om at oprette en akkrediteringsinstitution. På universitetsområdet udvikles evalueringsstyringen i et internationalt netværk, den såkaldte Bologna proces. Samtidigt udarbejdes og offentliggøres der stadigt flere rangordninger, der påtager sig at vise, hvordan danske universiteter klarer sig i et globalt perspektiv. Endelig er der på uddannelsesområdet en anden ordens evalueringsinstitution under tilblivelse. Ministeriet for Videnskab, Teknologi og Udvikling, Undervisningsministeriet samt Danmarks Pædagogiske Universitet har besluttet at oprette et såkaldt Clearingshouse. Clearingshouse får til opgave, at udarbejde og formidle systematiske forskningsoversigter, hvori resultaterne af allerede foreliggende evalueringsforankret viden af god kvalitet syntetiseres. Håbet er, at initiativet skal føre til, at uddannelsespolitik og -praksis fremover i større omfang bliver evidensbaseret. Også i sundhedssektoren er evalueringsstyring udbredt. En del sygehuse har i de seneste år gennemgået akkrediteringsprocesser. Indenrigs- og Sundhedsministeriet, Sundhedsstyrelsen, Amtsrådsforeningen og Hovedstadens Sygehusfællesskab har i forlængelse heraf sammen udviklet Den Danske Kvalitetsmodel (DDKM), hvilket har ført til etablering af Institut for Kvalitet og Akkreditering i Sundhedsvæsenet (IKAS). Instituttets opgave er at udarbejde kvalitetsstandarder og iværksætte målinger af hvorvidt disse opfyldes. Målingerne skal gennemføres som selvevaluering efterfulgt af uvildig ekstern akkreditering. IKAS har indgået en rådgivningsaftale med en engelsk akkrediteringsorganisation. Der er udarbejdet en række standarder vedrørende generelle temaer (som f. eks. visitation, modtagelse og intensiv behandling), organisatoriske temaer (som f. eks. patienttransport, kvalitetsstyring og hygiejne) samt sygdomsspecifikke temaer (f. eks. apopleksi, brystkræft og mavesår). Et tidligere kvalitetsudviklingsinitiativ benævnt Den Gode Medicinske Afdeling er blevet integreret i DDKM, mens Det Nationale Indikatorprojekt, der blev etableret i 2000,

9 Side 8 skal fungere som dataleverandør til IKAS i forhold til en række af de sygdomsspecifikke temaer. Udover disse initiativer har Amterne, H:S samt Indenrigs- og Sundhedsministeriet via Enheden for Brugerundersøgelser i en årrække gennemført landsdækkende undersøgelser af patienters oplevelse af landets sygehuse. Enheden har også gennemført et stort antal lokalt iværksatte evalueringer. Senest har indenrigs- og sundhedsministeren besluttet at indføre en national karakterskala for sygehusene. Sygehusene tildeles efter engelsk forbillede stjerner på samme måde som restauranter tildeles smileys. Sagen har været på dagsordenen i nogen tid, men der har øjensynligt været en del uenighed om, hvordan karaktererne skal udregnes. Endelig findes der også anden ordens evalueringsinstitutioner på sundhedsområdet. Siden 1993 har Nordic Cochrane Center, der indgår i det verdensomspændende Cochrane samarbejde, udarbejdet og formidlet systematiske forskningsoversigter, der syntetiserer foreliggende viden om givne behandlingsformer, ligesom skiftende organisatoriske enheder siden 1996 har udarbejdet medicinske teknologivurderinger, der udover klinisk evidens også inkluderer organisatoriske, ledelsesmæssige og økonomiske evalueringskriterier. Medicinsk teknologivurdering udføres i dag i regi af Center for Evaluering og Medicinsk Teknologivurdering (CEMTV), der er en enhed i Sundhedsstyrelsen. I både uddannelses- og sundhedssektoren er det karakteristisk, at de nævnte tiltag til evalueringsstyring er iværksat ved siden af de evalueringselementer, der længe har været institutionaliseret i mere klassiske organisatoriske styreformer som f.eks. mål- og resultatstyring samt kontraktstyring. Der udvikles således stadigt flere evalueringslag i de to sektorer, herunder i stigende omfang lag af international karakter. I øvrigt kunne eksempler fra andre sektorer have været trukket frem. Også på det sociale og arbejdsmarkedspolitiske område stilles der i disse år større krav om dokumentation af resultater. Monitorering, benchmarking og evidenstiltag vokser frem. Det samme gælder på forskningsområdet og i forlængelse af strukturreformen skal kommuner og regioner kigges efter i sømmene af det nyoprettede kommunale evalueringsinstitut, KREVI. Hvilke potentialer knytter sig til styreformen evalueringsstyring? Hvilke former for dynamik kan evalueringsstyring skabe? Og er der risiko for at der opstår uønskede konstitutive virkninger? Evalueringsstyring har blandt andet til formål at fremme gennemsigtighed om resultater og kvalitet.

10 Side 9 Øget gennemsigtighed kan påvirke de offentlige organisationers medarbejdere og ledelse. Via læring og sammenligning med søsterorganisationer kan medarbejdere og ledelse få et afsæt for at forbedre deres virksomhed. Øget gennemsigtighed kan også påvirke brugernes adfærd. Brugerne kan via evalueringsbaseret information klargøre deres forventninger og krav til de offentlige organisationer, og på områder med frit valg kan de søge hen mod de organisationer, som de opfatter som succesfulde. Evalueringsstyring kan derved fremme organisatorisk konkurrence. Endelig kan øget gennemsigtighed påvirke de offentlige organisationers ejere altså principalernes adfærd, fordi evalueringsstyring giver principalerne et grundlag, hvorpå de kan belønne og sanktionere. En forudsætning for at opnå de nævnte potentialer er, at der er tillid til den evaluative information. Og tillid kræver blandt andet opdateret information. Megen evalueringsstyring er af indirekte, kognitiv karakter, fordi evalueringsstyring søger at påvirke adfærd via påvirkning af aktørernes opmærksomhed og vidensgrundlag. Et centralt spørgsmål bliver derfor hvilke aktører, der sikres adgang til, selv opsøger, har tillid til, forstår og ønsker at benytte sig af evaluativ information. Der er mange tænkelige barriere. Evalueringsstyring kan også udformes som en mere direkte styreform, som det er tilfældet, når resultatkrav f. eks. kobles direkte til bevillingstildeling eller graden af ledelsesmæssig autonomi, eller når manglende dokumentation af resultater kobles til sanktioner som f.eks. afskedigelse af ledelsen. Vi har desværre ikke megen systematisk empirisk baseret viden om, hvilke effekter evalueringsstyring har. Men evalueringsstyring synes at være en ganske kraftfuld styreform. Dens dagsordenssættende og opmærksomhedsskabende funktion, der ofte forstærkes via mediernes interesse for evalueringsbaseret information, kan have både funktionelle og dysfunktionelle effekter. På den funktionelle side kan den ovenfor nævnte gennemsigtighed forøges. Udover at dette kan skabe læring, øget organisatorisk konkurrence samt effektivisere den hierarkiske styring, kan øget gennemsigtighed i sig selv være en demokratisk gevinst. På den dysfunktionelle side kan flere typer af dynamik tænkes. Baseres evalueringsstyring på (for) mange kriterier, hvilket der bl.a. er eksempler på i uddannelsessektoren, svækkes bevidstheden om hvad der er det centrale. Det vigtige forsvinder så at sige i mængden af kriterier. Baseres evalueringsstyring på komplekse områder modsat på (stærkt) forenklet information, hvilket kan ske i forbindelse med rangordning og karaktergivning (stjerner, smileys), er spørgsmålet, om ikke aktørerne vildledes mere end de vejledes.

11 Side 10 Hertil kommer risikoen for decideret uønskede konstitutive virkninger. Erfaringer fra andre lande viser, at lærere f.eks. fristes til at undervise eleverne, så de kan bestå de indførte test, i stedet for at arbejde på at udvikle mere bæredygtige kompetencer hos eleverne. Ligeledes rapporteres der om, at hvis et evalueringskriterium på hospitalsområdet er, hvor længe der går fra en patient kommer ind til vedkommende har mødt en fagprofessionel, så fristes hospitaler til at ansætte personale ( hello nurses ), hvis rolle alene er at hilse på nyindkomne patienter uden varetagelse af en egentlig sygeplejefaglig funktion. På tilsvarende vis kan man forestille sig at offentlige organisationer, der evalueringsstyres, fristes til at selektere mellem potentielle brugere, således at brugere, der bidrager til pæne evalueringsresultater (de dygtige elever, de helbredelige patienter) prioriteres højt, mens andre brugere (de svage elever og patienter) prioriteres lavt eller måske endog afvises. Lad os afslutningsvist kort se på den beskrevne udvikling i fugleperspektiv. Hvordan kan fænomenet evalueringsstyring tolkes i et samfundsmæssigt perspektiv? For det første kan der argumenteres for, at udviklingen af evalueringsstyring ligger i forlængelse af de forvaltningspolitiske moderniseringsreformer, som vi har været vidne til siden begyndelsen af 1980 erne, den såkaldte New Public Management bølge. Sigtet har været og er at øge effektivitet og brugerresponsivitet i den offentlige sektor. Hvor den tidlige evalueringsstyring altovervejende var kvalitativ, dialogbaseret og lærings- og udviklingsorienteret, synes udviklingen i disse år at vippe mere i retning af kvantificering og kontrolorientering. Måske afspejler dette, at tilliden til de fagprofessionelle i de offentlige organisationer er lav. Måske er det udtryk for, at en organisationsudviklingstilgang til evaluering er erstattet af en mere systemisk, sektoriel tilgang. Selvom der er en vis kontinuitet i udviklingen siden starten af 1980 erne er der også nybrud. For det første bliver det internationale element i evalueringsstyringen stadigt mere tydeligt. Evalueringsstyringsopskrifter rejser hurtigt mellem lande og evalueringspolitikker og -principper drøftes og fastlægges i stigende omfang i internationale netværk. Kvantificeringstendensen i dansk evalueringsstyring i disse år kan således ses som udtryk for afsmitning af og reaktion på den internationale udvikling. For det andet er anden ordens evalueringsinstitutionerne, som i øvrigt også oftest har en international profil, klart i vækst. Disse institutioner opstår i kølvandet på en betydelig vækst i mængden af evaluativ information i samfundet. De etableres med sigte på at skabe gennemsigtig i og synteser af allerede foreliggende viden. I denne proces evalueres den allerede foreliggende viden. Det, som anden ordens evalueringsinstitutionerne definerer som valid viden, sorteres fra anden viden. I denne proces autoriseres bestemte former for viden og vidensproduktion. Viden-

12 Side 11 syntetisering og videnstyring er givet nødvendigt i et vidensamfund, men præmisserne for hvordan dette skal foregå er og bør være til diskussion. Ovenstående bygger på et plenumoplæg holdt på Dansk Evalueringsselskabs konference om Nye styringsformer og evaluering, Nyborg Strand september Eventuelt interesserede kan læse mere i følgende publikationer af samme forfatter: Evaluation in and of Public-sector Reform: The Case of Denmark in a Nordic Perspective (Scandinavian Political Studies, vol. 28, no. 4, 2005: ), Choosing Evaluation Models (Evaluation, vol. 11, no. 4, 2005: ), Evaluering i staten: Kontrol, læring eller forandring? (Samfundslitteratur, 2003), Evidensbevægelsens udvikling, organisering og arbejdsform (akf forlaget, 2006, medforfattere med Yosef Bhatti og Olaf Rieper).

13 Side 12 Bringing Evaluation into Public Life By Ray C. Rist, The World Bank, Washington, DC, USA Introduction There is an arena in which evaluation is now increasingly directly and deeply involved the public arena. This is in contrast to traditional thinking of evaluation being focused on the relation of producers and users, users who are presumed to be in the policy or the administrative areas. We think of evaluation being for stakeholders or for the middlemen of bureaucrats, public managers, policy analysts, and senior officials (Pollitt, 2006). But to view evaluation more broadly as a public good means that there are new groups of end users that are not on the radar screens of evaluators. Stated differently, evaluators have largely and consistently ignored focusing their work to inform citizens. This present note will briefly consider the implications of evaluation beginning to expand its scope to include citizens as users. Evaluation is coming into public life, and in so doing, public life is changing evaluation from its traditional closed system of working just for middlemen. But there is an irony in this change underway that evaluators themselves seem not to realize that the change is underway. Evaluators continue to perseverate in their behavior that the world has not changed, that working with the administrative bureaucrats is the order of the day, and that so long as the bureaucrats are satisfied, evaluators are doing their jobs. What here is described as an emerging trend, one that is going to re-define and re-shape the discipline of evaluation, is clearly underway. Four key aspects of this trend are now noted below. Evaluation and Public Life As evaluation moves into the arena of public life, several changes occur. First, there is the simplification of the evaluative information as it is disseminated through the media and other public channels. As the evaluation community thinks it is nicely focused on working with policy makers and analysts, it emphasizes methodological sophistication and rigor going all the way to randomized designs to attempt to capture outcomes and impacts. Organizations such as the Campbell Collaboration that focus on building definitive knowledge assertions from strong empirical foundations are increasingly defining what is or is not factual and in an epistemological sense what is true. But sharing information in public life means going in the opposite direction. In the public arena, the emphasis is on making the message simple and straightforward. It is to be packaged for the message bar of CNN, for a five line summary on a web-based news

14 Side 13 source, or for one line in a political speech. Complexity and nuance are not the order of the day. One or two facts and the bottom line of what works or not (crime prevention programs), what is good or not (comparing test scores of different schools), or the quality of a program (surgurical survival rates at different hospitals) are what are shared. What also happens in this reporting within the public arena is that there is a distortion of the data to emphasize simplicity and brevity. Contextual understandings are omitted. Strengths of statistical analysis (reporting on standard deviations, survey response rates, and measurement error are also all generally omitted. Again, it is the emphasis on simply and quick that carries the day in reporting findings. And when that emphasis on simplicity takes on a political spin, spin and evaluation become adversaries. In contrast, the concerns of full and accurate reporting on the methodology are important, indeed, crucial to the professional norms of evaluators as information is released. And further, the methodological (if not arcane) atmosphere for so much of evaluation discussion and publication means that there is little to no experience or shared knowledge in dealing with the realities of real political combat over evaluation findings. The cross pressures on evaluators whose information comes into the arena of public life are not fully understood, in part because they are seldom discussed. Thus evaluators are not often prepared to undertake to simplify their own messages; to do what otherwise will inevitably be done to their work by others. Message simplicity and selectivity are expectations, even demands, in the public arena. Evaluators should thus not be surprised when this happens. The public arena is a new arena for evaluators. And the rules of the game are different here from those of the experiences of most evaluators. For evaluators to presume that the rules of the game will stay the same as they and their work move into the public arena is simply incorrect. And holding to this incorrect view will lead to confusion and a complete misreading of the new situation. A second key change that happens when evaluation information goes into public life is that there are new and different recipients of that information who then act on it in different ways than what one would expect from program managers and policy analysts. Parents making decisions on where to send their children to school or persons deciding in which hospital to have their surgery are not program managers or analysts. They have pragmatic needs for information it is not about evaluation information being for transformation, provided for enlightenment, or produced for an audience of one. As Markus Spinatsch has suggested elsewhere (2007, forthcoming), there exists a real cultural gap be-

15 Side 14 tween the world of expertise (complexity lovers) and the world of the public (simplicity lovers.) As noted above, Christopher Pollitt (2006) argues that evaluative information in general and performance information in particular has focused on providing data to the middlemen of program managers, policy analysts, and senior officials in ministries. This is in contrast to providing the information to the end users in democratic societies of citizens, elected representatives, and even ministers who have a different status in liberal democratic theory than do stakeholders, managers, etc. Stated differently, Pollitt is arguing that evaluation information has not gone into the public arena to sustain democratic institutions, but into the bureaucratic and organizational milieus of our societies. His paper is a call for moving evaluative information into public arenas where democratic debate can occur, where the citizens as end users can inform debates on what constitutes the public interest, and where citizens are not presumed to be simply another stakeholder among many. Citizens are the foundation for the creation and sustaining of representative democracy. It cannot be stressed too strongly how much the governance and public management literature has consistently omitted attention to the public arena and focused on, again to cite Pollitt, the middlemen. Consider just these two examples. Mark Moore, a professor at the Kennedy School of Government at Harvard, in his influential book, Creating Public Value: Strategic Management in Government (1995) directs his analysis fully towards the middlemen and is singularly quiet on how public value is created or sustained in public space. (It is also relevant here to note that in his index of seven full pages of double column citations, he gives three lines to evaluation.) And to carry this point one step further, in the 2000 report of the World Bank, Reforming Public Institutions and Strengthening Governance, this entire document is devoted to improving the public sector and its institutions without linking this reform to the consequences of the public arenas of the societies. Again, the audience is that of the middlemen. And here there is no analysis at all on the contributions of evaluation to improvement in the public sector, let alone to improvements in pubic spaces. (Note: this final observation should be reiterated the public sector is not the same as public space.) A third significant shift that comes with the movement of evaluative information into public space is that the defining relation between the producer of evaluative information and those who use the information is that of anonynimity. In the traditional world of producer and user, there was the general presumption of some personal relation the commissioner of the evaluation talks to and come to know the evaluator, the evaluator comes to

16 Side 15 know and talk to the stakeholders, stakeholders come to know and talk to one another, etc. In fact, traditional evaluation training stresses getting to know the commissioner, the stakeholders, even some of the program s or policy s target groups. It is about being up close and personal. But when the evaluative information moves to the public arena, the evaluator is not likely to know the consumers of the information be they in any of Pollit s three key end-user groups (citizens, elected officials, or ministers.) This third shift necessarily forces a re-thinking by the evaluator of what it means to be utilization focus, to be conducting participatory or collaborative evaluations, or to undertake stakeholder analysis. Evaluation in this area is not about getting stakeholders and program managers to participate more fully in the shaping, executing, and using evaluations. It is much more about being clear about what to release into the public arena via internet, preparing for media briefings, writing executive summaries that are short, concise, and simplified; and responding effectively to the likely distortions of the findings in the media and political discourse. Breul and Dahler-Larsen (2007, forthcoming) discuss in detail the consequences of when evaluators fail to grasp the consequences for their own work when it moves into this new arena. The fourth and final shift to emerge from moving evaluation into the public space is that the ethical equation changes. There are, for example, questions that focus on: whether or not to provide disclosure of all findings; to dumb down (or not) the findings so as to lessen the chance of misinterpretation and abuse of the information, the accepting of the assertion that NGOs, associations, interest groups, etc. do in fact represent citizens: and thus should be part of the negotiation process in working out the evaluation design and have a say in how and when the reporting on the findings should take place, how one decides which groups or organizations should be chosen to represent the various constituencies that claim to speak for one portion or another of the public. whether the evaluation information being involved in a partisan advocacy role is acceptable; and what responsibilities, if any, does the evaluator have when a fundamental distortion of the evaluative information has taken place. These questions may not be new or original to evaluators now working in the public arena. But they clearly have a different context in which they are to be considered and answered

17 Side 16 from when evaluators are back working in the conventional framework and the sounds of silence from the evaluation community are not helpful. A Final Thought Evaluation communities in democratic societies have a responsibility to address the issues raised by evaluation coming into the public arena. It can no longer be a business as usual approach, assuming that the paradigms and models of evaluation work in the preceding decades still hold and should define the present realities. We can hope the conversations in the evaluation community will begin to turn towards how to operate and behave in the context of democratic values and democratic institutions. The present emphasis in evaluation of quality being defined by technical adequacy of methodology and usefulness to stakeholders is outmoded as soon as evaluative knowledge moves into the public arena. Evaluation has no precedent or experience in how to approach its work in this new context. And sadly, there is, at present, little to no discussion in the evaluation world on the implications of working in democratic societies working the space of public life. References Breul, J. and P. Dahler-Larsen, eds., (2007, forthcoming) Evaluation: Open to the Public. New Brunswick, N.J.: Transaction Publishers. Moore, M. (1995), Creating Public Value: Strategic Management in Government. Cambridge, Massachusetts: Harvard University Press. Pollitt, C., (2006). Performance Information for Democracy The Missing link?, Evaluation: The International Journal of Theory, Research, and Practice. Vol. 12, no.1. The World Bank (2000), Reforming Public Institutions and Strengthening Governance. Washington, D.C.: World Bank Publications.

18 Side 17 Daniel, Nebuchadnezzar, and the Birth of Government Program Performance Management Dr. David E.K. Hunter, Hunter Consulting Introduction In order to introduce the idea that government must invest in the creation of social good by building local capacities to deliver and evaluate it, I want to start at the beginning with the focused, high-stakes evaluation of a dietary program undertaken some 26 hundred years ago in 605 BCE. That was the year in which Nebuchadnezzar, King of Babylon, conquered Jerusalem and carried off into servitude thousands of the People of Israel. Now it seems that Nebuchadnezzar was a progressive ruler who wanted to assimilate his captives and turn them into productive citizens (but not too productive he had the men castrated). Nebuchadnezzar therefore commanded Ashpenaz, the master of his eunuchs, to select a group of Israelite aristocrats and teach them the language, culture, and accumulated knowledge of their hosts. To ensure the success of this undertaking, Nebuchadnezzar ordered that they receive a daily provision of the king's own meat and wine, intending thus to nourish them for three years so that, at the end of this period, they might appear before him healthy, well-educated, and ready to earn their keep. Among the selected group was the future prophet Daniel. Whereas other Israelites agreed to Nebuchadnezzar s terms, Daniel and his friends could not bring themselves to eat what Jewish dietary laws considered to be unclean food. He therefore demurred, threatening a hunger strike. This put Ashpenaz, the newly appointed Director of the Babylonian Government Acculturation and Dietary Program for Israelite Guest Workers, in a terrible bind. As Ashpenaz said to Daniel, The buck stops with me and the King is one unforgiving boss He went on to explain that if Daniel and his companions were to go on a hunger strike and stand before Nebuchadnezzar looking sick or emaciated, The King ll have my head on a platter. But Daniel, being after all a biblical prophet in the making, proposed a dietary program evaluation, suggesting that he and his friends be given ten days to eat a porridge of leg-

19 Side 18 umes (thus avoiding unclean meat) and drink water according to Jewish law. And he made it a high-stakes evaluation, saying, in good King James idiom, Then let our [faces] be looked upon before thee, and the [faces] of the children that eat of the portion of the king's meat: and as thou seest, deal with thy servants. Ashpenaz, upon hearing this plan, rather courageously agreed. What happened? Well, after ten days Daniel and his three friends looked healthier and chubbier than their compatriots who ate the king s dishes. And so, Ashpenaz, accepting these outcome data as sufficient to make a decision, allowed Daniel and his friends to follow Jewish dietary law and gave them a steady Kosher diet. And we can assume he didn t leave matters at that, and instituted a performance tracking system through which the ongoing health of Daniel and his friends was monitored. In any event, as we all know, they thrived and Daniel, at least, was a star performer. Now it is clear that, from a research perspective, this evaluation is what we might call weak. First, there are only four subjects, a ridiculously small number to use for the purpose of comparison. Secondly, the four who participated in this evaluation were selfselected. Furthermore, the intervention dosage was minimal; the time trial was ludicrously short; and the indicators ambiguous. When we also consider that the outcomes were, at best, impressionistic, and the measures highly subjective, it becomes clear that this evaluation was not what we would call scientifically sound. But on closer inspection, we have to acknowledge that this evaluation has some virtues, too. It assessed a clearly articulated theory of change (kosher diet leads to good health), it included program participants who had a full voice in the evaluation design (actually, they proposed it), and it was led by a program director who accepted the research methodology as dispositive. In addition, all key constituents had full prospective buy-in on how the findings would be used, and they accepted the outcome assessments as settling the question of diet, even though they were qualitative in nature. When we also consider that this evaluation had a course of action that was indicated by the short-term health outcomes, and that subsequent to the evaluation this course of action was implemented, and further that an ongoing evaluation capacity was built within the program so that it could be used

20 Side 19 for continuous performance monitoring and quality improvement, and finally that it allowed for some spectacular long-term performance outcomes (Daniel, after all, correctly prophesized the fall of Babylon to the Persians), we are able to see that this evaluation had some noteworthy qualities. So, here we have an evaluation that at first glance is problematic at best and useless at worst. Nevertheless it was designed to be useful, was inexpensive, built local evaluation capacity, supported high-stakes decision-making, was used to develop an ongoing program performance monitoring and quality management system, accomplished what it was designed to do, and consequently in fact was useful to all involved. That is not bad at all and worth keeping in mind as we consider the ways in which evaluation can be meaningful and useful for the purposes of developing, delivering, and sustaining social good. Government and the Need to Invest in Social Good The core premise of this article is that government must redefine its efforts from "spending to create social good" to "investing in the creation of social good." As simple shift, but one with profound consequences, since "investing" inevitably involves a series of fundamental requirements, including (but not limited to): building and sustaining local organizational capacity to assure the delivery of a return on the investment, in this case a social return on investment; developing performance standards and the systems to maintain and using them to manage the quality of local efforts to produce the targeted social return on investment; and building the local capacity to assess (evaluate) the delivery of social return on investment. In short, "spending" simply presumes the need to act as if one wants to create social good, and assumes the acceptability of going through the motions of deploying funds to support activities. In contrast, "investing" presumes the creation of assured return, and assumes the need to deploy resources (consisting of funds, people, and knowledge) into the capacity to create

21 Side 20 the return. Simply put, government no longer can afford to "spend" the people's funds. Thus, this paper argues that government should not buy or deliver activities to create social good. Rather, it should invest in the provision of measurable social good itself be this improved health, less crime, increased employment, better school performance, etc. If you pay for activities, that is exactly what you get be it from government employees or vendors providing services on behalf of government. And activities do not necessarily add up to or create social good, no matter how well intended. In the USA there are many highly funded social, educational, employment, health, and other programs delivered by or on behalf of the government that, when evaluated rigorously, have no measurable impact. I would imagine the same is true in Denmark. Why? For many reasons, including that they: lacked an evidentiary basis beyond face validity, lacked initial evidence of implementation feasibility, lacked clearly defined target populations, lacked clearly defined outcome objectives with measurable indicators, lacked implementation standards (measures of program fidelity), lacked design that allows for evaluability, lacked post-implementation assessment of program fidelity (process or formative evaluation) lacked local organizational capacity to deliver the program reliably at high levels of quality, lacked the local use of sophisticated performance management systems to track agreed-upon measures of program performance and lacked the capacity to track client outcomes. All of this can be summarized by saying they lacked what it takes to produce and maintain social good locally, and monitor it centrally namely, (a) a robust, empirically grounded theory of change, and (b) the local organizational capacity to assure implementation with fidelity to design, at high levels of performance and quality.

22 Side 21 Government, Evaluation, and the Delivery of Social Good How then, can government invest in the provision of measurable social good? And what can evaluation contribute? Here are a series of steps that, it seems to me, are critical for government to take in the creation and delivery of social good, where evaluation-grounded thought and practice are essential. 1. When seeking to address a social need, resist the urge to innovate the first step is to look for evidence-based programming that rigorous evaluation has proven in fact produces the desired impacts, and proceed to implement such programming where possible. 2. In the absence of pre-existing evidence-based programming, government should engage practitioners and evaluators in the design of new programming that is grounded in what has been learned from other programming in this area that is know to be effective, is well codified, has clear implementation standards, has operational outcome objectives, and has agreed upon measures for assessing effectiveness. Then it is useful to pilot test the program and, if initial evidence shows the program to be effective, broaden pilot testing for all populations for which it will be used. 3. From the very beginning, government should dedicate funds to build the capacity of each site to monitor implementation against pre-defined standards, track utilization patterns (especially with regard to key sub-groups defined by such things as age, gender, and ethnicity), develop real-time data to drive continuous quality improvement, track program participant outcomes, and attempt to assess impacts. In these efforts there is a clear role for evaluators who are willing to move away from a narrow, technical focus on research methods and sectarian arguments advancing one or another "school" of evaluation (e.g., (empowerment evaluation, theory-based or theory grounded evaluation, theory neutral or "black box" evaluation, case study evaluation, participatory evaluation, experimental evaluation, feminist evaluation, qualitative versus quantitative evaluation and on and on ad nauseam) but rather put their best and most creative work toward helping in the design, building, and delivery of programs that add to the greater social good and the local capacity to manage them at high levels of quality and effectiveness.

23 Side 22 Konferenceoversigt Nedenfor er oplistet internationale evalueringskonferencer. Send meget gerne et link, hvis du er blevet opmærksom på en konference Tidspunkt Tema Sted Organisation Link 5-8. februar Managing for Development Results Hanoi, Vietnam Mfdr - Managing for Development Results marts Monitoring and Evalua- Johannesburg, SAMEA tion South African South African Monitoring and Evaluation Assocation april Self-evaluation: illu- University of SEVAL sions, confusions, Fribourg, Svejts Swiss Evalutation solutions and fusions, Society 2-6. juni Culture, Community Winnipeg, Can- CES and Social Justice in ada Canadian Evalua- Evaluation Do the tion Society Pieces Fit? 3-7. septem- Doing Evaluation Bet- Melbourne, AES ber 2007 ter Australien Australian Evaluation Society sep- Ikke besluttet endnu Kolding, Dan- DES tember 2007 mark Dansk Evalueringsselskab no- Evaluation 2007 Orlando, Florida AEA vember 2007 American Evaluation Assiciation 5-8. novem- Evaluation 2008 Denver, Colo- AEA ber 2008 rado American Evaluation Assiciation no- Evaluation 2009 Baltimore, AEA vember 2009 Maryland American Evaluation Assiciation

24 Side 23 Om Dansk EvalueringsSelskab DES er åbent for alle med interesse for evaluering. Det kan eksempelvis være personer, som: 1. Forsker og underviser i evaluering 2. Er opdragsgiver til evalueringer 3. Gennemfører interne evalueringer 4. Gennemfører eksterne evalueringer 5. Andre personer med interesse for evaluering, eksempelvis studerende. Formålet med DES er at være en platform for erfaringsudveksling og diskussion af evalueringsfaglige problemstillinger, herunder: 1. Udviklingen inden for evalueringsteori- og metode 2. Nyttiggørelse af evalueringsresultater 3. Faglig udvikling 4. Standarder for god evalueringspraksis. Deltagelse i selskabet rummer følgende fordele 1. Adgang til at diskutere faglige problemstillinger med andre personer, som arbejder med området. 2. Mulighed for at deltage i arbejdsgrupper. 3. Deltagelse i evalueringskonferencer til en reduceret pris. 4. Gratis adgang til gå-hjem-møder. Oplæggene kan komme fra såvel foreningens medlemmer som fra eksterne oplægsholdere. 5. Gratis nyhedsbrev. Medlemskab Tegning af medlemskab sker ved at udfylde og indsende en tilmeldingsblanket, som kan findes på. Medlemskabet gælder for resten af kalenderåret. Medlemskabet kan både tegnes personligt eller som firma/institution. Prisen er kr. 825 pr. år for et personligt medlemskab, kr pr. år for et firma/institutionsmedlemskab og kr. 275 pr. år for studerende. Spørgsmål kan rettes til selskabets sekretariat, Marie Louise Møller, des@eva.dk.

25 Side 24 Bestyrelsen Bertil Michael Mahs Anette Dørge Jessen Styrelsen for Social Service Danmarks Evalueringsinstitut Socialministeriet Østbanegade 55, 3. sal Skibhusvej 52 B 2100 København Ø 5000 Odense C Tlf: bertilmahs@webspeed.dk adj@eva.dk Poul Skov Dahl Dorte Kabell Muusmann Research & Consulting Kabell Konsulting APS Haderslevvej 36 Toldbodgade 19B, 2. sal 6000 Kolding 1253 København K Tlf.: Tlf.: psda@muusmann.com dk@kabell-konsulting.dk Thomas Pallesen Steffen Bohni Nielsen Institut for Statskundskab Rambøll Management Aarhus Universitet Nørregade 7A Bartholins Allé, 8000 Århus C 1165 København K Tlf.: Tlf: pallesen@ps.au.dk sni@r-m.com Tim Jeppesen KREVI Olof Palmes Allé Århus N Tlf.: tj@krevi.dk Solgerd Just Mikkelsen Solgerd@sol.dk Bianca Albers Sekretariatet Styrelsen for Social Service Dansk EvalueringsSelskab Socialministeriet Att.: Marie Louise Møller Skibhusvej 52 B c/o Danmarks Evalueringsinstitut 5000 Odense C Østbanegade 55, 3., 2100 KBH Ø Tlf.: Tlf bal@servicestyrelsen.dk des@eva.dk