Guide to Online Participation/3.3 Evaluate/Full story
From ParticipatioNZ
For the full story on how to evaluate online participation read more here.
Why evaluate?
Evaluation is too often an afterthought, or left out altogether. Unwittingly perhaps, proponents and detractors of public participation conspire to maintain the current 'evaluation gap' -- albeit with different ends. Given the lack of benchmarks against which to measure the costs and benefits of this emerging field of practice, proponents are loathe to lay bare the real costs of participation as they are unsure what counts as too much or not enough. They are also unsure how to account for the tangible and intangible benefits of public participation. Detractors benefit from the lack of hard data on either costs or benefits as it allows them to vociferously maintain that whatever is spent, is certainly misspent.
In the end, it is the public that pays twice over -- first, as taxpayers funding government's efforts to inform and engage with them; second, as participants who have to make do with poorly planned and executed public participation initiatives. As public servants we owe them a better deal.
Evaluation will allow us to identify what works best, learn from past mistakes and improve the application and experience of online participation.
How to use this evaluation framework
What does successful online participation look like? Do we need to evaluate process, outcomes or both? Who defines success and who measures it? We need to find answers to these questions if online participation is to benefit New Zealanders today and in the future.
The evaluation framework we offer here can be used for a variety of purposes. For example, you can use it to assess:
- whether specific online participation initiatives have met their stated objectives
- the extent to which the principles for online participation have been adhered to
- whether online participation provides a cost effective approach
- how online participation can best be integrated with 'offline' participation
- ways of improving the practice of online participation.
Above all, this section invites you to build considerations of evaluation in from the outset and embed evaluation in the proces, and to consider what follows as a tool for self-assessment.
Evaluation as learning
"If public engagement in policy making is a recent phenomenon and evaluation is itself a relatively young discipline, then it may safely be said that the evaluation of public participation is still very much in its infancy." -- OECD 2005, p. 11
There are many ways to evaluate. So, the first step is to choose an approach that is appropriate for online participation. Broadly speaking, evaluation can be conducted for the purposes of audit (to exercise control), management (to support decisions) and learning (to foster greater understanding and better practice). Given that the theory and practice of online participation is still in the early stages of development, an evaluation approach that maximises learning is more appropriate than one aimed at audit or management.
The next step is to decide who evaluates. Will the main role be played by an independent evaluator, an in-house evaluator or by participants themselves? These choices also influence the extent to which evaluation contributes to learning within an agency. Successful evaluation will assess online participation from the perspective of the agency, the participants and, where relevant, third parties who have a stake in the process or outcomes.
Several approaches to evaluation are available and they can all be applied to online participation. Goal-bound evaluation will measure the extent to which predefined objectives have been met. In a goal-free evaluation the evaluator will not be told what the programme objectives are but will focus solely on measuring the actual effects. Goal-free evaluation is, in Michael Scriven's (1974) words, "oriented towards final results, not original rhetoric". One of the advantages of this approach is that it can capture the unwanted side-effects as well as the unintended benefits of a given initiative. The main disadvantage is that it is generally more resource and time intensive.
Finally, a key question to consider is what will count as 'evidence' in an evaluation? There are two main ways of answering this question. We could choose to focus only on what can objectively be known ('just the facts'). Or we could focus only on what participants say they mean ('it's all relative') (see Guba and Lincoln, 1989). But there may also be a third way which combines the strengths of both. Deliberative democratic evaluation, as proposed by Ernest House and Kenneth Howe (1999), recognises that people need an opportunity to make their values and perspectives explicit in the course of an evaluation, but also requires 'proof' or reason to play an important role. By embedding deliberation into the process of evaluation we allow people to express themselves in their own words. But we also open their statements up to scrutiny and challenge by others in the course of open debate.
Choosing an appropriate model for evaluation involves answering six basic questions. The approach to online participation proposed here answers these questions in the following way.
| 1. | Why evaluate online participation? | The primary purpose of evaluating online participation is for learning. |
| 2. | When should we evaluate online participation? | Evaluation should start early on and continue throughout. |
| 3. | How will we know what counts as evidence? | Evaluation will encompass facts, values and perspectives of all relevant stakeholders (including non-participants). |
| 4. | Who does the evaluation? | Participatory evaluation will be the default position subject to feasibility. |
| 5. | What aspects of online participation are we evaluating? | All three components of online information, consultation and participation will be subject to evaluation. |
| 6. | Which level are we pitching our evaluation at? | The main focus will be on the programme level (ie evaluation of online initiatives) rather than on evaluating specific ICT tools. |
The answers to these six questions each have implications for practice which can be expanded upon further.
Evaluation against what?
There is no 'off-the-shelf', 'one-size-fits-all' evaluation template which will fit every circumstance. This is the bad news. But the good news is that many benchmarks are available. We could evaluate online participation against the following benchmarks.
- Traditional participation methods This makes sense if we are trying to decide whether online avenues represent a significant 'value-added' compared with more established offline practices. However, this is only possible when both avenues exist and are equivalent (e.g. online petitions versus traditional petitions). It's of little help when online participation opens up entirely novel methods for participation (e.g. wikis, mashups) which have no offline counterparts.
- Stated objectives A given online participation initiative can be evaluated against what it set out to achieve. This may require a significant investment in evaluation design (i.e. reinventing the wheel). It also has the drawback of not allowing you to make comparisons across initiatives (i.e. it doesn't allow for cumulative learning).
- Principles Several countries, intergovernmental organisations and associations have developed principles for public participation (see links below). Principles provide a solid foundation for developing specific evaluation questions. They also provide a common framework for comparative analysis and collective learning based on accumulated evaluation results.
The evaluation framework we propose here is based on the set of seven principles outlined at the beginning of this Guide. For each principle, it develops a set of indicators and measures which you can adapt to suit specific initiatives. Taken together, the set of principles provide a basis for evaluating both the process and the outcomes of online participation.
We suggested earlier in this Guide that the strengths of both 'administrative' and 'democratic' perspectives on online participation can be combined to good effect (see Designing for participation section). The implication for evaluation is that you should measure the success of each online participation initiative against its own (immediate) goals as well as evaluate how well it supports the wider democratic system (long term). By evaluating specific online participation initiatives against the set of principles outlined at the beginning of this Guide, we can do justice to both perspectives.
“... a single consultation should not only be evaluated as a problem-solving exercise (i.e. administrative efficacy), but also in how it helps or hinders longer-term democratic processes” -- G. Honor Fagan et al (2006) p. 44
Moving from principles to indicators
At its most basic, designing an evaluation means thinking about the following three questions.
- What do we want to know? Answering this will allow us to develop meaningful evaluation questions.
- What type of information will be most relevant? We need to decide how our evaluation questions could be answered.
- How will we capture this information? Data that can't be collected is of no practical use -- we need a reality check.
Indicators and measures can be developed for each of the seven principles set out earlier in this Guide. This section will help you to construct an appropriate evaluation plan for your online participation initiative and suggests where you might find data for each indicator.
Clarity
Be clear. Be open and transparent about the objectives, limits, resources and potential impacts of online participation.
| Indicators | Measures | Sources |
| Goals, scope, available time and resources stated |
| Document review |
| Relevant information provided and plain language ensured |
| Document review and user survey |
| Respective roles and obligations are stated |
| Document review and user survey |
| Online tools are designed to be easily used |
| Web statistics (e.g. on useage rate), user survey, survey of non-users and key informant interview |
Respect
Demonstrate respect. Demonstrate respect for the contributions, perspectives, values and prerogatives of people, stakeholders, elected representatives and public servants.
| Indicators | Measures | Sources |
| Diversity is acknowledged and valued |
| Document review and key informant interview |
| People can choose how they participate |
| Document review and key informant interview |
| People’s contributions are acknowledged |
| Operations statistics, key informant interview and user survey |
| Safety and security is ensured |
| Operations statistics, document review and key informant interview |
Confidence & commitment
Build confidence as a basis for commitment. Online participation is a new practice for decision-makers, people and stakeholders. Give it time, prove its worth.
| Indicators | Measures | Sources |
| Confidence in tools and processes is justified |
| User survey, Member of Parliament (MP) and chief executive (CE) surveys if possible, key informant interview otherwise |
| Commitment is demonstrated |
| MP, CE and staff surveys if possible, key informant interview otherwise |
Creativity
Be creative. New tools mean new approaches. Success hinges on innovation.
| Indicators | Measures | Sources |
| New tools are piloted and shared |
| Document review and key informant interview |
| New forms of collaboration are explored and tested |
| Operations statistics and key informant interview |
| Future risks and opportunities are regularly scoped |
| Document review and key informant interview |
Inclusion
Be inclusive. Go to where people are. Make every effort to ensure accessibility and connect with all relevant communities, online and off.
| Indicators | Measures | Sources |
| A broad range of people are informed and engaged |
| Document review, key informant interview and surveys of users and non-users |
| Māori perspectives and voices are included and valued |
| Document review and user surveys |
| Communities are catered for |
| Document review, key informant interview. User survey and survey of non-users |
| Accessibility and multiple channels for participation are ensured |
| Document review, key informant interview. User survey and survey of non-users |
Accountability
Be accountable. Online participation is a multi-stakeholder process where everyone is accountable.
| Indicators | Measures | Sources |
| Effective and efficient use is made of public resources |
| Document review and key informant interview |
| People’s time and attention are treated as valuable and scarce resources |
| User survey, survey of non-users, document review and key informant interview |
| Resources donated by people are given value |
| User survey, document review and key informant interview |
| All participants are accountable for their actions |
| Operations statistics and key informant interview |
| Content generated by online participation is stored, archived and accessible |
| Document review and key informant interview |
Achievement
Make a difference. Strive for, build on and celebrate achievements in using online participation as a means for people, government, communities and businesses to achieve their goals.
| Indicators | Measures | Sources |
| Final outcomes are influenced |
| Content analysis, MP and CE surveys if possible, key informant interview otherwise |
| Procedures for the design and delivery of public policy and services are adapted |
| Document review and key informant interview |
| Greater reciprocal understanding is achieved |
| User survey, CE and staff surveys if possible, key informant interview otherwise |
| People gain immediate personal benefits from participating online |
| User survey |
| Social capital and trust in government are both fostered |
| Social network analysis, user survey and survey of non-users |
Note: The approach used here owes much to the excellent 2005 E-democracy evaluation framework, prepared by M & P Henderson & Associates Pty Ltd and commissioned by the e-Democracy Unit of the Queensland Government (Australia).
Why choose participatory evaluation?
"As citizens become more educated and want to be better informed there will be increasing pressure on public agencies to admit citizens and interest groups as co-evaluators." p. 16 4QC Conference 2006, Scientific Rapporteurs Report
It is now commonplace to ask the people who use public services for their opinions, conduct regular opinion polls and test new ideas with focus groups. This is a good example of how people can participate in evaluation through answering questions. But their role could be far wider and more active. Both upstream, in formulating criteria for success and developing pertinent evaluation questions, and downstream, in analysing evaluation results. This is participatory evaluation.
"The participatory approach gives us reason to consider a new role for evaluation -- not as an exercise where external or internal evaluators come to the target groups and discover the truth, but as a process of mutual learning." Forss, K in OECD (2005), p. 71 OECD report on evaluating public participation
Participatory evaluation can be a valuable approach even when the subject of evaluation is a complex or highly technical piece of public policy (e.g. bioethics), service (e.g. mental health) or project (e.g. siting nuclear waste disposal). Given that the subject matter of online participation is participation itself, we can't ignore or dismiss the potential value of participatory evaluation outright. Worse still, a non-participatory approach to evaluation could undermine participants' confidence in the process and put the online participation initiative we have worked so hard to design and launch at risk. Finally, participatory evaluation helps build greater understanding and capacity among participants. They can then invest this in any future public participation initiatives, whether online or off.
How to start using participatory evaluation
Online participation offers a challenging, but promising, new ground for participatory evaluation. The online medium offers many advantages over traditional settings -- for example, the ability to track discussions, automatically generate traffic data on use, conduct online surveys and display results in real time by using attractive graphics and visuals (see more on data visualisation here). Specifically, you could invite participants to play a more active role in the following areas.
- Formulating questions You could make this the role of the moderator of an online discussion group, the subject of a quick poll on your website's home page or solicit questions via email. At the very least you could invite participants to review a proposed set of evaluation questions (e.g. the SSC's invitation to members of the Community of Practice to help in evaluating the Participation wiki). You can invite participants early on in an online participation initiative to help design the evaluation by giving their views on these questions.
- What will constitute success from your point of view? (criteria)
- What would success look like? (indicators)
- How will we know if we have been successful? (data)
- Answering questions You can, of course, ask participants to answer a brief set of evaluation questions in an online survey. But they may even take on the role of interviewer by forwarding the questions to other people they know (e.g. by including an 'email a friend' option). This form of 'viral marketing' may be especially valuable when you are seeking the views of non-participants. These could fall into two groups -- those who are online but uninterested, or those who are offline but potentially interested. You could use public recognition and reward for the most prolific participant-interviewers to encourage action, e.g. run a weekly feature on 'cyber-evaluators'.
- Analysing the results You can invite participants to review evaluation results by regularly providing updates, annotated research findings and direct access to the underlying raw data. Evaluation results could be subject to online discussion, ranking (e.g. for public importance) and rating (eg for personal relevance), and the initial draft evaluation report posted for comment and critique.
- Using the results You can publish evaluation results online, but they are only useful if they are known and acted upon. You can encourage participants to disseminate them and use them to generate discussion and debate on their own blogs, websites or discussion boards. In addition, participants could be encouraged to use the underlying data to generate their own mashups and graphics. If evaluations reveal problems, as they inevitably will, participants might be a key part of generating solutions. It is therefore essential to be clear about what happens once the evaluation report is issued and who will receive any suggestions or proposals.
"There is no such thing as a free lunch. The path to increased participation may well go over participatory evaluation, but it requires effort to master some of the jargon and some methodological skill. The key question is, how much of the methods can be used without alienating the participants." Forss, K in OECD (2005), p. 71 OECD report on evaluating public participation
- Limits to participatory evaluation Getting people to participate is hard enough. Getting them to focus on the quality of the participatory process and its outcomes is harder still -- especially if it is pitched as an extra step or additional time burden. Weaving evaluation into the process of getting an online participation initiative underway is your best bet. It's also essential to provide tangible support for capacity building so people feel confident in expressing their views about the quality and value created by online participation.
Learning to evaluate online participation
The theory and practice of evaluating online participation is in its infancy. Given the steep learning curve ahead, we'll build our competence more quickly if we pool our knowledge. This Guide provides a first step in this direction. But it does not take us all the way.
Improving our tools and skills for evaluating online participation calls for concerted efforts across New Zealand's public, private and community sectors to:
- collect and compare concrete experience over time (e.g. by a 'Participation Observatory')
- build professional skills and competence
- develop networks and mentoring
- design appropriate ICT tools to support evaluation.
Tips for evaluating online participation
- Given your goals, define criteria for success.
- Develop your own indicators and measures based on the seven principles.
- Identify your sources of data.
- Embed data collection in the activity.
- Leave time for data analysis.
- Consider the scope for participatory evaluation from the outset.
- Invest in communicating evaluation results.
Examples
The 2005 E-democracy evaluation framework adopted by the Queensland Government (Australia) provides concrete examples of how to approach the evaluation of e-petitions, online consultation and webcasting. An example of how to evaluate the use of a wiki is provided by the State Services Commission: Online Participation Project Case Study.
e-petitions
Queensland's E-democracy evaluation framework, 2005, p. 5
online consultation
Queensland's E-democracy evaluation framework, 2005, p. 10
webcasting
Queensland's E-democracy evaluation framework, 2005, p. 13
wiki
State Services Commission (New Zealand) ParticipatioNZ wiki
Links
Guides to evaluation of online participation:
- E-democracy evaluation framework, Queensland, 2005, M & P Henderson & Associates Pty Ltd.
- Guidelines for Community Engagement Using Information and Communications Technology (ICT), Western Australia, 2005 (pp. 28-29)
- Electronic Engagement: A Guide for Public Sector Managers, Chen, P., 2007 (pp. 79-82)
- Promise and Problems of e-Democracy: Challenges of online citizen engagement, OECD, 2003 (pp. 73-76)
Guides to evaluation of public participation:
- Making a Difference: A guide to evaluating public participation in central government, Involve, 2007
- Evaluating Public Participation in Policy Making, OECD, 2005
Examples of evaluation of online participation
- From the Top Down: An evaluation of e-Democracy Activities initiated by Councils and Government, 2005, International Teledemocracy Centre (ITC), UK
- Digital Dialogues: Interim Report (December 2005-August 2006), Hansard Society and UK Department for Constitutional Affairs
Case studies
For concrete examples of the evaluation issues discussed here see the set of seven case studies in the Resources section and in particular:
Families Commission: The Couch Case Study
SafeAs! Roadsafety Stakeholder Engagement Case Study
State Services Commission: Online Participation Project Case Study
Relevant reading
- Bouckaert, G., Loeffler E. and C. Pollitt. 2006. 'Making Quality Sustainable: Co-design, Co-decide, Co-produce, Co-evaluate', 4QC Conference 2006 Scientific Rapporteurs Report.
- Guba, E.G. and Y.S. Lincoln. 1989. Fourth Generation Evaluation. Thousand Oaks, CA: Sage.
- House, E.R. and K.R. Howe. 1999. Values in Evaluation and Social Research. Thousand Oaks, CA: Sage.
- Involve. 2006. The True Costs of Public Participation. London: Involve.
- OECD. 2005. Evaluating Public Participation in Policy making. Paris: OECD.
- Ryan K.E. and L. DeStefano (eds.) 2000. Evaluation as a Democratic Process: Promoting Inclusion, Dialogue, and Deliberation. New Directions for Evaluation, no. 85, Spring 2000.
- Scriven M. 1974. "Evaluation Perspectives and Procedures: A Publication of the American Evaluation Association" in W. James Popham (ed.) Evaluation in Education: Current Applications. Berkeley, CA: McCutchan Publishing Corporation.
