International workshop on measuring effective access to justice – an overview

OECD Conference Centre, Paris

The following overview sets out a summary of a recent workshop on access to justice hosted by the Open Society Justice Initiative (OSJI) and Organisation of Economic Co-Operation and Development (OECD). It was held in Paris, at the OECD Conference Centre, on 3-4 November 2016.

By Cris Coxon

About the workshop

The Open Society Justice Initiative (OSJI) and Organisation of Economic Co-Operation and Development (OECD) jointly hosted the event, to facilitate the exchange of information and experiences on how we define, measure and evaluate access to justice and legal need. Around 50 participants attended from 20 countries, some within Europe but also further afield e.g. the US, South America, Indonesia, South Africa, Australia.

A cross-cutting purpose of the event was to help implement and monitor the UN Sustainable Development Goal 16.3 – to ‘promote the rule of law at the national and international levels, and ensure equal access to justice for all’. Although criminal justice was discussed, the focus was more on civil, administrative, and family justice. Presentations were given to kick off each session, but much of the workshop was dedicated to roundtable discussion.

A link to the presentations and the agenda is here

Identifying key access to justice issues

The different social, physical and economic conditions within countries highlighted how governments must start with an understanding of their population’s legal needs and experiences in accessing justice. There was discussion that justice problems are really everyday issues people have exercising their rights and therefore cross-cut many areas of social policy not just justice departments.

In some places the very stark differences in levels of income, education, gender equality etc underline the need for both an inclusive legal framework and high levels of support so it can be accessed by all groups of the population. Ensuring equity was therefore a key overall theme, underpinned by various issues such as:

• cost, complexity and timeliness of justice services
• flexibility (or lack of it) within justice systems – including provision of ADR
• legal awareness and capability
• availability of early advice and support (formal and informal)
• availability of legal aid
• prevalence of corruption
• human rights violations
• trust and confidence in public services
• geo-spatial access
• supply of, and market for, legal services

Approaches to improve the equity of access to justice

 Countries shared a number of approaches, many of which fit into these high level themes:

• Legal empowerment interventions, designed to give people – in particular the poor and marginalised – the power to know and use the law – e.g. provision of legal help and information in the Ukraine, including a ‘WikiLegalAid’ tool.

• Ensuring a continuum of services, including understandable legal information, legal representation, non-legal support, and support for ADR.

• Targeting of services to make sure limited public funds are used to assist those facing the most severe problems and least able to otherwise access help – e.g. the White House Legal Aid Interagency Roundtable in the US).

• Early intervention to avoid legal problems being left unaddressed, which can in turn entrench social disadvantage. Early resolution services need to migrate users to the least expensive services that adequately meet their needs.

Some countries are expanding the scope of legal aid to civil and administrative cases (Estonia, France, Ukraine, Georgia, Indonesia) and others are extending this to cases resolved through alternative dispute mechanisms (Armenia, Germany, France, Lithuania, Switzerland).

Techniques to measure legal need and the impact of interventions

Different approaches were discussed, but legal needs surveys were a particular topic of interest. Attendees agreed they are still probably the best way to understand the big picture, in particular experiences outside the formal justice system where most problems are resolved.

National legal needs surveys are now carried out by 37 countries. The UK pioneered the design of some of the first legal needs surveys, and is still seen as a leading authority. Prof Pascoe Pleasence (UCL) gave an overview of their history and use, explaining that while earlier surveys tended to focus on processes used to resolve problems, later ones have reflected greater interest in issues such as problem impact, clustering, legal capability, and drivers of problem experience and problem resolving behaviour. Some high level insights from legal needs surveys have included:

• Justice problems are fairly prevalent, but are not necessarily perceived as legal problems requiring lawyers or legal services.

• The majority are perceived as everyday issues involving rights, which are resolved by taking informal action at an early stage. ‘A tenant with a leaking roof may be regarded as having a legal problem. However, he may choose to get a ladder and not a lawyer’ (Philip Lewis, 1973).

• Justice problems impact broadly on people’s lives and can lead to wider social problems and into ‘cycles of decline’ as shown below (adapted from Tyler et al, 2011).

Issues with legal needs surveys

We had several presentations of legal needs surveys in different countries (e.g. Canada, Australia). Attendees agreed that they are effective for highlighting big issues and assessing whether people make use of the interventions available to them (e.g. demonstrating low take-up of formal legal services in New South Wales relative to informal methods).

There was also discussion of the limitations of legal needs surveys, including (but not exclusively):

• They generally require sampling the general population, and the numbers reached with specific types of problem can be low, even before trying to stratify by demographics, problem severity etc.

• This presents problems when assessing policy impacts for specific groups. In particular where those affected are minority populations not well represented in general population surveys.

• They are expensive, time consuming, and can be tricky to implement properly.

• They need to be carried out regularly to understand change over time.

• In large countries, such as the US for example, they can be problematic where scale and logistics are more of a barrier.

The business case for continuing legal needs surveys was discussed – it was noted that in some countries it may be difficult to sustain the level of funding needed to keep them in place. Identifying the costs of unmet legal need is challenging, but given current financial pressures in many countries this was seen as an important way to highlight the case for future monitoring and evaluation.

Other techniques to measure legal need and the impacts of interventions

Alternative ideas and examples of existing approaches were discussed, including:

• Piggybacking on other general population surveys administered by others. Several participants made the point that, since legal problems cut across all aspects of life, a survey on housing, employment, education, health etc will touch on problems experienced and on how citizens’ rights have been affected.
• Better collection and sharing of administrative data (noting concerns that frameworks built around admin data can be unstable and contain hidden error).
• More digitisation and automated electronic data capture within the justice system. The UK experience will be of particular interest as we modernise our services.
• Combining sources e.g. spatial displays of the association between legal need and census-based measures of socioeconomic disadvantage (e.g. as used in Australia – shown in the screen shot taken from one of the conference presentations).

• Multi-methods evaluations to measure specific policies, which can be tailored to specific outcomes and provide opportunities to engage with stakeholders.
• Experimental designs: The World Bank Project presented on work using RCT and difference-in-difference designs, e.g. comparing outcomes for those who had assistance from paralegals.
• Data science approaches, such as use of open source and social media data, have potential, and more work could to be done to tap into their possibilities.

More generally, there was discussion that since justice issues cut across social policy areas, partnership working across sectors, and academic engagement, are crucial to help identify critical points in the wider system where interventions can be targeted. There is also a need to look beyond narrow definitions of effectiveness such as costs and case outcomes, and consider the broader impacts on individuals and those around them.

Examples of international comparative work

We saw some examples of tools which use consistent indicators to compare performance between countries. For example:

• The 2016 EU Justice Scoreboard, which focuses on factors that can help improve the quality of justice systems such as easy access to justice, adequate resources, effective assessment tools, appropriate standards.

• The World Justice Project Rule of Law Index: annual assessment tool based on general population and expert surveys to provide first hand information on the experiences and perceptions of people when dealing with the government, interacting with state bureaucracy, or resolving common disputes.

We also saw some examples of how data on legal needs and problem resolution can be presented in interactive ways, to help decision makers and the public people engage. For example, HiiL showed some visualisations (screen shots from the conference presentations below) being used to collate justice demand indicators, and advice-seeking behaviour.

Next steps and further information

Following the conference, OECD will be coordinating further work on mapping access to justice priorities in different countries and the strategies being used to tackle them.

Collating information on methods and lessons learned on evaluation design and implementation is of particular interest, as is understanding more about the range and quality of people-focussed measurement data and how it is used.

As a longer term ambition, OECD would like to explore the feasibility of standardising a core set of indicators to help implement and monitor SDG 16.3.

Another roundtable meeting will be scheduled in spring next year. In the meantime, a link to all the presentations given and to the full agenda is here

About the author

Cris Coxon is Head of Civil and Administrative Justice Research, Analytical Services, Ministry of Justice