RISE in India: Research Overview (Technical)

Improving school performance at scale: Indian states as laboratories for innovation, evaluation, and evidence-based education system reform

Principal Investigators: Professor Karthik Muralidharan (University of California San Diego and the Abdul Latif Jameel Poverty Action Lab at the Massachusetts Institute of Technology),  Dr Abhijeet Singh (University College London) and Dr Alejandro Ganimian (the Abdul Latif Jameel Poverty Action Lab South Asia)

Contracting organisation: The Abdul Latif Jameel Poverty Action Lab South Asia at the Institute for Financial Management and Research

Enrolment in primary school in India is almost universal, but learning outcomes are abysmally low: in 2014, only 48% of students in grade 5 could read a grade 2 text and only 26% could do simple division. Although there are myriad studies looking at the effect of individual inputs on schooling outputs, the inability of translating these findings into clear policy agenda results stems from a lack of comprehensive understanding of the ecosystem where scaled up interventions would be implemented. There is a need to understand what the feasible levers are that would enable education systems to deliver better learning.

The project proposes to examine three main reforms in the Indian education system across three states, including both quantitative and qualitative methods. RCTs will be used to generate random variation in the implementation of the reforms such that a rigorous assessment of the impact on learning processes within schools and classrooms can be done. The team will also conduct qualitative fieldwork into each of the reforms to document the institutional framework surrounding the reforms in each state to understand how the reforms affected the system as a whole. This will allow for insights into how the interventions are implemented in the existing system, organisational and political economy constraints and dependence of the reform on political or organisational circumstances.

The team also plans on collecting several new datasets, including student achievement tests, surveys of student background and perceptions, surveys of principals, classroom observations, teacher attendance and punctuality and teacher knowledge assessments.

The unifying theme behind the proposed studies is of improving education governance in India, which has been identified as one of the key binding constraints to better system performance. 

This proposal takes a systems-level approach by looking at two types of relationships between the key set of actors within the Indian school system:

a.  accountability (monitoring) relationships and

b.  information (collaboration) relationships. 

 

Source: India technical proposal, based on World Bank (2004) and City et al (2009) among others.

 

They are depicted as the solid and dashed lines respectively in the diagram. To understand these relationships within the system, and how the system affects these relationships, the team proposes to look at three to four interventions.

Across all interventions, the focus is solely on public schools. As these are attended primarily by students from disadvantaged backgrounds, equity concerns underlie the whole proposal.

  1. Madhya Pradesh School Quality Assessment: high/mid-level bureaucracy, principals, parents

MPSQA intervention aims to change the interactions between multiple actors in the state education system: mid-level bureaucracy and principals, high-level bureaucracy and mid-level bureaucracy, principals and parents. It is a system of continuous improvement based on monitoring, diagnostic feedback and changes in school management and instruction.

The intervention is along the lines of orienting the entire system towards continuous improvement. Following a successful pilot, the MPSQA was extended to 2,000 schools in 2014-15, with plans to expand to 20,000 in 2016 and eventually to all 120,000 government schools in the state. Add-ons to the programme will be randomised, including (a) public release of school quality data, akin to Ofsted reports in the UK; (b) increased frequency of follow-ups and/or hiring additional staff to build the administrative capacity of the state. A key question here is to understand whether such reforms can affect student achievement at scale and whether the scale-up dilutes implementation quality. Finally, it is also important to understand the mechanism that allows for improved effectiveness.

  1. CCTV in classrooms in Delhi: mid-level bureaucracy and teachers

Teacher absenteeism and time on task are serious issues with education systems in India (and elsewhere). Thinking about the system and theory of change, this reform intends to change the interaction between mid-level bureaucracy and teachers. The different treatments will shed light on different possible mechanisms: is it monitoring that is the binding constraint? Is it information on what is being done/benchmarking? Or is it information with incentive to act (bonus)?

The Government of Delhi plans to install CCTV cameras in 1,100 schools (22,000 classrooms) by the start of the 2016-17 school year. There is government commitment to improving service delivery.

The team proposes to randomly assign 600 schools in the 2016-2017 school year to one of four experimental groups to shed light on each possible mechanism behind the interaction between mid-level bureaucracy and teachers:

  1. no-CCTV control group;
  2. CCTV only: is the binding constraint insufficient monitoring?
  3. CCTV + time on task report: is it insufficient information about performance benchmarks?
  4. CCTV + time on task report + linked performance pay: is it lack of teacher motivation to act on information? Can financial incentives provide this motivation?

This reform is particularly helpful in understanding system-level effects because it targets understanding of the mechanisms of weak governance as well as the aspects of poor pedagogy. Furthermore, as this intervention is mandated by the government rather than an NGO, there is substantial power behind the mandate and accountability relationship.

The feasibility of the following proposed Rajasthan reforms is still being assessed and will be subject to confirmation in late 2016.

  1. Consolidating schools into larger units in Rajasthan: principals and teachers

There is a fundamental trade-off in designing school systems in developing countries between access and scale. Broader access in India has resulted in near universal enrolment but often the enrolment is in sub-standard small schools. The Government of Rajasthan has a programme to consolidate small primary schools (typically without a principal) with neighbouring secondary schools to bring them under the leadership of secondary schools: the “Adarsh” school consolidation programme. This reorganises the structure of schools, and the reform also provides teachers with pedagogical tools and upgrades school infrastructure.

Thinking about the system, the reform changes the interaction between principals and teachers. It requires monthly staff meetings focused on academic issues, regular inspection of classes and homework, and introduction of remedial classes. It will also change organisation and pedagogy by ending multi-grade teaching. The RISE India team proposes at this stage to undertake a feasibility assessment of a high quality evaluation of the program which may utilize randomized allocation, regression discontinuity design methods exploiting program rules of assignment and panel based methods such as value-added models.

  1. Public-private partnerships in Rajasthan: schools, parents, mid-level bureaucracy

The reform seeks to contract out the management of new and existing government schools to private education providers. The reform targets weak governance and pedagogy and allows schools to have substantial operational autonomy within minimum government standards.

Thinking about the system, the reform aims to change the interaction between multiple actors in the state education system: (a) schools and parents, by increasing school choice and competition; (b) mid-level bureaucracy and principals, by relaxing political and bureaucratic limitations relative to public school principals. Effects would be identified by student- and market-level randomisations. First, the admissions process would be conducted via a lottery for the PPP schools. Second, the locations where PPP schools would be allowed to operate would be randomised, allowing for an assessment of the effects of these schools on neighbouring schools.

In short, the study of these three to four reforms will shed light on different parts of the complex system of relationships in the Indian education system. The different treatment arms in the RCTs will allow for clean identification of the existence and magnitude of any effects, while the qualitative work will allow for a deeper understanding of the mechanisms and context of change.

The team is committed to communicating the results of the reforms and interventions individually, but most importantly also how they come together to form an understanding of system level reforms and what this means for policy. The precise output will depend on progress of the reforms and might be a monograph or an edited book on schooling reforms in India.