A new working paper from the RISE Indonesia Country Research Team has the incredible benefit of having panel data on learning—tracking the same children on what they know over time—a rarity in the development space. By assessing children over time, we can see the gains and also whether or not skills are retained. 

Members of the RISE Indonesia team take up skill acquisition and retention with the Indonesia Family Life Surveys data, looking at skills on some simple arithmetic questions. They use data on children in 2007 and 2014 which includes questions ranging from quite simple (e.g., 49 - 23) to slightly more difficult (e.g., reducing ⁵⁶/₈₄ and ¹/₃ -¹/₆), to more difficult (0.76 - 0.4 - 0.23 and 5% interest on Rp75,000). All of these represent skills that should be gained during a student’s primary education—and hopefully retained into adulthood. The authors use Item Response Theory to generate a score for numeracy skills for each child in each testing year that is the mean probability that the child answered any given question correctly. 

They analyze two groups: those who were 7 to 9 years old in 2007 (and hence 14 to 16 years old in 2014), and those who are 10 to 12 years old in 2007 (and hence 17 to 19 years old in 2014). What they find is astounding. Over the seven years most of these children see almost no gain in their ability to complete math problems. Those who are 7 to 9 years old in 2007 on average had about a 30 percent probability of answering a question correctly in 2007. By 2014 that probability had increased by only a few percentage points—despite seven years of potential learning.

The findings for those aged 10 to 12 in 2007 are even more drastic—their likelihood of answering a math question correctly declined by 2014. They were actually less likely to answer correctly than seven years earlier. To dig deeper, the authors divide this cohort into their household’s wealth quintiles, using an asset index. They find (Figure 4.11) that numeracy scores declined for all but the wealthiest income group. Among the poorest quintile in this cohort, the likelihood that children could answer a question correctly declined by 5.5 percentage points—from a low base of less than 30 percent to begin with. And, because these are the average changes across lots of children, this means some children surely gained skills over those seven years, while some children’s ability declined even more than the graph indicates. 

These findings have multiple implications:

1. Retention is likely a critical issue in the learning crisis. If children are not only not gaining many skills, but actually losing skills, then for many even the limited skills that are being gained are not being retained. The ASER survey of youth 14 to 18 in India, Beyond Basics found a shocking lack of skills, even for things as simple as telling time, even among youth who had attended many years of schooling. 
2. This lack of retention could indicate a lack of conceptual masteryin the first place, where, for example, children may gain the skill of being able to answer simple test questions correctly through the rote learning of an algorithm, but never really grasp the underlying concept. This purely rote or memorization of skills is perhaps less likely to persist. 
3. That some groups, in this case poorer children, retain less than other groups, poses concerns about equity and differential learning and mastery among different segments.
4. Poor retention indicates contemporaneous cross-sectional learning profiles (such as those generated from ASER and Uwezo data, which test learning among children) may actually overestimate learning when defined as conceptual mastery that will be retained. This is concerning since many of these learning profiles already suggest very low learning levels.
5. This also suggests retrospective learning profiles (such as those developed from DHS or FII data which test skills among adults) are an important complement to school and contemporaneous surveys of children as they measure learning that is retained and carried into adulthood. 

As international goals shift to addressing the learning crisis, we may need to address a learning retention crisis as well.