As Imagine Worldwide scales its tablet-based learning program across Sub-Saharan Africa, effective program monitoring becomes critical to ensure quality and consistency. Topic modeling can provide a way to quickly analyze qualitative data collected by field officers to identify emerging issues.

We tested the feasibility of topic modeling as a monitoring tool – using the “LDAgibbs” package in Stata – with field data from Liberia and Ghana. Here’s what we’ve learned so far.

Why Topic Modeling?

As we scale, we accumulate a vast number of observations from field officers who log qualitative feedback about various aspects of the tablet program: how well students use tablets, classroom setup, and technical challenges, among others. Analyzing these comments manually would be incredibly time-consuming, so we tested “Latent Dirichlet Allocation (LDA)”—a machine learning technique that groups words into “topics” based on word frequency patterns. This could allow us to quickly summarize key themes across thousands of comments without having to read each one individually.

Key Findings from Our Pilot

We conducted a pilot with data from 19 schools in Liberia and 13 schools in Ghana, involving over 500 observations. Using the LDAgibbs model, we identified five unique topics that addressed program performance and program issues. This analysis led to the following insights:

• Program Strengths: Comments related to support and supervision indicated that teachers and volunteers were actively helping students with the tablets. 
• Challenges: Recurring comments about classroom noise appeared to be caused by students standing outside the classroom windows and faulty audio cables. These insights helped us take immediate action to improve the learning environment.

Takeaways and Future Improvements

The results of our topic modeling pilot were promising. We were able to sort quickly through hundreds of observations to identify key topics and themes. This method could also help identify common topics that could become closed-ended (quantitative) items on observation protocols in future. Potential users should consider some challenges related to the method, however. Topic modeling works best with at least 1,000 observations in a time period and with comments that are at least 50-100 words long. Nevertheless, we found it useful with somewhat less data. 

Implementers and researchers may want to consider topic modeling as a possible valuable tool for monitoring programs at scale, particularly where large amounts of qualitative observations and open-ended comments are collected.

For more about our work, visit our resources page.