When technology first began to be used in schools, it was limited to setting up computer labs to equip students with basic computer skills. Since then, EdTech interventions have evolved from their earlier, more limited avatar. In their current version, they take the form of advanced solutions that are designed to support and enhance teacher-led learning and improve student outcomes. There is ever-increasing evidence, from countries like China, Ecuador and Pakistan, that points towards the potential of EdTech to bridge the learning gaps that students face.
In India, however, the evidence for EdTech is still nascent and in fact, needs to be strengthened to promote wider adoption. Most of the EdTech products available today are beyond the paying capacity of the majority of Indian parents and school systems, and largely cater to students from the high-income group. For EdTech to be able to create impact at scale, it needs to address a whole set of different requirements, such as bringing in contextual and vernacular solutions for the low-income segment, accommodating a wide range of learning levels, and being affordably priced in order to be relevant. This is what the EdTech Lab initiative launched by Central Square Foundation (CSF) is trying to highlight and address.
The EdTech Lab was conceptualized in 2018 with two key objectives. The first was to generate evidence on the usability and effectiveness of the EdTech solutions presently available in the market. Till date, the only clear example of a solution with tangible evidence of positive impact on learning outcomes is Educational Initiatives’ pedagogy-based Mindspark. When implemented in after-school learning centres serving middle-school students attending government schools in Delhi, it led to an improvement in Mathematics in just 4 and a half months. Beyond this, the lack of credible evidence and appropriate standards to evaluate EdTech has hindered the decision-making process of governments looking to procure such solutions for their state. This lack of evidence on other products in the market, therefore, presents an opportunity to start building it. Secondly, the currently available solutions need to be contextualized for the low-income, government school segment, as most of them have been created keeping high-end users in mind. In this way, this initiative aims to connect demand and supply by creating pathways for EdTech adoption in a more meaningful way.
In fact, the EdTech Lab is one of the first attempts to systematically build evidence around EdTech solutions in India. Bikkrama Daulet Singh, Co-Managing Director at CSF, explains, “By addressing the evidence gap in the EdTech space in India and generating impact in the low-income segments, we are trying to help the ecosystem understand the potential of meaningful deployment of EdTech. For product companies, on the other hand, this is an opportunity to evaluate their product using expert-designed tools and boost their credibility.”
For the first cohort of the EdTech Lab, the focus was on math and literacy products for the K-8 segment. CSF conducted an initial landscaping exercise that indicated the existence of over 4,600 EdTech products in India. But, when narrowed down to Hindi and Mathematics products targeting the K-8 segment, only about 10% were eligible. After a thorough shortlisting process, the first cohort of 12 products was selected and each product was invited to participate in a ‘rapid evaluation’ – a short-cycle evaluation that spanned over 5-6 weeks per product. The objective was to gauge their usability and effectiveness in their home ground, as well as assess their potential for contextualization for the low-income segment. On the other side, the organizations behind these products were drawn to the evaluation due to its uniqueness, because it was for the first time that such an exercise was being taken up in a structured manner. The evaluation was carried out by IDinsight, an experienced evaluating agency that uses a wide range of analytical tools to rigorously test ideas and provide actionable insights.
The rapid evaluation itself was a three-pronged process which synthesized: (i) data from qualitative interviews of students, teachers, parents and headmasters; (ii) quantitative data generated by students’ usage of the products; and (iii) expert evaluation of the products for their pedagogy, UI/UX and tech support. The research questions were prepared to capture information not just about the product, but also on its implementation, which is what ultimately makes any solution effective. Rob Sampson, Associate Director at IDinsight who led the rapid evaluation process, says, “The rapid evaluations helped understand how, and how well, the products were working in their current environments. They also helped anticipate how the products may function in government schools at scale.” Sonakshi Sharma, Manager of the rapid evaluation process, adds that when it comes to contextualization of EdTech products for lower-income schools, “adapting to a low infrastructure environment is a big concern, among many other considerations.”
After an extensive process that covered over 1,500 school staff members and students and analyzed product-generated user data for over 17,000 students and 3,900 schools, the findings revealed that the overall performance of each EdTech product depended on the details of its features and how it was implemented. In most cases, the content in Hindi was found to be in need of improvement. Another key finding was that the major barriers to the usage of EdTech solutions in government schools include lack of adequate hardware, lack of adequate teacher training and fear of technology. While the EdTech Lab report goes into more detail to provide insights for researchers, funders, product companies and implementers, these EdTech Lab evaluation tools will soon be made openly available and may be useful for other product companies to evaluate their own products and make improvisations to increase their usability quotient. They can also be used by governments and investors to evaluate EdTech solutions for the context in which they want to scale.
The findings from the rapid evaluations have provided clarity for the next stage of the EdTech Lab process, which is to contextualize the available solutions to the needs of government schools. Based on the findings, two personalized adaptive learning products from this EdTech Lab cohort were identified by CSF. These products will be given financial and strategic support to customize their solutions for the specific requirements of government schools. ConveGenius is one of the products, which is already being used in more than 500 schools with over 75,000 users, 83% of whom belong to the low-income category. The support provided to them includes language translations and bridging platform gaps noticed during the evaluations. Jairaj Bhattacharya, Managing Director of ConveGenius, gives an example, “In the mock test template, the tendency towards guesswork by students was minimized by flipping the multiple-choice-type questions into fill-in-the-blanks or match-the-following type of questions.”
After the process of contextualization, the products will be set up to undergo deeper and more rigorous evaluation through ‘test-beds’, which would span almost a year, to help thoroughly understand any adoption and implementation issues in their target segment. Through feedback from the end-users, i.e., teachers and students, the objective of the test-beds is to arrive at a scalable and effective model of EdTech implementation. This will eventually help the product to get absorbed meaningfully at the ground level. Currently, product contextualization and test-beds are planned to take place in government schools across Andhra Pradesh.
While test-beds are process driven, they will create a segue for an impact evaluation to assess the efficacy of the EdTech intervention towards improving student learning outcomes. This will offer the right inputs for decision-makers (including governments and funding agencies) to procure and invest in the right solutions, and for product companies to improvise and contextualize their platform and content to enable an ever- wider reach. Ultimately, the EdTech Lab process intends to contribute to the creation of evidence and knowledge in the EdTech ecosystem. The journey has just begun.
Learn more about the EdTech Lab, view an infographic on the main findings, or download the full report.
