Blog

Khan Academy is a global, not-for-profit organization working with the mission to provide free, world class, education to anyone anywhere. Khan Academy platform hosts world class content for various subjects, including math, and has been designed to meet the needs of students and teachers. Students have an opportunity to learn at their own pace and gain mastery in concepts, while teachers can readily understand student- and classroom level progress on their dashboard.

We caught up with Sandeep Bapna, Vice President of International Operations at Khan Academy, who talks about the organisation’s view of the current EdTech ecosystem in India and the need for EdTech solutions to build an evidence-led strategy towards shaping quality-products. EdTech Tulna evaluated Khan Academy’s Math offering for grades 1 – 10 in 2021, the results of the evaluation are available on EdTech Tulna Evaluation Center.

1. What are your observations about the recent boom in the EdTech industry in India, specifically with regard to building system capacity?

Most of the action that has happened in the EdTech sector in the last few years has been in the D2C (direct to customer) space comprising a large share of the private players. When I joined Khan Academy back in 2016, the concept was almost non-existent. There were few places where any use of technology was happening (technology for learning). It was towards 2018-19 that we started seeing a high degree of openness about bringing in technology-based solutions in classrooms. That’s the period during which we were running pilots in Jawahar Navodaya Vidyalayas and Kendriya Vidyalayas across Kerala and Rajasthan. I think we were on a great trajectory and then the pandemic happened.

While there was an initial response in which governments and players launched digital resources, many of these solutions were ushered in without adequate capacity building support. As time went by, this exposed the need to develop capacity, the need for teachers to see many of these products as tools and as assistants (to their teaching process) rather than something which is complementary to what they do. So I feel there is a long way to go on that front.

Product companies have a key role to play in addressing the capacity gap because it speaks directly to product design, whether the product can be easily integrated into classroom instruction. A critical use-case for EdTech products is for teachers to use it as a resource in the classroom for remediation, on-grade learning or acceleration, and to use the data insights in real time to provide personalization and differentiation. The product itself has to be easy to use.

2. Could you elaborate on the role of evidence for an EdTech product in the current Indian and global ecosystem; and how Khan Academy approaches evidence-building?

Evidence is critical. When Tulna was at a concept-stage, I remember having a chat with the team. Difference between business and consumer products (vs non profit products such as Khan Academy) is that you are selling something (directly) that has some value proposition and the consumer has a choice to use the product or not. However, when you start bringing EdTech through teachers into classrooms, that’s when states and large school systems have to make decisions for a very large body of students and teachers. That’s a very different decision from an individual decision about using a product. Evidence plays a huge role in that. Tulna has done a great job in setting out the criteria on which different products are measured. I feel one additional piece that we will have to think about is that there is a product and (then) there is a service component, especially if the context is a classroom involving teachers and students, which many times affects implementation i.e. you may have a great product but it may not have the intended impact if the implementation is weak, so the service component is equally important.

Khan Academy conducted its first study within 3 years of inception and today more than 50 studies have been done on Khan Academy globally. For us, the evidence and efficaciousness of the product is super critical. As we started in India, even the first pilot in India was set-up to study the base line, end-line impact. Since then, we have done 4-5 studies in-house that cover approximately 45,000 students in India. The objective is to see if there are any gains and then to understand what the gains are based on, for example, (a) is it due to the time that the students are spending on the product or (b) is it due to the skills that the students are getting proficient at. This has been a continuous process for us and nearly 10% of our team is (focused on) data and efficacy who look at this on a continuous basis. (Hence, evidence generation helps product developers to learn and develop more efficacious products and programs).

On the implementation front, an important question (we focus on) is how to engage students and teachers on a longer term basis, where EdTech solutions are not used because it’s being monitored in a top-down model but it is being truly incorporated in the learning journey. There is much research to be done on this front and is something that I am personally excited about with the work Khan Academy is doing in India. We are now operating at a scale where the lessons learnt can hopefully be more generalizable and taken to other contexts across India and across the world.

3. EdTech Tulna is a first-of-its-kind tool not just in India but also globally. What was the motivation behind Khan Academy’s engagement with EdTech Tulna for an evaluation and what value did it unlock?

First motivation was that (EdTech Tulna) it is a public good and it should be out there. Khan Academy was very excited to see a tool like Tulna being created and wanted to be a part of it. We didn’t know how we would get evaluated, whether we would come out nicely or not. Our evaluation brought out concrete insights, especially around localized content that we were happy to share with our in-house content team. (Tulna evaluations are free of cost providing independent and unbiased review of product design). Second, I think Tulna is a pretty solid product for decision makers to look at.

In our experience, decision makers are well-acquainted with Khan Academy, however, they don’t have a granular understanding of the product. Tulna evaluation provides that base of knowledge and comforts decision makers that the product has been looked at and vetted by an independent organization. Several state governments, as they advance in their journey towards digital products, are taking bold bets and I believe EdTech Tulna gives them a base level of assuredness.

4. We are keen to learn about Khan Academy’s work with the state and central governments in India. How have you leveraged your experience of working with communities & stakeholders for India 2 & 3?

When we started out, all we could think of offering were small pilots, whereas, the state required solutions at scale. If you have an EdTech product, scale should not be a barrier. So, it forced us to think – what are the true barriers to scale? Of course devices and availability will always be limiting, but let’s say even where there are devices, we went from 12 schools to 7,000 schools in 3-4 years time.

From 6,000 students to 1.2 million students in government schools in that period. And I think what was really interesting is that we were responding to this need and demand, which forced us to think about different parts of our delivery model, The ah-ha moment for me was that traditionally we start via small pilots and then scale up sequentially, 2x and then another 2x, whereas the need is to go to 200x. Now you could do this sequentially, but interestingly, the challenges at scale look very different from the challenges at the pilot level. And in fact, I am more and more inclined to think that we may be learning some wrong lessons at a pilot level. Because that is what we are looking at, that is what we are considering, and so we try to optimize for that, but then when you get to scale you have to solve for something completely different. (the type of challenges encountered may be similar or different, however, the approach to solving that challenge is significantly different at small-scale vs large-scale, such that it becomes a new set of challenge to solve completely).

I’ll provide a concrete example of that, one of things that we have started this year is a live call center. A call center that’s live during school hours, where teachers can call (for troubleshooting). Why did we do that? One primary driver for this is that we saw (from back-end data) that a lot of teachers were dropping off when they were experiencing the product in the classroom. They tried to do something that didn’t work, they had no help and they thought “okay forget about it I can go back to my textbook and what I was doing”. This (the call center) is still in its infancy, we want the call volumes to go up significantly because that is what the idea is – for teachers to use this facility if something doesn’t work for them in the classroom with the product. It also starts generating a lot of insights for us because we are now talking and hearing from teachers and discovering new insights. So this is an example of a challenge that in a 50-100 school pilot, you’ll have 2-3 project managers to sort out issues, however, when you have to solve for that at scale, you need to approach it differently. So that’s just one concrete example. But there are many others, and we are beginning to think more and more that you need to really solve for scale, figure out the challenges, and then solve for those challenges rather than the other way around.

As the EdTech market in India grows both in volume and complexity, product companies are increasingly valuing the role of generating evidence for improving and for signaling the efficaciousness of their products. To spur quality-aligned innovation in the supply-side ecosystem, Tulna has developed product design standards for ‘what good EdTech looks like’ and is evaluating a diverse portfolio of products to measure their quality against its standards. For further information on evaluations please contact us here.