During the past decade, the distribution of content over the Internet and its consumption on computers and mobile devices has disrupted several industries — newspapers, book publishing, music and films, among others. Now education joins that list, thanks to the emergence of massive open online courses, or MOOCs. These courses, which are offered for free to tens of thousands of students, cover topics ranging from artificial intelligence and computer science to music and poetry appreciation. As millions of students around the world flock to participate in MOOCs, universities are being compelled to rethink what it means to teach and to learn in a networked, globally connected world. During the past 18 months, many educational institutions have initiated or joined ventures that can help them explore, experiment in and gradually understand this phenomenon.
Among the most active MOOC providers today is Coursera, a start-up that offers some 200 online courses to 1.5 million students. It does so by providing a technical platform to 33 educational institutions, including the University of Pennsylvania. Other MOOC initiatives include Udacity, which originated at Stanford, and edX, a venture of Harvard and MIT. How do MOOCs deal with the challenge of scale posed by the massive numbers of students they attract? How do they retain and evaluate their students? How can they monetize their free content? Knowledge at Wharton posed these questions and more to Daphne Koller, co-founder of Coursera, during her recent campus visit.
An edited version of the conversation appears below.
Knowledge at Wharton: How does a massive open online course (MOOC), like the ones taught through Coursera, differ from the traditional educational experience? What are the pros and cons?
Daphne Koller: The experience differs from the traditional face-to-face educational experience, but it varies based on which audience you’re talking about. MOOCs have been offered to the general public, but they are also a way of transforming the educational experience of on-campus students.
Students in the general public have a tremendous opportunity to be exposed to knowledge that has, until now, only been available to a tiny sliver of the world’s population. These students don’t get the same experience necessarily as the on-campus students, because they don’t get that one-on-one connection with the instructor. But on the other hand, the availability of this content is a tremendously enabling experience for those students. It is still a hugely interactive experience in terms of working with the material, which is not just video. There are a lot of exercises and assessments. Furthermore, an educational community is created based on students interacting with each other.
At the same time, students on campus can benefit from that same content and have the experience of going and watching the content outside the classroom and then coming into class and having more time for active learning and an interactive experience with their instructor.
Knowledge at Wharton: How does Coursera fit into the higher education landscape? Does a MOOC supplement traditional learning as you just described it, or do you see it as a substitute as well?
Koller: I think it depends on the population. For those students who are privileged enough to be enrolled in an academic institution such as Wharton, Princeton or Stanford, there is a tremendous opportunity to supplement standard instruction and repurpose classroom time for a much more meaningful and interactive experience.
However, many students will never have access to this quality of education. So you can view this as a substitute for the education that they would have wanted. Many students taking our courses are, in fact, already educated professionals. They have a degree or sometimes have an advanced degree. They want to expand their minds and learn new things. They will never go back to school in a formal setting, whether for financial or geographical constraints, but this provides a tremendous learning opportunity.
Knowledge at Wharton: I understand 33 institutions offer MOOCs through Coursera. How many students have they served?
Koller: That’s right. We currently have more than 1.5 million students enrolled on our platform. And there are five million distinct course enrollments because students often enroll for more than one class. However, a lot of these students have not yet been served because many of our nearly 200 classes have not yet started to run. I think we’ve served hundreds of thousands of students so far, but not the full 1.5 million that have been enrolled.
Knowledge at Wharton: What have you learned so far about the participation, completion and dropout rates of this online population?
Koller: This is what we see: Enrolling is easy. It’s a matter of just clicking a button, and it’s free. A lot of students do that. Then when it comes time for the course to actually begin, about 70% of students show up, possibly because those who do not attend have found their life has moved on, and they’re now busy doing something else.
Of the ones who show up, we see a bifurcation of the population. There are students who come in and they are primarily there to watch the videos. Of those who start watching videos, about 30% to 40% watch the course to its completion, with a fairly constant drop off rate week-on-week as students get busy.
Then there is the second population: These are students who really do intend to take the course for real. We gauge that by seeing who submits the first assignment. If you submit the first assignment, you probably intend to take the class for real. The retention rate for this population is comparable: About 30% who start the first assignment will complete the last one.
In terms of what you might traditionally call “retention” — that would be the number of students who submit the final assignment relative to the population who initially enrolled for the course. In those terms, we have a retention rate of 7% to 9%, depending on the course. But that’s really the wrong way of looking at it because many of these students never really intended to seriously take the class in the first place.
Knowledge at Wharton: I understand that the gamification course that was offered through Wharton had 70,000 people who initially enrolled, and about 9,000 completed it. That’s quite a big gap. What have you learned about the different kinds of courses where you have seen greater success than others?
Koller: We don’t have a sufficiently large sample size right now to gauge what makes for a successful class and what doesn’t. In fact, we’re currently researching what factors increase retention and what factors increase engagement.
We’re now running a test where students are sent encouraging e-mails once a week saying, “You did so well on the last couple of assignments, and the next one is due tomorrow. Don’t you think it would be good if you log in and try to complete the next one? You don’t want to break your winning streak.” We are trying to test whether that actually increases student engagement.
One of the really beautiful things about having such a large population of participants is that you can run these experiments and get statistically significant results that will help you improve the learning experience for students.
Knowledge at Wharton: Some of the institutions that offer these courses offer credit, and some do not. Why does that difference exist?
Koller: Currently only one institution, the University of Washington, offers credit. They’re offering credit for an enhanced version of a course where students have interaction with the instructor and some additional homework besides what’s available on Coursera.
I think that granting credit is a very sensitive issue for institutions; they need to think carefully about what makes a credit-bearing class and what doesn’t. The transformational change we’re seeing right now is so new. It was only a year ago when the first real MOOCs launched. The educational community is still trying to assimilate the change and figure out where to take this. I believe that over time more institutions will move into the credit arena, but it’s still early days.
Knowledge at Wharton: Do any institutions offer certification?
Koller: Most of the institutions we’re working with have agreed to offer some kind of unofficial certification. But most of our current population are people who already have a degree, so the question of whether the course carries credit is less relevant compared to whether they have a meaningful certificate that they can present to an employer to demonstrate competence in a particular topic. I’m very pleased that most institutions have decided to offer that benefit to students, which allows them to gain something substantial for their work.
Knowledge at Wharton: Is it Coursera that offers the certification or the institution? And is there any charge for it?
Koller: So far we’ve been able to offer a statement of accomplishment or something comparable: It’s an unofficial letter from the instructor with the instructor’s signature that does not carry university authorization. For some courses in the coming months, we plan to offer university-branded certificates explaining that the certificate was awarded for an online, non-credit bearing course. This will be different from what the university offers its own enrolled students, and will also carry the Coursera brand. When we begin to offer those certificates we believe there will be a small charge because this has substantial value for students.
Knowledge at Wharton: When you have certification, the question of assessment becomes very critical. Given the large volumes you have in these MOOCs, I would imagine that would be a challenge. How have you dealt with that?
Koller: We have devoted extensive attention to assessment and feedback. We believe that exercises where students receive meaningful feedback are a critical part of the learning experience. If you don’t get feedback, you may not complete the work or learn the material properly.
We’ve developed an extensive set of exercises that can be graded automatically by the computer. There’s the fairly mundane multiple choice, fill-in-the-blank or numerical answers. There are also in-depth assessments to grade — for example, programming assignments and Excel spreadsheets with financial models.
We’ve also recently introduced peer grading to roughly a dozen classes, where students critically assess one another’s work and provide both a quantitative grade as well as qualitative feedback. By combining ideas from the peer review process that’s used in academia, with the crowd sourcing ideas that are used on websites like Wikipedia, we believe we can provide students with meaningful feedback for the open-ended work that is so critical for subjects in the humanities and in business.
Knowledge at Wharton: I understand that peer-reviewed assessments help to address the scale question. But isn’t the absence of feedback from a qualified instructor a limitation?
Koller: In certain disciplines and contexts, it is a limitation. For example, I would not use peer review to grade 10,000-word essays where an instructor would be able to provide an in-depth assessment and give substantial, expert feedback. This is not going to be an experience that is comparable to the experience that a student attending the University of Pennsylvania will receive. There will be differences.
The question we should ask ourselves isn’t whether we’re going to achieve equality between students at the University of Pennsylvania and students in the general public. Instead we should ask if, through the use of technology, we have improved the quality of the experience for each of these students separately. We want all students to be better off than they were before.
Knowledge at Wharton: You said that a lot of the people who take these courses are already professionals who have degrees. But there may be some young people who are trying to become professionals. How do you address issues related to verifying a student’s identity or plagiarism? How have you dealt with these issues?
Koller: As the mix of courses that we’re offering changes to include more introductory classes, we’re seeing an increase in the population of 18- to 25-year-olds, and often even younger students from high schools. I think that’s a really exciting direction because it is an important segment of the population to which you want to offer high quality education.
There is a range of technological solutions that one can adopt to address issues of plagiarism, cheating and identity verification. But keep in mind that you will never be able to stop cheating entirely, even in the most controlled environments. It’s always going to be an arms race. If you prevent some forms of cheating, others will emerge to replace them. You have to set realistic expectations.
Meanwhile, I believe discussions about cheating on online courses have been inflated relative to the actual amount of cheating that takes place. For example, in the online Princeton sociology class, the teaching assistants graded every single one of the 2,500 midterms. They estimated the proportion of plagiarism was around 5%. Even if that’s a low estimate and the actual rate is closer to 10%, is that really so much higher than what we see in on-campus classes? Not clear to me.
Knowledge at Wharton: I have some questions about the business model. Coursera is not the only MOOC initiative. I wonder how you compare your approach to that of edX or Udacity.
Koller: Let’s start with the Udacity comparison: We are a technology platform that enables the best institutions to offer their education to the rest of the world. We’re not intending to become an educational institution in our own right. We do not produce our own content. We rely on some of the world’s best instructors to provide content and set academic standards. We distribute amazing content by having relationships with some of the world’s best universities. That’s one comparison.
The edX initiative, in some respects, is more similar to what we’re doing because it’s also an institutional-level effort. It’s a small consortium of some of the world’s outstanding institutions. I think that it’s wonderful for students around the world to have access to content from those universities as well. This arrangement between institutions provides economies of scale, since a single platform is an expensive and complicated thing to develop. We also believe it’s beneficial to have a single place where people can find a very large amount of content. We have almost 200 courses right now and more coming up on this hub. That’s why we have 1.5 million students, and the population is growing.
Finally, I think an advantage that may not be apparent from the outside is having collaboration across institutions, which allows institutions to learn a tremendous amount from one another. Online learning is a completely new paradigm in teaching. This is not your standard class being videotaped and placed online; this is a real shift in how we teach. We’re all kind of learning together. Having peer institutions that are all communicating with [each other] helps to develop sound pedagogy and good ways of exploiting this new paradigm and this new medium so much faster.
Knowledge at Wharton: Since Knowledge at Wharton has been giving away free content and free knowledge for the past 14 years, this is a question that is very dear to my heart. There is clearly a cost to producing the content. How do you plan to monetize your service?
Koller: Yes. There is not only a cost to produce the content, there is also a cost to host and disseminate the content. There is also a cost to develop the platform. We cannot simply count on some miraculous influx of funds from philanthropic foundations to keep us going indefinitely. We have to make this venture sustainable.
We have several ideas on revenue models, all of which remain to be tested because we’re still in our early days. But one strategy, discussed earlier, is the potential to charge for certification. Even though we’re committed to keeping the content free for everyone, if you get a certificate at the end that allows you to go and apply for a job and gives you some kind of tangible benefit, then it’s reasonable to charge for that.
We also believe employers will put in money to be granted access to the records of students who have taken some of our courses. Employers would receive much more quantitative and detailed information about these students than just their resume, which is what they say about themselves. This way, employers can learn if the students did well, both in hard skills, like on the exercises, and in soft skills, like helping others in the discussion forum and in study groups. That’s really important information in a 21st century workplace, and I believe employers would be willing to pay for this. We have already had some interest from employers about this.
Knowledge at Wharton: Will the institutions that are part of Coursera also monetize their content in some way?
Koller: When we bring in revenue, the university gets part of it. We share the revenue with the institutions, specifically to support content production.
Knowledge at Wharton: What’s next for Coursera over the next five to 10 years?
Koller: I think that in five years we will be able to offer most curricula in most disciplines. Eight months after starting, we’re now able to offer 200 courses. A typical institution has something like 3,000 courses. I think in five years we will have 3,000 courses, so that if you want to be educated as a mechanical engineer or as a businessperson or as a lawyer, you will be able to access some of the world’s best courses for your field.
I also think that in five to 10 years, from the perspective of the higher education ecosystem, people will look back on the 20th century and say, “I can’t believe that we spent so much of our students’ time shoveling them into auditoria and having them sit there for 75 minutes while somebody lectured at them.” We will all clearly recognize that this is not the best form for getting people to learn material and use it effectively. I think our notion of what makes for a good education will shift drastically.
