Wharton professor Christian Terwiesch said he “fell in love” with ChatGPT after reading its answer to an exam question in his graduate-level operations management course.
The professor fed the question to the controversial new chatbot as part of an experiment to see how the software would perform on a typical test. When prompted to explain the bottleneck process at a hypothetical iron ore factory in Latin America, ChatGPT aced it.
“Wow! Not only is the answer correct, but it is also superbly explained,” Terwiesch wrote in a recent white paper about his experiment. “I don’t see any reasons to take points off from this answer: A+!”
ChatGPT has been making headlines since it was launched in November by San Francisco-based OpenAI, mostly because it can be used by students to cheat on homework assignments and exams. The artificial intelligence software, which OpenAI made available to the public, can produce high-quality written responses to complex questions in a matter of seconds. Some public school systems, including New York and Seattle, have already banned the use of ChatGPT on their devices and networks.
Concerns about plagiarism are legitimate, said Terwiesch. But the professor, who chairs the Department of Operations, Information and Decisions and serves as co-director of the Mack Institute for Innovation Management, sees expansive potential in the chatbot’s artificial intelligence. With its incredible speed and accuracy, ChatGPT can be a powerful tool to improve the teaching process, customize learning, make business more efficient, and save precious time that could be used more productively by humans.
“We have new technology in the game now, and it would be a pity if all we could do as a society is be as good as we were before having that technology,” Terwiesch told Wharton Business Daily on SiriusXM. “This is going to be big, and there is reason to believe we have only seen the beginning.”
Wharton management professor Ethan Mollick shares that belief. In an opinion piece for Harvard Business Review, he wrote that ChatGPT is a “tipping point” in artificial intelligence. The technology is far better than previous iterations, making it more than just a clever toy. A wide range of people and industries can use it to conquer the mundane and free themselves to focus on more important work and innovation.
“This is a very big deal,” Mollick wrote. “The businesses that understand the significance of this change — and act on it first — will be at a considerable advantage.”
Mollick introduced his undergraduate entrepreneurship students to the chatbot during a class. Before the class was over, one student used it to create code for a startup protype using code libraries they hadn’t seen before.
“They completed a four-hour project in less than an hour,” Mollick wrote.
“The businesses that understand the significance of this change — and act on it first — will be at a considerable advantage.”— Ethan Mollick
Good Enough to Get a B Minus
Terwiesch’s white paper has garnered media attention with its intriguing title, “Would Chat GPT Get a Wharton MBA?” The answer is a solid “yes,” with the professor giving the chatbot a final grade of B to B-minus for its performance on a five-question experiment he designed. While the bot earned top marks on the first question about bottlenecks, it did not do as well on every question. Surprisingly, it performed the worst when prompted with a question that required simple math calculations.
“On some problems, the math was horrible,” Terwiesch said. “If you think about your computer, it might be stupid and dumb at many things, but at least it will get the math right. This was the opposite.”
His main takeaways from the experiment are:
- ChatGPT does an amazing job at basic operations management and process analysis questions, including those based on case studies. The answers are correct, and the explanations are excellent.
- ChatGPT at times makes mistakes in relatively simple calculations at sixth-grade level math. These mistakes can be massive in magnitude.
- The present version is not capable of handling more advanced process analysis questions, even when they are based on fairly standard templates.
- ChatGPT is remarkably good at modifying its answers in response to human hints. In instances where it initially failed to match the problem with the right solution, ChatGPT was able to correct itself after receiving an appropriate hint from a human. Even more remarkable, it seems to be able to learn over time so that in the future the hint is no longer needed.
Terwiesch also wanted to determine whether ChatGPT could come up with exam questions, a task he’s been doing for 25 years as a professor.
“After a quarter of a century on the job, I’ve exhausted all the barber shops and restaurants and all the settings you can imagine,” he said with a laugh. “I got desperate for some creative advice, so I did two experiments.”
The software spit out some interesting and well-worded questions that were even a bit humorous. But Terwiesch said they required substantial refinement before they could be useable on an exam. Still, it shows what’s possible with ChatGPT.
“I still have to do some work, but I feel that even in the current state of the world, it has the opportunity of boosting my productivity as well as the productivity of our students,” he said.
Mollick echoed that sentiment in his HBR opinion piece, writing that businesses could save time and resources using the technology to generate written communication for clients and stakeholders or build customized data sets, and educators could use it to generate a syllabus or lecture notes.
But Mollick warns that ChatGPT is not without problems and limitations — and it’s certainly not magic.
“For one, it is a consummate bullshitter, and I mean that in a technical sense,” he wrote. “Bullshit is convincing-sounding nonsense, devoid of truth, and AI is very good at creating it. You can ask it to describe how we know dinosaurs had a civilization, and it will happily make up a whole set of facts explaining, quite convincingly, exactly that. It is no replacement for Google. It literally does not know what it doesn’t know, because it is, in fact, not an entity at all, but rather a complex algorithm generating meaningful sentences.”
“We’re not going to get the toothpaste back into the tube. This technology is out there. It’s only going to grow.”— Christian Terwiesch
Back to the Classroom
Despite the promises of AI, the worries over cheating remain. Terwiesch said the concern got him thinking more deeply about why teachers test in the first place. He came up with three reasons: 1) to certify a skill; 2) to customize the learning journey for each student; and 3) to make the student engage with the material on their own and grow critical-thinking skills.
ChatGPT and similar software can help with all three, he said, although he shares concerns about cheating and has joined the growing number of educators banning its use in homework assignments and tests. He wants to make sure his students still have foundational knowledge in a subject.
Terwiesch encourages educators and business managers to explore ways to use the software to “raise the bar.” Instead of tweaking the old, students and workers can be challenged to come up with something entirely new.
“We’re not going to get the toothpaste back into the tube,” he said. “This technology is out there. It’s only going to grow. The idea that we could go back to the old world, it never works.”
