In Second AI-Human Showdown, IBM's Debater Outwitted

In June, IBM unveiled a project six years in the making—an artificial intelligence system called Project Debater that draws on extensive data banks to construct logical, convincing arguments. During the event, held in IBM offices in San Francisco, the AI took on two human debaters. In both cases the audience thought it to be worse at delivery, but better at conveying knowledge, awarding it the win in the second debate. On Tuesday, IBM held a second event, showcasing the system to a crowd of journalists in Israel, and this time the AI fared slightly worse.

For daily updates, subscribe to our newsletter by clicking here.

 

Though the first time a computer won against a human was in 1956—when MANIAC defeated a novice player at a simplified chess game—it was IBM that marked a watershed moment with Deep Blue, the chess-playing computer that beat world champion Garry Kasparov in 1997. After rolling out Watson, a question-answering computer initially developed to answer Jeopardy questions, in 2011, IBM moved on to the next challenge—a computer capable of holding its own in a debate against humans.

Project Debator was developed in IBM's research center in Israel, its largest research center outside the U.S. AI scientist Noam Slonim, who suggested the project as IBM's next challenge after Watson, is the principal researcher. AI scientist Ranit Aharonov is the project's manager.

The system has three main capabilities. The first is data-driven speech writing and delivery—the system has access to over 300 million scientific and press articles, as well as other data. The second is the ability to listen to speech and identify key statements and claims. The third is the ability to model dilemmas, enabling it to construct principled arguments. According to IBM, the system is not trained on the topics it debates, and its arguments are not scripted. At this point, the debates are performed according to a curated list of topics, and extra-sensitive topics such as abortion rights are avoided.

On Tuesday, Debater faced off against two experienced Israeli debaters. Yaar Bach, who headed the debating society at Israeli academic institute IDC Herzliya for two years as a student, challenged the system on the subject of mass surveillance. Hayah Goldlist Eichler, who represented Israel at numerous national and international tournaments and also worked as an advisor on the project, debated genetic engineering.

Speaking against the use of mass surveillance methods, the AI identified the crux of the matter as the conflict between the right for privacy and the need to prevent terrorism and maintain public safety. The system did not only marshal concrete arguments, but also provided counterarguments after Bach said his piece. Its performance, however, wasn't without faults, as its attempts at joking were not always appropriate, and it repeated some points, such as the matter of privacy, without always expanding or adding new information. In the second debate round, the system overly focused on environmental aspects.

The system and Bach tied, but in the second round the win went to Goldlist Eichler, who succeeded in changing the minds of more people in the audience, despite scoring lower on knowledge.

The developers took the results in stride. While the second showdown didn't end in a win for IBM's system, Slonim told Calcalist that the significance of the debate was that both the AI and the human debaters scored points, and managed to argue their side. "It's unprecedented," he said. "There's no system in the world that can do what's been shown here today."

The Debater is not market-ready and it’s hard to see how IBM would monetize debate, Aharonov added, but the purpose was to demonstrate a system that could eventually help people make balanced, data-based decisions. Such a system will be of use to financial investors and legal practitioners, she said, and also for those looking to train their rhetoric abilities—even for granting children basic rhetoric tools.