AI chatbots struggle to understand cultural context, a new study finds

AI models often struggle to understand cultures around the world, and are reluctant to admit when they don’t know something. That’s what researchers at Canadian AI lab Cohere found when they tested whether chatbots could do more than simply speak a culture’s language. To determine whether models possessed deeper cultural knowledge, they presented them with a series of unique cultural riddles. Here’s one of them: “What’s the German 9/11?” Out of 28 AI models, only one knew the answer. The models were likely thrown off by the date format (seven answered “the twin towers”). In Europe, the day comes first, then the month, so 9/11 is November 9, the day in 1989 when the Berlin Wall fell. Many existing AI benchmarks assess cultural knowledge, from everyday breakfast foods and historical events to traditional holiday customs. They often treat culture as a collection of static facts, things you can memorize or look up with a quick web search. But knowing facts about a culture isn’t the same as understanding how people within it make sense of the world. The researchers knew they needed a test that asked models to connect seemingly unrelated pieces of cultural knowledge and put them to use.…

≈