What happens when people have to jump in to clean up AI’s mess?

Imagine an artificial intelligence system that has performed reliably for months. Its recommendations are almost always right, and the employee reviewing them has learned that close scrutiny rarely changes the outcome. Then something unusual happens. The AI makes an error or can’t proceed and needs human intervention. The employee is expected to step in and remains responsible for the outcome. But does the employee know how to do the task? I’m a professor of management information systems and the academic director of Drexel University’s Center for Applied AI & Business Analytics. In my research on human-AI relationships, I call this latent risk: the possibility that people may lose capabilities their organizations still depend on them to exercise. AI can continue to perform well while the people overseeing it become less practiced at exercising independent judgment and less prepared to intervene. The risk becomes latent when the organization still depends on that human capability to step in but has little cause to notice that that capability may be weakening. The concern is not entirely new. The idea of latent risk builds on decades of research into how people respond to automatio…

≈