Secure human oversight of AI: threat modeling in a socio-technical context
Publikationstyp
Conference Paper
Date Issued
2026-10
Sprache
English
Author(s)
Lazar, Veronika
Lichtmeß, Elmar
Plesch, Carola
Heck, Matthias
Start Page
843
End Page
856
Citation
Ninth AAAI/ACM conference on AI, ethics, and society, AIES 2026
Contribution to Conference
Publisher DOI
Publisher
AAAI Press
ISBN of container
978-1-57735-912-8
Human oversight of AI is promoted as a safeguard against risks such as inaccurate outputs, system malfunctions, or violations of fundamental rights, and is mandated in regulation like the European AI Act. Yet debates on human oversight have largely focused on its effectiveness, while overlooking a critical dimension: the security of human oversight. We argue that human oversight creates a new attack surface within the safety, security, and accountability architecture of AI operations. Drawing on cybersecurity perspectives, we model human oversight as an IT application for the purpose of systematic threat modeling of the human oversight process. Threat modeling allows us to identify security risks within human oversight and points towards possible mitigation strategies. Our contributions are: (1) introducing a security perspective on human oversight, (2) offering researchers and practitioners guidance on how to approach their human oversight applications from a security point of view, and (3) providing a systematic overview of attack vectors and hardening strategies to enable secure human oversight of AI.
DDC Class
006.3: Artificial Intelligence
005.8: Computer Security
340: Law