New technology allows ai agents to read and respond to people’s facial expressions

Technology now allows the creation of increasingly realistic AI agents. Human-like AI agents, such as the digital characters that appear as virtual assistants, game characters, and the increasingly lifelike “metahumans” used in film and interactive media, can look remarkably realistic, but their facial behavior during conversation remains stiff and generic. A doctoral thesis defended at Tallinn University addressed this gap by building a system that enables an AI agent to both interpret and generate facial expressions.

How does it work?

Abdallah Hussein Sham‘s thesis developed the Enactive Facial Expression Pipeline (EFEP), a modular camera-based system built on the idea that meaningful interaction emerges from the ongoing exchange between a person and an AI agent, rather than from reading a single snapshot of a face.

Using only a standard webcam, the system reads the person’s expression, estimates the reaction it would naturally elicit, and synthesizes the agent’s facial response accordingly.

The research combined existing facial expression datasets with a new dataset collected from 60 participants across five social scenarios. A key finding was that working with individual facial muscle movements, rather than broad labels such as “happy” or “sad,” gave the agent finer, more stable control over its responses.

The research also placed strong focus on fairness, finding a significant accuracy gap when models were trained on data from only one demographic group; a gap that disappeared with diverse training data.

Real-world impact

This matters because AI agents are increasingly used in low-risk settings, such as interactive media, creative applications, and user experience research, where the quality of the interaction depends on nonverbal communication. A human-like agent that responds with an appropriate expression, rather than a scripted animation, could meaningfully improve how people experience these technologies.

As a next step the technology could be expanded to include voice and body language, making interactions between humans and AI agents feel even more natural.

Ethics are important

In line with the EU AI Act, the system was intentionally limited to low-risk areas like interactive media and user experience research.

This article was sent to us by Tallinn University.

Teaching AI to read the room is one challenge; helping it understand the learner is another: Tallinn University researchers developing student-friendly AI algorithm.