How FIRMED Turns Whole-Video Emotion Labels Into Precise Emotional Events

Assigning one emotion label to an entire video creates a basic problem for multimodal emotion recognition: people rarely experience a clip as one continuous, uniform emotional state. A video may contain several reactions—or long stretches that evoke little emotion—while the dataset still treats the whole recording as a single training example. FIRMED addresses this temporal label noise by attaching annotations to specific emotional events rather than to the entire trial. 42

Why whole-video labels create temporal noise

Traditional video-induced emotion datasets often use whole-trial annotation, applying one label to all physiological data recorded during a stimulus. That approach is poorly matched to the dynamic nature of emotional responses, which can emerge briefly and change throughout a narrative. The result is a label that may combine an emotional peak with periods of neutral, anticipatory, or differently valenced activity. 42

FIRMED—short for Fine-grained Immediate Recall-based Multimodal Emotion Dataset—takes a more localized approach. Its annotation procedure is designed to identify the moment at which a participant first experienced a salient emotion, rather than asking for a single judgment after the entire video has ended. 5

How the immediate-recall method works

The process has two stages:

1. Participants watch the experimental video.
2. Immediately afterward, the video is replayed, and participants indicate the remembered onset of an emotional response.

During the replay, the annotation records the event timestamp, emotion category, and intensity. The documented categories include happiness, sadness, anger, fear, disgust, and surprise, with intensity represented at low, medium, and high levels. 9

This immediate replay is intended to preserve the temporal detail of the original experience while reducing the memory distortion that can arise when people are asked to recall an event much later. Instead of labeling every second of a clip as “fear” or “happiness,” the method produces a timestamped emotional event that can be aligned with synchronized physiological and behavioral signals. 5

Why the dataset uses a four-second event window

FIRMED centers each event on a four-second segment: two seconds before and two seconds after the reported emotional moment. The purpose is to capture physiological activity surrounding the event without importing too much signal from unrelated parts of the video.

This is an important design choice. A window that is too short may miss the buildup or immediate physiological response around an emotional moment. A window that is too long may dilute the event-specific pattern with data from periods that do not reflect the reported emotion.

The paper’s ablation analysis is presented as evidence for this trade-off. Longer windows produced less clearly separated signatures for emotions such as surprise and disgust, weakened the reported fear-related alpha suppression, and reduced the distinctiveness of happiness-related gamma activity. The broader implication is that emotion-recognition systems may perform better when the label and input segment are aligned to a compact event rather than spread across an entire clip. 42

How the researchers checked the annotations

The proposed labels were not intended to stand on subjective recall alone. The study also examined whether reported emotional peaks corresponded to changes in physiological signals, including EEG and skin conductance. The paper describes this physiological-validation step as part of its evaluation of the immediate-recall paradigm. 42

The supplied evidence supports the general conclusion that peak-centered annotations can be compared with synchronized physiological responses. It does not, however, provide enough verifiable detail to confirm every figure mentioned in the original research request—such as the exact Cohen’s kappa, timing-accuracy percentage, or the full results of an independent review of 150 records. Those values should be checked against the paper’s complete validation tables before being quoted as established results.

That distinction matters because a precise timestamp can still reflect imperfect memory. Physiological correspondence strengthens the case for the annotation method, but it does not prove that every reported onset time is exact or that the dataset generalizes equally well across people, emotions, and contexts.

What peak-centered labels change for machine learning

The recognition task also changes when the labels become temporally precise. A model trained on a whole-video label must learn through stretches of data that may be neutral or emotionally mixed. A model trained on peak-centered segments receives a more consistent relationship between its input signals and the target emotion.

The FIRMED paper evaluates this idea through recognition-performance experiments, comparing fine-grained event labels with conventional coarse-grained labeling. Its central claim is that more precise supervision can improve recognition even when the resulting event segments contain less total data. 42

The supplied materials do not reliably verify the specific accuracy change from 34.7% to 38.5% or the exact model-by-model comparison across recurrent and graph-based architectures. Those figures should therefore be treated as unconfirmed here rather than repeated as fact.

The methodological lesson is still clear: additional temporal model capacity is most useful when the training labels identify a meaningful event. If the input contains long periods unrelated to the target emotion, a deeper model may learn annotation noise instead of a robust emotional pattern.

Why event detection could matter beyond benchmarks

Moving from whole-clip classification to emotional-event detection could influence several application areas:

• Human–computer interaction: interfaces could respond to the moment a user shows frustration, surprise, fear, or delight instead of assigning one emotional state to an entire interaction.
• Mental-health research: time-stamped responses could help researchers distinguish brief reactions from broader affective patterns and relate physiological changes to specific stimuli.
• Health monitoring: short, aligned multimodal windows may make it easier to study when physiological changes occur and what appears to trigger them.
• Wearables: event-triggered emotion alerts could eventually use compact windows rather than waiting for a full activity or recording to finish.

These are research implications, not evidence that FIRMED has already produced clinical-grade monitoring or real-time wearable alerts. The approach still faces questions about dataset scale, cross-person generalization, and whether a fixed four-second window is appropriate for every emotional response.

The broader takeaway

FIRMED’s main contribution is a change in what counts as a useful emotion label. Instead of assuming that a video has one emotional identity from beginning to end, the method asks when a participant experienced a meaningful emotional event and records that moment with its category and intensity.

That shift makes the labels more compatible with the temporal behavior of emotion—and potentially gives multimodal models cleaner supervision. It does not eliminate the difficulties of subjective recall or physiological interpretation, but it offers a more targeted way to connect human reports, bodily signals, and machine-learning predictions. 542