PsycheMME: A Psychology-Informed Benchmarkfor Social Understanding and Interaction in Videos

Anonymous ACL submission · project page

We introduce PsycheMME, a psychology-informed video benchmark for diagnosing human understanding and interaction judgments. Its 1,000 questions from 630 episodes span eight capabilities, separating state and need inference from decisions about when and how to respond. Independent answering and expert review ground the questions in observed cues; Reasoning-Verified evaluation assesses the support and logical consistency of explicit rationales alongside answer credit.

Timing has the lowest Verified scores across the 17 models; similar Need scores can conceal differences in answer completeness.

Code Dataset Models Evaluation Logs

01 / Overview

Diagnosing human understanding and interaction

Social Information Processing and the Perception-Action Cycle guide a shared taxonomy: infer states and needs from observable cues, then judge an appropriate response. Relevant history and social boundaries can change that judgment.

Figure 1. Overview of PsycheMME Enlarge
Figure 1. Diverse human-centric social situations; eight capabilities from social cues to interaction; and overall results for 17 models. Vector PDF ↗

Human understanding

Associate social cues with the right person, interpret their psychological state, and identify the support they need and are willing to receive.

Interaction decisions

Judge whether and when to respond, what to say, and which physical or nonverbal action fits the situation.

Rationale verification

Assess the support and consistency of explicit rationales, alongside answer correctness, relevant history, and social boundaries.

A closer look at the benchmark Composition & statistics

630 episodes across eight source groups support 890 choice questions and 110 open questions. The overview illustrates 16 scene categories. Uneven capability counts motivate macro averaging across the eight capabilities.

Figure 4. Benchmark statistics Enlarge
Figure 4. Capability taxonomy, dataset composition, video duration, question length, and frequent keywords. Vector PDF ↗

02 / Results

Performance across eight diagnostic capabilities

17 frozen models: 8 proprietary and 9 open-weight. All are evaluated on the same 1,000 questions under a shared input policy and scoring rubric. Two adapted checkpoints are evaluated separately.

Raw and Reasoning-Verified macro scores for 8 proprietary and 9 open-weight frozen models, plus the separate Qwen3.5-4B adaptation comparisonEnlarge ↗
Main results. Raw Answer and Reasoning-Verified macro scores for (a) eight proprietary and (b) nine open-weight frozen models. (c) Qwen3.5-4B Base, SFT, and SFT+GRPO; the two adapted checkpoints are separate from the 17-model frozen comparison. All values are from Table 2. Vector SVG ↗

Selected model comparisons

Table 2 · overall macro scores (%)
ModelRawVerified
Gemini 3.6 Flash85.6880.03
Claude Opus 582.3476.71
GPT-5.6 Sol81.0975.98
Qwen3.8-Max81.4875.59
Qwen3.8-27B79.9772.74
Qwen3.5-9B76.5466.45

Gemini 3.6 Flash leads the proprietary group; Qwen3.8-27B leads the open-weight group. All 17 frozen models and both adapted checkpoints are available in the complete table below.

The Raw–Verified gap reflects reduced weighted credit under the fixed rationale rubric, including its normalization.

Figure 5. Performance breakdown Enlarge
Figure 5. Capability profiles, verification stages, credit distributions, and loss attribution. Vector PDF ↗
Explore the complete model comparison Table 2 · all capabilities
17 frozen models. Overall scores are macro means; capability columns are Reasoning-Verified scores (%).
ModelOverallHuman understandingInteraction decisionContext & constraints
RawVerifiedGWNTSAMR
Gemini 3.6 Flash85.6880.0391.7679.6869.9747.8179.3289.5890.9291.21
Claude Opus 582.3476.7189.3177.8069.1040.7079.1886.2486.5684.82
GPT-5.6 Sol81.0975.9892.1378.7567.2832.9974.8684.2591.1186.47
Qwen3.8-Max81.4875.5990.8777.6169.1739.3372.8482.4684.4887.94
Kimi K380.6075.3591.5276.1160.6238.8279.3884.2885.1786.87
Gemini 3.1 Pro81.3674.3391.3076.2660.2944.7168.2276.8090.4986.57
GLM-5V-Turbo78.4071.1989.1876.7261.8234.2368.1479.8776.2383.37
Seed 2.1 Pro76.3769.6788.5376.8753.2432.9363.2778.0982.4281.98
Qwen3.8-27B79.9772.7490.9380.5366.9628.5868.8783.0177.7785.27
Qwen3.5-9B76.5466.4590.2476.8654.6629.1362.6373.4568.1476.52
Gemma-4-12B-it66.4157.7579.4871.1946.4113.4949.4362.3265.0774.58
InternVL3.5-14B52.2444.6863.3656.6319.9015.6334.0545.1059.5363.22
MiniCPM-V4.569.6558.1381.2670.6333.8124.3250.0862.4565.4077.13
InternVL3.5-8B67.4155.4172.3064.3935.3820.6550.3759.3272.2068.67
Phi-4-multimodal61.0348.6352.5050.3222.1420.8545.9658.1764.6374.44
InternVL3.5-4B61.3048.9772.4660.0626.9021.6137.4447.0063.7462.54
Qwen3.5-4B47.6837.7160.7046.4934.1823.1724.0235.8533.6143.66

G Grounding · W Psychological Understanding · N Need · T Timing · S Verbal Response · A Action · M Memory · R Risk. The two adapted checkpoints are excluded from the 17-model frozen comparison.

Finding 01 / Timing

Interaction Timing remains challenging.

Timing has the lowest Verified score and credit retention for all 17 frozen models. Among correct single-choice responses, weak or contradicted cue support occurs in 49.74% of Timing responses, versus 7.62% for the other capabilities.

Finding 02 / Completeness

Similar Need scores conceal completeness differences.

Gemini 3.6 Flash and Claude Opus 5 have similar Need Verified scores (69.97 / 69.10). On the same 35 open Need questions, their full-credit rates are 28.57% / 68.57%, revealing a difference in answer completeness.

Finding 03 / Consensus

Voting can override correct minority answers.

For the eight proprietary models on 72 Timing questions, oracle answer coverage reaches 91.67%, while plurality voting reaches 57.64% and the best individual model reaches 66.67%. Shared wrong choices can outvote correct minority answers.

Finding 04 / Context

Performance depends on relevant context.

Full inputs improve Raw scores over Question-only by 12.50 points for Qwen3.5-9B and 12.43 for MiniCPM-V4.5. In the Memory intervention, gains depend on matched, relevant history.

Input ablations

Table 3 · same 1,000 questions · macro scores (%)

InputQwen3.5-9BMiniCPM-V4.5
RawVer.RawVer.
Question-only64.0457.0557.2249.93
ASR-only67.0457.7560.0549.36
Visual-only73.4465.5065.6755.56
Full76.5466.4569.6558.13
Single-image72.2861.7866.1654.99

Visual-only retains most Full accuracy. ASR adds another 3.10 / 3.98 Raw points. Single-image uses one midpoint frame with the remaining inputs fixed.

Memory intervention

Table 4 · Qwen3.5-9B · 40 Memory questions / 24 episodes

Raw AnswerReasoning-Verified
No History 42.5025.28
Mismatched History 45.0023.87
Matched History 82.5074.55
Micro scores (%). Mismatched History averages three seeds. Matched History improves Raw by 40.00 points and Verified by 49.27 points over No History.
Task adaptation & human reference Tables 2 & 5
Qwen3.5-4B adaptation · same 1,000 questions · macro scores (%)
CheckpointRawVerified
Base47.6837.71
SFT72.5852.13
SFT+GRPO73.1952.62

Human baseline. Five annotators independently answer 300 sampled questions (1,500 responses): 85.27% Raw micro accuracy and 84.82% Raw macro accuracy. This is a sampled subset.

Judge review. Five psychology experts endorse 1,352 of 1,500 presented Judge decisions (90.13%) across 300 sampled cases. This measures endorsement of the presented decisions.

03 / Methodology

How PsycheMME is built and evaluated

Social Information Processing and the Perception-Action Cycle motivate a shared taxonomy. Independent answering and expert review ground the questions; a fixed rubric evaluates answers and explicit rationales.

A /

Capability taxonomy

The taxonomy distinguishes observable cues, interpretations of states and needs, and response selection. Relevant history and social constraints can shape each judgment.

Figure 2. Illustrative scenarios and capabilities Enlarge
Figure 2. Human understanding, interaction decisions, memory, and social constraints in an illustrative egocentric scenario adapted from CASTLE. Vector PDF ↗

Human understanding

GGrounding 186 questions
WPsychological Understanding 320
NNeed 97

Interaction decision

TTiming 72 questions
SVerbal Response 97
AAction 94

Context & constraints

MMemory 61 questions
RRisk 73
B /

Benchmark construction

Video curation combines six public datasets with public vlogs, interviews, and daily-life recordings. Human screening assisted by VLMs selects episodes with observable social cues and adequate visual quality.

Figure 3. Benchmark construction framework Enlarge
Figure 3. Video collection, multi-agent QA generation with independent answers, and human quality review. Vector PDF ↗

01 / Question Agent

Propose a grounded question.

Generate a question, candidate answer, options where applicable, and supporting cues, guided by the target capability.

02 / Answer Agent

Answer independently.

Re-examine the video using the question, options, and public context. The proposed answer, private cues, and rationale are withheld.

03 / Critic Agent

Check both reasoning paths.

Review answerability, cue support, and logical coherence. Unresolved candidates are discarded after at most two refinement rounds.

Five independent reviewers.Researchers with psychology backgrounds check validity, correctness, and psychological consistency. Only unanimously approved items enter the frozen benchmark.

C /

Reasoning-Verified evaluation.

Raw Answer

Choice answers must exactly match the reference option or complete option set. A fixed semantic Judge assigns open answers zero, partial (0.5), or full credit.

Reasoning-Verified

Weight answer credit by the presence of an informative rationale, support from annotated reference cues, and logical integrity. Qwen3.8-27B serves as the fixed Judge; private references enter scoring only.

Answer correctnessInformative rationaleEvidence SupportLogical Integrity

Both headline metrics are macro-averaged across the eight capabilities. Evaluation concerns offline judgments in recorded interactions.

Scoring example

How rationale labels change the score

Change the answer credit and Judge labels to see how they enter the scoring rule. This illustration assumes a valid response with an informative rationale; it does not run a new evaluation.

Raw credit100.00%
Verified credit100.00%

1 × 1.2 × 1.2 ÷ 1.44 = 1.0000

Verified = answer credit × rationale coverage × evidence support × logical consistency ÷ 1.44. This illustration fixes coverage at present (×1). Invalid final-format responses receive zero answer credit. Full scoring contract ↗

04 / Case Study

Illustrative model responses

The paper’s four examples compare model responses with supplied context and frozen references. The Gemini example is especially diagnostic: the answer matches, but the cited conversation falls outside the queried time window.

Figure 6. Illustrative model responses Enlarge
Figure 6. Preference mismatch, temporal cue mismatch, interaction-timing disagreement, and goal substitution. Cases and outputs are reproduced from the paper. They illustrate specific discrepancies, not their prevalence across the benchmark. Vector PDF ↗

Claude Opus 5 / Memory

Preference mismatch

The supplied history specifies a preference for less personal follow-up. The response asks for more details, conflicting with that context.

Gemini 3.1 Pro / Response

Temporal mismatch

The selected response strategy is correct, but its rationale cites a later conversation window rather than the queried interval.

GPT-5.6 Sol / Timing

Timing disagreement

The prediction anchors waiting to a later speech boundary than the frozen reference, illustrating a disagreement about the annotated timing interval.

Qwen3.8-Max / Need

Goal substitution

A fulfilled inquiry about why redwoods were cut is interpreted as an unmet request by substituting a different goal.

Released task examples

Browse the released questions

These text examples illustrate the eight capabilities. References are shown separately from model inputs; evaluation also requires the original video and ASR.

Loading public text examples… Read the source files ↗

EasyCom-derived text · CC BY-NC 4.0 · Attribution and selection notes · License. Curated for clear cue and context links; original records and review metadata are preserved. Text inspection does not certify video-level correctness.

05 / Release

Code, models, data and evaluation logs

Selected code, a text benchmark subset, paper model adapters, and evaluation logs are available through the anonymous project account.

CodeInstallable inference, Judge and scoring tools; frozen score-factor replay.Open resource ↗ Trained ModelsExact SFT458 and GRPO-Step32 LoRA adapters, with pinned base weights and paper configurations.Open resource ↗ Dataset78 text questions with references and transcripts; source locators for 630 video episodes.Open resource ↗ Evaluation LogsAll 19,000 score-factor records and scores; 18 detailed prediction/Judge records.Open resource ↗

The release includes selected core code, 78 text questions, source locators for all 630 episodes, two paper LoRA adapters, and complete main-table numeric results. Videos remain at their original sources.

Start here

Understand it. Run it. Build on it.

Start with an offline example, check the archived results, or connect your model. Each path explains what you need and what you should see.

Python 3.10+ · no GPU or API key

Your first checked result

Eight synthetic items show how answers, rationale weights and failures become scores. Separate files make each step inspectable.

Expected: Raw 75.00% · Verified 71.18%
Software demonstration; no real model or Judge calls.

Step-by-step guide ↗
git clone https://github.com/psychemme/PsycheMME.git
cd PsycheMME
python3 -m venv .venv
source .venv/bin/activate
python -m pip install .
psychemme doctor
psychemme demo --output runs/demo
After installation · public download

Check the archived paper table

Download a fixed, hash-verified data snapshot and recompute scores from persisted answer credit and Judge labels. No videos, weights or API key required.

Expected: PASS · 19,000 rows · 190 table values
Formula replay; no independent re-grading of all answers.

What reproduction establishes ↗
psychemme download --bundle replay \
  --output data/PsycheMME
psychemme doctor --data-root data/PsycheMME
psychemme replay --root data/PsycheMME \
  --output runs/paper_replay.json
Authorized media · model backend · separate Judge

Bring your own model

Use the local Qwen runner, a compatible image-chat API, or import responses from your existing system. The scorer keeps missing and failed records in the declared population.

Follow the complete model recipe ↗
  1. Prepare public inputsGet the 78-question review subset and legally obtained, hash-matching video clips. Full clip reconstruction is not verified for every source.
  2. Generate predictionsUse the final public prompt and nine-key response format. Gold references never enter the tested model.
  3. Judge, then scoreReferences go to the separate text Judge. Report the exact subset, configuration and failures.

Current public scope: 78 text questions, metadata for 1,000 questions, and source locators for 630 episodes. All videos remain at original sources. Release scope ↗ · Get help ↗

Vector PDF ↗