Logic Score: a frame-level evaluation rubric
A practical, open-source rubric for scoring video outputs where the central question is whether the model follows causal, symbolic, and spatial logic across time. This methodology provides a structured framework for evaluating reasoning capabilities in video generation systems beyond traditional visual quality metrics.
Six frame-level dimensions scored 0–10, aggregated into a composite Logic Score. Human review remains required for causal and semantic dimensions.
Instruction adherence
Temporal coherence
Object permanence
Spatial anchoring
Causal consistency
Occlusion recovery
2–4 second clips at native frame rate
78% agreement within ±1 point
Always report evaluator version with scores
Scoring Dimensions
Logic Score combines six distinct evaluation dimensions, each scored on a 0-10 scale and aggregated into a composite score. The dimensions are: instruction adherence (does the output match the prompt?), temporal coherence (do objects and properties persist consistently?), object permanence (do objects maintain identity across occlusion and reappearance?), spatial anchoring (do objects maintain consistent world coordinates?), causal consistency (do events follow logical cause-effect relationships?), and recoverability after occlusion (can the model restore occluded content accurately?).
A high score requires more than a good-looking frame. The sequence must preserve the reason why each frame follows from the previous one. Visual appeal without logical consistency receives low scores; logical consistency without visual quality also receives low scores. Both dimensions must be satisfied.
Each dimension includes specific failure criteria that deduct points. For example, temporal coherence loses points for: symbol morphing (2 points), position drift exceeding 5% of frame width (3 points), material property changes without justification (4 points), and complete object identity loss (6 points).
The rubric is designed to be applied by human evaluators with domain expertise in the relevant subject matter. Automated scoring is possible for some dimensions (temporal coherence via optical flow, instruction adherence via CLIP similarity) but human judgment remains essential for causal consistency and semantic correctness.
Frame-Level Review Protocol
Reviewers inspect short windows (typically 2-4 seconds) at native frame rate, marking discontinuities that change meaning, position, material behavior, or causal order. The protocol requires pausing at each frame and comparing against the immediately preceding frame, creating a chain of pairwise validations.
Minor texture noise can be acceptable if it does not break the symbolic or physical interpretation of the scene. A small amount of visual variation is expected and even desirable for photorealism; the key distinction is whether that variation preserves or violates the underlying logic.
Reviewers document specific failure events with frame timestamps, enabling precise diagnosis of model weaknesses. A typical review session produces a timeline of scored frames with annotations explaining each deduction.
Inter-rater reliability is assessed through overlap scoring: when two reviewers evaluate the same clip, what percentage of their frame-level scores agree within 1 point? Current inter-rater reliability on Logic Score dimensions averages 78%, with highest agreement on temporal coherence (89%) and lowest on causal consistency (71%).
Rubric Versioning and Evolution
The rubric should be versioned because model capabilities change quickly. A score from one generation of models should not be compared to scores from another generation without recording the evaluator version. We maintain a public changelog documenting rubric modifications and their rationale.
The current draft (v0.3) is optimized for research notes and public case studies rather than formal academic benchmarking. Future versions will include: expanded dimension definitions, automated scoring tools, calibration datasets for training evaluators, and statistical methods for aggregating scores across multiple reviewers.
We invite community contributions to the rubric evolution. Researchers using Logic Score in their work are encouraged to report edge cases, suggest new dimensions, and share calibration data that can improve inter-rater reliability.
Application Guidelines
Logic Score is most informative when applied to clips that explicitly test reasoning capabilities: multi-step processes, causal chains, symbolic manipulation, and instruction following. It is less informative for purely aesthetic or atmospheric video where logical consistency is not the primary concern.
When reporting Logic Scores, include the version number, the number of reviewers, and the specific dimensions that contributed most to the score. A composite score alone provides limited information; the dimension breakdown reveals the model's specific strengths and weaknesses.