Instruction-native editing vs. pixel diffusion

Pixel diffusion treats the edit as a localized visual replacement problem. The model receives a mask or region specification and generates new pixel values to fill that region, guided by surrounding context and the edit instruction. This approach operates entirely at the visual surface level.

Instruction-native editing treats the edit as a scene-state transition problem. The model must understand the instruction as a transformation of the underlying scene state, then render the visual consequences of that transformation. This requires maintaining an internal model of physical properties, object relationships, and causal dependencies.

The second approach is substantially more demanding because the model must preserve hidden variables such as gravity, material response, object identity, and camera intent. When a user requests 'make the ball bounce higher,' the model must understand that this implies increased initial velocity, not merely stretching the visual trajectory.

This distinction becomes critical in multi-step editing scenarios. A pixel-diffusion approach may successfully change an object's appearance in one frame but fail to propagate that change consistently through subsequent frames where the object interacts with other scene elements.

Each reference clip receives a three-stage editing protocol: a base instruction establishing the initial scene, a modification instruction requesting a specific change, and a refinement instruction adding constraints or corrections. The final output is scored across three dimensions: instruction adherence (does the edit match the request?), continuity (does the edit preserve unchanged elements?), and unintended drift (does the edit introduce unexpected changes?).

The 14 reference clips span diverse scenarios: object manipulation (changing color, size, position), environmental modification (lighting, weather, time of day), action editing (speed, direction, outcome), and compositional changes (adding/removing elements, camera angle shifts).

The strongest examples preserve unchanged regions while propagating the requested edit through every dependent visual property. For instance, when changing a car's color from red to blue, the reflection of that car in nearby windows must also update to blue, and any colored lighting cast by the car onto surrounding surfaces must adjust accordingly.

Scoring employs both automated metrics (CLIP similarity for instruction adherence, optical flow consistency for temporal coherence) and human expert review for semantic correctness and physical plausibility.

The most common failure mode is shallow compliance: the requested object changes, but lighting, reflections, shadows, or secondary motion do not update consistently. The model performs a surface-level visual substitution without understanding the physical implications of the change.

A second failure mode is over-editing, where the model treats a narrow instruction as permission to restage the entire shot. A request to 'change the shirt color' might result in the character moving to a different location, the background changing, or other elements being added or removed without justification.

A third failure mode involves causal inconsistency: the edit creates a visual state that could not have arisen from the preceding frames. For example, a character might appear to teleport across the frame, or an object might change position without any visible motion connecting the before and after states.

These failure modes are particularly pronounced in complex scenes with multiple interacting objects, dynamic lighting, or camera motion. The model's ability to maintain physical and causal consistency degrades as scene complexity increases.

Across the 14 reference clips, instruction-native editing achieved 87% instruction adherence compared to 72% for pixel-diffusion baselines. More significantly, instruction-native editing maintained 94% physical consistency (lighting, shadows, reflections) versus 61% for pixel diffusion.

Temporal coherence scores showed similar gaps: instruction-native editing preserved object identity across 95% of frames, while pixel diffusion exhibited identity drift in 34% of frames. These results suggest that explicit reasoning about scene state provides substantial benefits for maintaining consistency across edits.

However, instruction-native editing showed slightly higher latency (average 2.3 seconds per edit versus 1.8 seconds for pixel diffusion), reflecting the additional computational cost of maintaining and updating the internal scene model.