Tutorial · Evidence checked 2026-08-30
How to Evaluate AI Video Consistency Without Cherry-Picking
A practical, evidence-led decision guide. Product capabilities and limits are separated from anything that would require hands-on testing.
AI video consistency is not “the face looked similar twice.” A usable production system must preserve the parts the brief declares immutable while allowing the requested parts to change.
For a character, that may mean face, age, wardrobe and body proportions. For a product, it may mean package geometry, cap colour, label hierarchy and logo placement. For a scene, it may mean lighting direction and spatial relationships. Write those invariants down before opening a generator.
Build a reference pack that can fail
Use only assets the team has permission to upload. Include:
- front, side and three-quarter reference views;
- a close-up of identity-critical details;
- a colour target;
- allowed and prohibited changes;
- intended aspect ratios and output destinations;
- consent/release identifier for every identifiable person.
Do not create an immaculate reference pack for one provider and improvise for another. Hash or version the pack so every system receives the same evidence.
Test three kinds of continuity separately
Within-shot continuity asks whether the product or person mutates during one clip. Watch hands, teeth, text, reflections, packaging edges and background objects frame by frame.
Across-shot continuity asks whether separate generations can belong to one sequence. Compare identity, wardrobe, scale, colour and environment between establishing, medium and close shots.
Revision continuity asks whether a targeted change leaves everything else alone. Request “slower camera move” or “blue background” and measure unintended changes to the subject.
A product can pass one and fail the others. Report them separately.
Use a six-shot benchmark
Create six prompts around one subject:
- static establishing view;
- controlled lateral camera movement;
- close-up of identity-critical detail;
- simple hand or object interaction;
- return to the original framing;
- one revision of shot 2 changing only motion speed.
Run three seeds or attempts per shot. That produces 18 outputs per platform—enough to reveal recurring failure without pretending to be a universal model ranking.
Lock duration, resolution and underlying model where possible. If one product cannot expose a setting, record “not controllable” rather than choosing a hidden default that favours it.
Score observable defects
Use a 0–2 scale for each required invariant:
- 2: preserved without repair;
- 1: usable after bounded repair;
- 0: rejected or requires regeneration.
| Dimension | Evidence to inspect |
|---|---|
| Identity | face landmarks, body proportions, wardrobe |
| Product fidelity | silhouette, label, colour, component count |
| Temporal stability | flicker, morphing, disappearing objects |
| Spatial continuity | subject position, light direction, room geometry |
| Camera obedience | requested move, speed and framing |
| Revision isolation | unintended differences after one requested change |
Define pass thresholds before seeing results. If the product logo is legally or commercially critical, its score may be a hard gate rather than one-sixth of an average.
Measure survival, not only average score
`shot survival rate = shots passing every hard gate / shots reviewed`
`sequence survival rate = complete six-shot sequences passing / sequences attempted`
`revision isolation rate = revisions changing only requested dimension / revisions attempted`
A high average can conceal a fatal failure. Five perfect dimensions and a mutated package still produce an unusable advertisement. Report the hard-gate survival rate beside any mean.
Also measure reviewer agreement. If two reviewers disagree on product fidelity, clarify the rubric before blaming the model.
Prevent cherry-picking
Pre-register the number of attempts. Preserve every output, including failures. Do not swap prompts mid-test without recording a new round. Separate default-prompt results from provider-specific prompt tuning.
Publish or retain a contact sheet in generation order. A portfolio of the best six outputs cannot reveal whether they came from six attempts or sixty.
The same rule applies to vendor examples: they demonstrate possibility, not expected yield under your inputs.
Include the workflow around the model
A platform may improve consistency through reference management, scene tools, camera presets or revision controls even when the underlying model is available elsewhere. Run one native-workflow round after the controlled model round. Record what the platform adds and what it obscures.
For a multi-model environment such as Krea or Freepik AI Video, save the underlying model and version. For a camera-led system such as Higgsfield, distinguish preset obedience from identity survival. For Runway or Kling, record every reference and model-specific mode.
The decision packet
Procurement should receive the reference-pack version, prompts, settings, all outputs, defect sheet, survival rates, credits consumed, reviewer time and unresolved rights/data questions. The verdict should name the content class tested. “Model A is more consistent” is too broad; “Model A preserved this package across a six-shot 9:16 sequence at a 67% sequence survival rate” is auditable.
Primary next step: Choose an AI video candidate and run the six-shot test before expanding the subscription.