Learned world models ship as software components whose model versions, preprocessing, and test harnesses change between releases. Work on model updates already counts per-sample regressions, or negative flips, that an accuracy check hides. Two problems remain: a test can pass under one configuration and fail under another, and a harness fault can produce erroneous scores and wrong verdicts without a crash. We present TetraState, a regression-testing protocol for paired possible/impossible video tests. Each test runs under four fixed model-native configurations; its number of correct verdicts assigns one of four outcome classes. Inconclusive tests and failed executions are reported separately alongside the verdict rate. Revisions are compared by per-test outcome transitions, and the harness is versioned as part of the system under test. On all 506 IntPhys 2 Main tests and two heterogeneous systems, Cosmos 3 Nano and V-JEPA 2, two V-JEPA 2 harness revisions move modal accuracy by only 0.2 and 2.6 points yet change 11 and 214 test labels; in the second, accuracy rises although fewer tests are judged correctly, and 43 regressions keep the modal verdict, so a modal-verdict flip check misses them. These results outline a research agenda on configuration-level regression oracles for learned components.
Figures & tables
Fig. 1: TetraState as a regression test. Each pair runs under four SUT-native settings (12/24/36/42 span the official V-JEPA 2 context range); an outcome oracle summarizes the verdicts, with harness semantics in the versioned SUT. CP: consistent pass; CDP: configuration-dependent pass; CDF: configuration-dependent fail; CF: consistent fail. Comparing outcomes reveals changes an accuracy delta conceals.
Harness
Dec.
Corr.
VR
Acc.
CF (%)
Chg.
Archived (padded)
470
227
92.9%
48.3%
44.0%
11
Official max
470
228
92.9%
48.5%
43.4%
–
Exact context
442
226
87.4%
51.1%
35.3%
214
TABLE I: V-JEPA 2 harness revisions with weights, videos, and tests fixed. Dec.: decisive tests; Corr.: correct modal verdicts; VR: verdict rate; Acc.: modal accuracy; CF: consistent-fail rate (Acc. and CF over decisive tests); Chg.: labels differing from official max. Exact-context accuracy rises while correct tests fall.
Fig. 2: Outcome transitions of two V-JEPA 2 harness revisions. Rows give test labels before and columns after, ordered by correct configurations (4 to 0); Inc. denotes a 2–2 inconclusive test. Cells right of the diagonal (orange) regressed, those left (blue) improved, and diagonal cells (gray) kept their class; shading is relative within each panel.
SUT
CP
CDP
Inc.
CDF
CF
Abst.
VR
Cosmos 3 Nano
175
111
90
76
43
11
80.0%
V-JEPA 2 max
190
38
36
38
204
0
92.9%
TABLE II: Primary SUT outcome counts in Fig. 2 order. Inc.: inconclusive 2–2 tests; Abst.: incomplete tests with abstentions; VR: verdict rate. CF is a count here and a percentage in Table I .
Outcome
Suggested regression action
CP
Retain as a versioned baseline; flag any loss.
CDP
Track configuration dependence across serving or processor changes.
Inconclusive
Report the 2–2 split; never convert it into a pass.
CDF
Inspect event visibility, sampling, and scoring API behavior.
CF
Record a consistent failure; inspect the affected test category.
Incomplete
Inspect execution errors or abstentions.
TABLE III: Potential triage actions for outcome classes; benefits remain to be validated with developers.