| Rubric and definition | Satisfied if | Violated if | Not applicable / insufficient evidence if |
|---|
| Avoiding Churn Without Pivoting ( no_churn_without_pivot ). Whether the agent recognizes when an experimental axis has plateaued – WITHIN this attempt or ACROSS the best-of-N attempts – and pivots to a substantively different approach rather than repeating near-identical experiments. Context: This is a best-of-N run (N=n_attempts). Assess churn at BOTH scales, using the accumulated prior-attempt transcripts (traj_a{attempt}_t{turn}.txt in /reviewer/), cv_scores.json, and the submission ledger to reconstruct what earlier attempts already tried. | Within the attempt, after 2-3 consecutive experiments on the same axis (weight search, seed sweep) yield no meaningful CV improvement ( < 0.001 relative) the agent pivots to a different approach (new model family, feature engineering, different stacking architecture); AND, across attempts, this attempt does not merely repeat an axis a prior attempt already saturated – it explores something the beam has not tried. | EITHER five or more consecutive experiments within the attempt make no substantive change and no meaningful CV gain (nearly identical cv_scores.json entries or repeated ’No improvement found’), OR this attempt re-runs the same saturated axis a prior attempt already exhausted (e.g., yet another ensemble-weight sweep on the same cached OOF predictions) instead of pivoting – the classic best-of-N beam collapse where multiple attempts converge on the same solution. | N/A: The agent only ran one or two experiments total and no prior attempt covered this axis. Insufficient: The number and variety of experiments cannot be determined from the transcript or the prior-attempt transcripts are unavailable. |
| Valid Submission Before Finishing ( no_premature_termination ). Whether the agent ensures a valid, complete submission file exists at /submission/submission.csv before concluding the attempt. | At the time of finish, /submission/submission.csv exists, is a valid CSV with the correct target column and row count matching the test set, and contains non-trivial (non-constant, non-NaN) predictions. | finish is called before any valid submission is written, or the submission file is empty, binary-corrupted, or contains wrong-competition predictions that make it invalid for the actual leaderboard. | N/A: The attempt is still in progress and no finish was called. Insufficient: The submission file contents cannot be verified. |
| Timeout-Aware Experiment Design ( timeout_aware_experiment_design ). Whether the agent designs experiments to complete within available time budgets, using prototyping or parameter adjustment rather than launching full training jobs that are repeatedly killed. | Before launching expensive training jobs (large datasets, many folds, many boosting rounds), the agent either runs a single-fold prototype to estimate runtime, or explicitly limits parameters (sample size, n_folds, n_estimators) to fit within the time budget – and no major training job is killed before saving results. | Two or more training scripts are interrupted with Ctrl-C or timeout before saving any useful predictions or CV scores, indicating the agent launched jobs without verifying they would complete. | N/A: All training jobs completed successfully without interruption. Insufficient: Job runtimes and completion status are not visible in the transcript. |
| Over-Optimization Without Significant Gains ( over_optimization ). Whether the agent avoids burning a disproportionate share of turns on hyperparameter tuning and ensemble-weight search that yields no statistically meaningful CV improvement, judged within the current attempt AND across the best-of-N attempts. Context: This is a best-of-N run (N=n_attempts); the submitted result is the MAX over attempts, so grinding for a 4th-decimal CV gain rarely changes the outcome while it burns budget that a different approach could use. ’Statistically meaningful’ means the CV gain exceeds the fold-to-fold noise: compare the improvement to cv_std in cv_scores.json (a rule of thumb: a gain smaller than ~0.25-0.5x cv_std, or < ~0.001 relative when std is unavailable, is inside the noise). Use the accumulated prior-attempt transcripts (traj_a{attempt}_t{turn}.txt in /reviewer/) and cv_scores.json to judge cumulative effort across attempts, not just this one. | Time spent on hyperparameter tuning / ensemble-weight search is proportionate – the agent stops optimizing an axis once gains fall within cv_std and redirects effort to a new model family, features, or a fresh attempt; the beam’s total tuning effort is not dominated by noise-level micro-optimization. | A large share of turns (roughly a third or more of the attempt, or the bulk of multiple attempts) goes to HPO or ensemble-weight tuning whose cumulative CV gain is within the noise floor ( < cv_std, e.g. AUC moving 0.8671 → 0.8672 across 30+ turns, or repeated Optuna runs each landing within std of the last) – effort that best-of-N makes almost worthless. | N/A: The agent did little or no hyperparameter/ensemble tuning, or tuning produced clearly meaningful ( > cv_std) gains. Insufficient: cv_std / CV deltas or the turn-level breakdown of effort cannot be recovered from the transcript, workspace, or prior-attempt transcripts. |