BuySideFlow: A Point-in-Time Benchmark for LLM Agents in Investment Research
Abstract
Large language models (LLMs) can generate SQL and SQL--Python workflows for investment research, but ensuring point-in-time (PIT) validity remains challenging. Agents must enforce disclosure cutoffs and select the appropriate historical states and record versions throughout data retrieval and financial analysis. We introduce BuySideFlow, a benchmark of 408 practitioner-derived Chinese-language tasks spanning equities, funds, fixed income, and macroeconomics over a fixed snapshot from a licensed production financial database. We provide executable references and a taxonomy of temporal errors, pairing result checks with expert audits of executed workflows. We evaluate nine agent configurations on all tasks, with result accuracy ranging from 16.2% to 36.8%. An audit of 900 runs on 100 shared tasks confirms PIT violations in 193 of the 380 runs that pass result checks (50.8%), showing that output agreement alone does not establish temporal validity.
Figures & tables
| Requirement | Audit focus | Tasks, (%) |
|---|---|---|
| Disclosure cutoff | Task-defined availability dates for selection or calculation inputs, checked against the cutoff | 311 (76.2%) |
| Historical state | Listing, membership, and classification histories defining the universe at the research date | 272 (66.7%) |
| Version selection | Version keys and operation order (filtering, ranking, deduplication) for selecting applicable records | 233 (57.1%) |
| Decision sequence | State reconstruction or grouping at successive dates, checking inputs against each date’s information cutoff | 71 (17.4%) |
| Ex-post labels | Later outcomes for evaluation, without influencing earlier selection, grouping, or features | 58 (14.2%) |
| Asset domain | Reference workflow | |||||||
| Agent | Model | Overall | Equity | Fund | Fixed income | Macro | SQL-native | SQL–Python |
| (408) | (166) | (125) | (93) | (24) | (254) | (154) | ||
| SWE-agent | GPT-5.5 | 35.0 | 34.9 | 32.0 | 29.0 | 75.0 | 38.2 | 29.9 |
| DeepSeek-V4-Flash | 30.9 | 25.3 | 30.4 | 29.0 | 79.2 | 33.1 | 27.3 | |
| Claude-Opus-4.7 | 29.9 | 28.3 | 23.2 | 31.2 | 70.8 | 32.7 | 25.3 | |
| GLM-5.1 | 19.6 | 16.9 | 12.0 | 23.7 | 62.5 | 24.4 | 11.7 | |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Case and data dependency | Historical source and time fields | Interpretation and evidence boundary |
|---|---|---|
| stock_002 : listing state secumain.LISTEDSTATE | lc_liststatus , joined on INNERCODE . CHANGEDATE , CHANGETYPE , INSERTTIME . | Reconstruct the state at the research date from applicable changes. This requires an initial state, event-to-state mapping, same-day ordering, and market coverage. CHANGEDATE is not automatically a public-disclosure date. |
| stock_002 : industry lc_exgindustry.IFPERFORMED | Same-table history by COMPANYCODE , STANDARD , and classification level. INFOPUBLDATE , CANCELDATE . | The schema describes the flag as the latest valid classification. Historical selection follows the task’s interval convention, without an additional current-validity filter. Using publication as the interval start follows that convention, not a general equivalence between disclosure and business validity. |
| fund_041 : manager tenure mf_fundmanagernew | Same-table tenures by INNERCODE , PERSONALCODE , and POSTNAME . ACCESSIONDATE , DIMISSIONDATE , INFOPUBLDATE . | Check both the appointment interval and disclosure cutoff. Preserve co-managers, distinct posts, and repeat tenures. The table records historical managers and assistants. |
| fund_110 : fund classification mf_jyfundtype | Same-table classifications with INNERCODE , TRANSCODE , and STANDARD . EFFECTIVEDATE , CANCELDATE , INSERTTIME . | Under this task’s rule, select the applicable version before testing its category. Identifier correspondence and version grain require care. The table has no dedicated public-disclosure field, so neither effective time nor insertion time automatically establishes public availability. |
| Research requirement | Reference operation | Submission evidence |
|---|---|---|
| Constituents at each date | Read members and weights from each dated index-weight snapshot | Dates and member keys used to select the universe |
| Industry applicable at | Restrict disclosure and cancellation dates at ; apply the effective-date bound required by the source | Historical source, date boundaries, and records retained for each company and date |
| Applicable version | Rank eligible records within each company–date group using the source’s version ordering | Partition keys and the order of temporal filtering and ranking |
| Industry exposure at | Join on company and date, sum weights by industry, and rank within each date | Join keys and the industry assignments contributing to each aggregate |
| Code | Error type | Definition and coding boundary |
|---|---|---|
| A. Temporal and condition filtering | ||
| A1 | Temporal-semantic misuse | Confuses business-valid time, system-recording or ingestion time, disclosure availability, or the observation cutoff. For example, a reporting period is treated as evidence of availability. Interpretation follows the task and field semantics. |
| A2 | Current state substituted for history | Uses current constituents, classifications, ratings, or status snapshots in place of the state valid at the research time. The same substitution is not also coded as faulty historical reconstruction. |
| A3 | Faulty historical reconstruction | Uses the correct historical source and time semantics but implements validity intervals, version selection, historical joins, or reconstruction deduplication incorrectly. The same historical-join defect is not also D1. |
| A4 | Task conditions and sample selection | Omits, adds, or misapplies a non-PIT eligibility or selection rule specified by the task, inputs, or mandatory rules actually supplied to the agent. Covers market, board, security type, rating, size thresholds, designated samples, comparisons, Boolean logic, quantifiers, NULL handling, and Top- or tie retention. Field meaning belongs to B2, time-series operations to C2, numeric metric or rank computation to C3, and joins to D1. |
| B. Schema linking | ||
| Code | Error type | Definition and coding boundary |
|---|---|---|
| C. Data analysis and query planning | ||
| C1 | Multi-step planning | Omits a necessary subtask or plans dependencies or subresult composition incorrectly. Requires direct evidence of a planning defect. Task complexity, incompletion, or a local coding error alone is insufficient. |
| C2 | Time-series and window implementation | Implements sorting, lags, rolling windows, resampling, or calendar alignment incorrectly without using unavailable information or misrepresenting historical states or versions. Future-information use, current-state substitution, and faulty historical reconstruction belong to A1, A2, and A3. C2 is co-assigned only for a distinct time-series computation defect. Financial formulas belong to C3. |
| C3 | Business metrics and numeric computation | Miscomputes financial formulas, non-temporal aggregates, statistics, or missing-value handling, including scaling, currency conversion, and premature rounding during computation. Misunderstood field meaning belongs to B2. Incorrect final presentation of a correctly computed value belongs to E2. |
| D. Joins | ||
| D1 | Join implementation | Understands fields and keys correctly but uses an incorrect SQL join or Pandas merge type, predicate, cardinality, or join deduplication. Key-identity errors belong to B2, and historical-reconstruction join errors to A3. |
| Code | Error type | Definition and coding boundary |
|---|---|---|
| F. Agent and context | ||
| F1 | Budget exhaustion | A documented step, token, time, or cost limit causes termination or incompletion. May coexist with an earlier error. A missing submission or truncated trace alone is not evidence of a limit being reached. |
| F2 | Hallucinated environment capability | Relies on a nonexistent file, tool, package, or runtime capability and leaves the issue unresolved. Invented tables or columns belong to B1 or B2. Actual service failures receive the separate Infrastructure/runtime marker. |
| F3 | Ineffective repair loop | Repeated ineffective attempts despite usable feedback leave an unresolved problem that causes incompletion. May coexist with the original defect or F1 when each has supporting evidence. Ordinary debugging that succeeds is excluded. |
| F4 | Unsupported early stopping | Actively stops or abandons the task despite necessary remaining steps and no justified stopping condition. Non-budget termination alone is insufficient. A limit-triggered stop belongs to F1. Justified stopping at an established task or environment barrier is excluded. A correct artifact submitted incorrectly belongs to E1. |
| G. Execution and dialect | ||
| Stage | Implemented rule |
|---|---|
| Inputs and reference | Load task inputs, output contract, and reference artifacts. Saved outputs take priority; execute the reference when direct results are unavailable. |
| Output extraction | SQL returns a table. Python uses named result variables and captured figures. Unresolved extraction can trigger model-assisted recovery and re-execution. Successfully extracted outputs are retained even when their values mismatch. |
| Strict tables | Match required semantic fields, result components, and complete rows. Enforce column order after allowed presentation normalization, subject to declared task-specific comparison rules. |
| Non-strict tables | Permit column reordering. Accepted aliases, auxiliary columns, and alternative layouts follow declared task mappings. Field identities and complete rows remain required. Non-tabular outputs may invoke a model judge. |
| Numbers and units | Apply declared field precision; otherwise use recognized question precision, defaulting to four decimal places. Convert explicitly identified, dimension-compatible units without guessing scale factors. |
| Order and missing values | Apply task-specific sort rules or recognized ordering instructions; otherwise ignore row order. Preserve complete rows and multiplicity. Normalize actual missing values and blank cells. |
| Endpoint | Primary positive | Independent positive | Agreement (%) | ||
| Any PIT | 200 | 119 | 105 | 91.0 | 0.818 |
| Any PIT, result pass | 83 | 39 | 34 | 94.0 | 0.878 |
| Any PIT, result fail | 117 | 80 | 71 | 88.9 | 0.759 |
| A1 | 200 | 98 | 80 | 90.0 | 0.799 |
| A2 | 200 | 16 | 14 | 97.0 | 0.784 |
| A3 | 200 | 57 | 28 | 85.5 | 0.580 |
| Comparison | Precision | Recall | ||||
| Revised primary audit: screening coverage | ||||||
| Any PIT, diagnostic | 883 | 538 | 188 | 182 | 96.8 | 33.8 |
| Any PIT, combined | 883 | 538 | 200 | 188 | 94.0 | 34.9 |
| Result pass, diagnostic | 380 | 193 | 48 | 46 | 95.8 | 23.8 |
| Result pass, combined | 380 | 193 | 54 | 49 | 90.7 | 25.4 |
| Revised primary audit: type-wise matching, combined flags | ||||||
| Evidence status | Records |
|---|---|
| Non-rounding row or value discrepancy | 111 |
| Runtime failure only | 2 |
| Last-decimal difference only | 11 |
| Matched in retained scenarios | 8 |
| Only retained run timed out | 1 |
| Executed only in excluded scenarios | 2 |
| Mechanism | Rows or values | Last decimal | Runtime only |
|---|---|---|---|
| Disclosure after the effective date | 28 | 0 | 1 |
| Recording or release after the cutoff | 39 | 11 | 0 |
| Listing or operating status around the research date | 22 | 0 | 1 |
| Later membership or classification applied to history | 7 | 0 | 0 |
| Missing latest value replaced by older data | 10 | 0 | 0 |
| Expired name selected | 1 | 0 | 0 |
| Benchmark | Task setting | Data and execution | Output evaluation | Temporal checks |
| BULL | Financial querying | Financial databases; generated SQL | Execution-result accuracy | NR |
| Spider 2.0 | Enterprise analytics | Databases and code projects; SQL/Python/dbt | Task-specific output checks | NR |
| DAB | Data analysis | Databases; SQL and Python tools | Deterministic answer checks | NR |
| FinGAIA | Financial analysis | Web, files, and Python | Answer accuracy | NR |
| FinSkillBench | Investment subtasks | Market/XBRL retrieval; condition-dependent financial tools | Structured outputs; deterministic checks | ✓ As-of data slices |
| AlphaQT-Bench | Factor generation | Price/volume data; single-turn Python generation | Reference-output accuracy | ✓ Prefix-causality safety gates functional accuracy |
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,583 papers