TRADING OPS & RISK / INTERACTIVE ENVIRONMENT
Quarryline Risk Systems
Audit feature availability in a synthetic backtest dataset; make no trading decision.
WHAT YOU’RE LOOKING AT
One environment. Your answer.
A fictional research dataset has four feature rows stamped for May 4. One row uses an earnings revision first published on May 6, two days after the simulated decision time.
Read the evidence and make your choices. The grader checks your numbers, the trap, and your decision. Edit and grade again.
The data room4 documents · Read the excerpts before you decide
Feature manifestFeature manifest rows 1–4
All four rows have observation_date May 4. Three features have availability timestamps on or before the May 4, 16:00 simulation cutoff; the EPS_REV feature has no checked availability timestamp.
Back to correction ↓Source release logRelease log, EPS_REV row
EPS_REV was first published May 6 at 08:00. It was unavailable at the May 4, 16:00 cutoff.
Back to correction ↓Dataset join configurationJoin configuration §1
The draft joins on observation_date only and backfills EPS_REV into May 4. It does not filter on source availability.
Back to correction ↓Point-in-time audit policyAudit policy §2
Every feature must be available at or before the simulation cutoff. Quarantine future-data joins before any backtest interpretation. This task scores data hygiene, not alpha or a trading recommendation.
Back to correction ↓Available tools in the scripted example4 tools
read_feature_manifestRead feature timestamps and intended simulation time.query_release_logCheck when each source became available.validate_asof_joinCompare availability timestamps with the decision cutoff.submit_data_auditAccept or quarantine the dataset with a leakage explanation.
Try the task.
The interactive task is loading. You can also follow the scripted example below.
THE SCRIPTED EXAMPLE
Follow the same task, step by step.
See a plausible mistake, the correction, and the reasoning behind each check.
AGENT ATTEMPT
Follow the attempt.
Scripted calls. Highlighted: the missed trap.
read_feature_manifestAll four rows share the requested observation date.
Tool arguments
{"observation_date":"2026-05-04","feature_count":4}submit_data_auditMissed trapObservation date is mistaken for information availability.
Tool arguments
{"eligible_features":4,"dataset_accepted":true}
Accept all four features. Their observation dates match May 4, so the backtest is point-in-time clean.
SENIOR CORRECTION
What changes, and why.
Authored illustration; no actual expert review.
Accept all four features. Their observation dates match May 4, so the backtest is point-in-time clean.
Quarantine EPS_REV: its May 6 availability is after the May 4 cutoff. Three of four features remain eligible; repair the as-of join before interpreting any backtest.
The correction
A date attached to a row does not show when the information was knowable. Availability, rather than observation date alone, determines eligibility.
Exact evidence
EPS_REV was first published May 6 at 08:00.
Release log, EPS_REV rowEvery feature must be available at or before the simulation cutoff.
Audit policy §2
GRADING
Same checks. Better answer.
The same exercise grader scores the starting answer and reference correction below. Try your own answer to compare.
| What is checked | Agent attempt | After correction |
|---|---|---|
01Numbers right? | 0 / 1 checks pass Four rows − one future-data feature = three eligible features. | 1 / 1 checks pass Four rows − one future-data feature = three eligible features. |
02Traps caught? | 0 / 3 checks pass EPS_REV was published May 6, after the May 4 16:00 simulation cutoff. | 3 / 3 checks pass EPS_REV was published May 6, after the May 4 16:00 simulation cutoff. |
03Sound decision? | 0 / 1 checks pass Quarantine the future-data join and repair it before interpreting any backtest. | 1 / 1 checks pass Quarantine the future-data join and repair it before interpreting any backtest. |
Fictional, authored environment. Your submission is scored by a deterministic exercise grader; the walkthrough shows a scripted example.
Similar agent failures?
Discuss a workflow without source records or confidential information.