Tasks we’ve built. Hard for frontier models.
Examples of the expert-built environments, evaluations and datasets we deliver to AI labs. Each one is written by a vetted expert, ships with a verifier or rubric, and is tested against current frontier models before delivery.
Biostatistics · R / Python
Sample task · Software · RL environment
Expert: Senior distributed-systems engineer
Verified · 8+ yrs · ~5 hrs plus independent review
Fix lost writes in a replicated key-value store
A 3-node key-value service copies writes from an elected leader to the other nodes. During certain network partitions, writes the system already confirmed disappear after the leader fails over. The agent must repair replication.go and election.go so every confirmed write survives any single partition. The public API can’t change.
A consistency contract that defines exactly when a write counts as confirmed
A seeded partition-schedule config
A network-simulation harness the agent can’t modify
Runs against hundreds of seeded partition and failover schedules
History checker confirms no confirmed write is lost or reordered
Build and API-surface checks confirm nothing else broke
Pass/fail and fully reproducible
WHY FRONTIER MODELS MISS IT
Retry on timeout. Passes ordinary tests but still loses data, because a deposed leader can confirm a write after a new leader has taken over.
Track leader terms across both files. This is the correct fix: a stale leader can no longer confirm writes.
Biostatistics · R / Python
Sample task · Software · RL environment
Expert: Senior distributed-systems engineer
Verified · 8+ yrs · ~5 hrs plus independent review
Fix lost writes in a replicated key-value store
A 3-node key-value service copies writes from an elected leader to the other nodes. During certain network partitions, writes the system already confirmed disappear after the leader fails over. The agent must repair replication.go and election.go so every confirmed write survives any single partition. The public API can’t change.
A consistency contract that defines exactly when a write counts as confirmed
A seeded partition-schedule config
A network-simulation harness the agent can’t modify
Runs against hundreds of seeded partition and failover schedules
History checker confirms no confirmed write is lost or reordered
Build and API-surface checks confirm nothing else broke
Pass/fail and fully reproducible
WHY FRONTIER MODELS MISS IT
Retry on timeout. Passes ordinary tests but still loses data, because a deposed leader can confirm a write after a new leader has taken over.
Track leader terms across both files. This is the correct fix: a stale leader can no longer confirm writes.
Biostatistics · R / Python
Sample task · Software · RL environment
Expert: Senior distributed-systems engineer
Verified · 8+ yrs · ~5 hrs plus independent review
Fix lost writes in a replicated key-value store
A 3-node key-value service copies writes from an elected leader to the other nodes. During certain network partitions, writes the system already confirmed disappear after the leader fails over. The agent must repair replication.go and election.go so every confirmed write survives any single partition. The public API can’t change.
A consistency contract that defines exactly when a write counts as confirmed
A seeded partition-schedule config
A network-simulation harness the agent can’t modify
Runs against hundreds of seeded partition and failover schedules
History checker confirms no confirmed write is lost or reordered
Build and API-surface checks confirm nothing else broke
Pass/fail and fully reproducible
WHY FRONTIER MODELS MISS IT
Retry on timeout. Passes ordinary tests but still loses data, because a deposed leader can confirm a write after a new leader has taken over.
Track leader terms across both files. This is the correct fix: a stale leader can no longer confirm writes.
Details are illustrative. Client work stays under NDA.