Testing what models can do inside real systems

The problem

Models are getting capable quickly, and demos of them are easy. What is harder to find out is what holds up once real users, existing systems and messy constraints are involved: what an agent should remember, which decisions a model can be trusted with, and whether a result survives more than one lucky run.

What I did

Since 2024 I’ve spent much of my time building with agents, developer tools and automation, and running small experiments to answer specific questions about new models. Some of it is written up in What I learned giving agents memory.

What we built

• iterate-to-confidence. A Claude Code skill that refines an artifact (a spec, plan, design doc or code change) in a score → fix-weakest → re-score loop until a confidence rubric reaches a target, 95% by default. It guards against the 2 ways such loops fail: score inflation, where every re-score must cite evidence such as file paths, line numbers or commands, and scope padding. It pauses for a human on trade-offs, new dependencies, public-contract changes and naming. Open source under MIT.
• JEV/98. A late-90s computer simulation in which TypeSafe’s Jev model chooses every action. Every ~650 ms the model receives the machine state and a finite set of legal actions and returns 1 typed choice with probabilities; ordinary deterministic code applies the consequence. The UI shows the full probability distribution, 3 missions come with it, and runs are shared as self-contained replay links, with no database. Without an API key a clearly labelled demo policy plays instead.
• JEV/78. The 1978 Oregon Trail as the same kind of typed decision loop: Jev chooses every legal wagon action, and the world is seeded so a death can be replayed.
• Jev × Hyperliquid lab. A Bun and TypeScript harness that tests whether Jev has short-horizon trading value on the Hyperliquid BTC perpetual. It covers paper trading with real spread, slippage, fees and funding; fixture replay with model answers cached once and reused; comparison against cheap baselines on the same tape after the same costs; and live execution that is deliberately hard to switch on.

What I learned

From the memory work:

• Memory is not a transcript. A transcript is history; memory is compression.
• The hard part is selection: what to surface, not how much to store.
• Memories have different lifetimes, and need provenance, confidence and time attached.
• Good memory requires good forgetting.

From the Hyperliquid lab:

• Compute the cost floor before spending anything on model calls.
• Small samples lie in your favour. A run of 8 correct calls out of 9 disappeared at 31 trades.
• Prompt wording moved the trade rate from 3% to 70% of decisions on identical states, so prompts are part of the experiment and are cached and compared as such.

What came of it

The memory note is published. iterate-to-confidence is public on GitHub.

The Hyperliquid lab reached a clear negative result: across about 2,100 decisions at a 30-second horizon, Jev’s signal was indistinguishable from noise and never came within an order of magnitude of the cost of trading. In the first runs, its own estimate that a trade would be profitable after costs never went above 0.51, so the default gate blocked every trade. Total model spend for the experiment was about 20 cents.