Zero crashes against a controller built to crash. The losses are on the page too.
Zero hits in CARLA against a controller built to crash. +32% on Space Invaders, from raw pixels. −12% on Freeway with the exact same code. Doom rules 17.9 against random 3.2 — and the memory layer on top only ties. Every number, good and bad, produced in-house and reproducible from a frozen experiment card.
You have been handed gear that worked in the demo and quit on you in the field. Every vendor shows the run that worked. We show all of them. A result that only ever gets better is less believable, not more.
The four
| domain | result | what it means |
|---|---|---|
| Driving CARLA, a controller built to crash | hits = 0 100–490 m at 38–40 km/h · ~9.7 m gap held · ~400 refusals logged | The floor takes the car away from a hostile driver and holds the gap. Floor off, the same driver crashes. |
| Space Invaders raw 210×160 pixels, zero emulator RAM | 200.6 vs 152.2 +32% · peak 276.2 @50 episodes | When one mistake ends it all, learning what to refuse helps. |
| Freeway same mechanism, same code | 9.2 vs 10.4 −12% · 3 rules formed, every one blocking up | It hurts when the only way to score is also the dangerous move. |
| Doom rule-based, no ML | random 3.2 · rules 17.9 · rules+memory 20.3 paired t = 0.78 | Rules beat random by a wide margin. The memory layer on top is a tie, not a win, and we report it that way. |
What this proves: the same mechanism held a line against a hostile driver in simulation, and it helps or hurts depending on the job. What it does not prove: anything about the real world — real terrain, real weather, damaged sensors or a real enemy. None of that has been tested.
Same code, opposite results
Invaders and Freeway are not two experiments. They are the same code pointed at two games — Freeway imports the Invaders failure memory unchanged, with no per-game tuning. The design carried over. The learning did not.
When a mistake is fatal
A death in Invaders wipes out every point still to come. Its true cost is everything that was left, so a flat “never do that” prices it correctly — by accident. An infinite penalty is right when the loss really is everything.
When a mistake just costs ground
A collision in Freeway knocks the chicken back and play goes on. The cost is the lost ground, nothing more. The flat “never” prices it wrong and takes away the only move that scores.
No single price for failure is right in both cases — and a confidence threshold is a single price. Any one-size safety gate that is right for one kind of danger will be wrong for the other. This is the most transferable result we have, and it is an argument against the common default gate, not just against ours.
A bug we caught in ourselves, and published
The emulator only counts a life as lost at the end of a 127-frame death animation. So every failure was recorded from frames where the player was already destroyed. Of 374 frames it blamed, 0 showed the hazard. At the real moment of impact, 17 of 17 did. It was learning from the wrong moment entirely — and still drawing a curve that looked like learning.
Five bugs like this are logged on the experiment cards. Each one quietly gave a believable result before it was caught. We list them because blaming the wrong moment is the failure most likely to be hiding in any system like this, anyone’s included. Ask to see the list for any system you are handed.