Notes · John Kingola

How I take problems apart

I check that something is true before anyone relies on it: trading strategies, AI agents, and the systems around them.

Each building block is a finding I keep reusing: what I was looking for, what I found, and the move it makes me take on the next problem. 20 of them, in 5 categories, plus essays that walk through one decision.

Getting the numbers right

Metrics that look correct and aren't, and how I catch them.

  1. The start is a peak
    Looking for
    Drawdown bugs in quant libraries, checked against hand-computed values on short series.
    Found
    A drawdown is measured from the highest wealth so far, and the starting wealth is the first high.

    Found 2026-09-26

  2. Annualisation is an inference
    Looking for
    Whether annualised Sharpe and volatility match a hand calculation on real trading calendars.
    Found
    Turning a daily number into an annual one means guessing how many periods a year has. Libraries guess silently.

    Found 2026-09-26

  3. A name is not a formula
    Looking for
    Which published formula each risk metric really computes, my own Cernos code included.
    Found
    Many metrics have several published formulas under one name. The label tells you the family, not the member.

    Found 2026-09-27

  4. Flipped inputs pass silently
    Looking for
    Bugs that still produce plausible numbers, in an Expert Advisor audit and across trading libraries.
    Found
    Feed a calculation the right number the wrong way round and nothing crashes. In a signal, you trade the opposite strategy.

    Found 2026-09-26

  5. Snapshot tests pin bugs
    Looking for
    Why correct fixes kept failing the upstream test suites they were sent to.
    Found
    A test that records what the code outputs, not what is right, protects the bug. The fix then looks like a regression.

    Found 2026-09-26

Testing whether an edge is real

How I separate a signal from the search that found it.

  1. A gate must beat no gate
    Looking for
    Whether a liquidity-void entry signal added anything beyond the exits and fees around it.
    Found
    To know what a signal adds, run the same rules without it. A signal that can't beat its own absence adds nothing.

    Found 2026-10-02

  2. Count your own trials
    Looking for
    How many looks my own research took before each result, so it can be discounted like any backtest.
    Found
    A finding reached after many tries deserves the same discount as a backtest picked from many variants.

    Found 2026-09-23

  3. Explaining a move is not predicting one
    Looking for
    Whether order-flow imbalance, which explains most of a BTC move, can predict the next one.
    Found
    A variable can explain most of a price move as it happens and forecast almost nothing about the next one.

    Found 2026-10-02

  4. Pooled numbers hide sign flips
    Looking for
    Whether forecast ambiguity tracks error the same way across data sources.
    Found
    A statistic pooled across segments can describe no segment that exists, and splitting it is where bias gets in.

    Found 2026-09-23

  5. Negative results are results
    Looking for
    Evidence that market makers pull depth from the book just before quotes move.
    Found
    A dead hypothesis still leaves the measuring tool behind, and the tool often finds the real result.

    Found 2026-09-22

  6. Decide the interval before the data
    Looking for
    When to stop collecting order-flow data, decided before looking at it.
    Found
    A result is a number plus an interval. Deciding in advance what width counts as settled stops motivated reasoning.

    Found 2026-09-22

Reading risk

Where losses hide in records and strategies that look safe.

  1. Smoothness hides the tail
    Looking for
    What sits underneath published trading records that look too good.
    Found
    A record that looks too smooth often means the risk hasn't arrived yet, or was averaged away.

    Found 2026-09-23

  2. Looping does not fix a negative edge
    Looking for
    Whether repeating a small losing trade in a loop can make it safe.
    Found
    Repeating a losing bet doesn't spread the risk. Cost grows linearly, risk as a square root, so the loss gets more certain.

    Found 2026-09-22

  3. One regime is one data point
    Looking for
    Whether a result from one session, or one half-year, holds in the next.
    Found
    A result measured in one market regime says nothing about the next, and the sample itself can't tell you.

    Found 2026-09-23

Building software that holds up

What only shows up when real people run the code.

  1. The default is not where it runs
    Looking for
    Why Cernos passed its tests and then failed on first real use.
    Found
    Code leans on defaults it never states, and tests on the developer's machine share them.

    Found 2026-09-27

  2. Trust the live system over the schema
    Looking for
    Which account field really says how much GPU credit you can spend.
    Found
    Documentation describes intent. A running system describes behaviour. Your code has to survive the second.

    Found 2026-09-21

  3. Scope the promise, not the code
    Looking for
    What CreditWatch 1.0.0 could honestly claim on each platform.
    Found
    When a release isn't ready everywhere, narrow what you claim, not what you built.

    Found 2026-09-21

  4. Alerts must not fire on the past
    Looking for
    How runway alerts should behave when someone switches them on late.
    Found
    An alert for a threshold already crossed fires at once, and teaches the user to ignore every alert.

    Found 2026-09-21

Working with AI

When a model's output can be trusted, and when it needs a person.

  1. Agreement is not corroboration
    Looking for
    Whether disagreement between forecasting models is a useful measure of uncertainty.
    Found
    Several sources agreeing is only as informative as their independence.

    Found 2026-09-23

  2. The generator-verifier gap
    Looking for
    Where an LLM loop can run unattended, and where a person has to check.
    Found
    Where checking an answer is easier than producing it, AI can run in a loop. Where it isn't, a person has to check.

    Found 2026-10-08

Essays

One decision, walked through end to end.

  1. Derive exits from costs, don't fit them
    Looking for
    An exit rule for a market-making test that couldn't overfit two trades of data.
    Found
    With two trades of training data, a fitted exit is a search. I built the exit out of measured costs instead.

    Found 2026-10-02

  2. My strategy test failed, and I published it
    Looking for
    Whether the void gate's edge on one session was real, or the only session I had looked at.
    Found
    A gate looked like an edge on the one session I had studied. I registered the test before replaying a fresh one, and all three hypotheses failed.

    Found 2026-10-02

  3. Graph, not grep: what coding agents need from a codebase
    Looking for
    What a coding agent needs from a codebase to change code safely.
    Found
    Grep gives an agent strings. To change code safely it needs callers, callees, and what a change will break. That is why Rig exists.

    Found 2026-05-25