Skip to main content

NotesTesting whether an edge is real

Count your own trials

A finding reached after many tries deserves the same discount as a backtest picked from many variants.

What I was looking for

How many looks my own research took before each result, so it can be discounted like any backtest.

What I found

The overfitting papers tell investors to ask others for their trial count. The same question applies to my own research, and attempts can't be reconstructed afterwards.

On the Liquidity Engine I counted 97 looks (72 detector configurations, 5 variants, 6 loosening steps, 12 momentum cells and 2 exploratory reads) against the one registered test that decided. Cernos now counts every parameter trial by design.

The move

I log every attempt, not just conclusions, so any result can be judged against how many looks it took.