Skip to main content

NotesGetting the numbers right

Snapshot tests pin bugs

A test that records what the code outputs, not what is right, protects the bug. The fix then looks like a regression.

What I was looking for

Why correct fixes kept failing the upstream test suites they were sent to.

What I found

Maintainers trust a green suite. When the pinned value is wrong, the suite argues against the fix, and a reviewer reasonably asks why the expected numbers changed.

My skfolio fix had to change four pinned values that encoded the start-is-a-peak bug. In ffn, a test added a month earlier pinned the unannualised Ulcer Performance Index, so the fix rewrote two tests to land.

The move

When a fix breaks a test, I ask where the expected value came from before I assume my fix is wrong.