Skip to main content

NotesTesting whether an edge is real

Pooled numbers hide sign flips

A statistic pooled across segments can describe no segment that exists, and splitting it is where bias gets in.

What I was looking for

Whether forecast ambiguity tracks error the same way across data sources.

What I found

The pooled number gets quoted; the split number gets acted on. The first can hide segments running the other way, and the second can be a false positive from trying many splits.

In my forecasting work, a pooled correlation of −0.59 split into +0.175, −0.005, −0.266 and −0.500 by source. Because the splits were registered first, the difference between sources counts as a finding.

The move

I register the splits before the data arrives, so the segments are a result, not a search.