Discovery's Real Bottleneck Is Geometry, Not Data

A new paper from researchers at MIT and the Santa Fe Institute argues the real bottleneck in scientific discovery isn't data volume or compute power. It's the geometry of hypothesis space.

The paper, posted to arXiv under "Discovery under Hypothesis Redundancy," formalizes a problem working scientists have felt for years. As fields accumulate more hypotheses – candidate explanations, drug targets, material structures – the number of distinct things worth testing grows. The number of genuinely new things that can be learned from any single test shrinks. The authors show this is not a statistical quirk. It is a geometric property of how hypotheses relate to each other.

When two hypotheses make overlapping predictions, testing one gives you partial information about the other. Test enough overlapping hypotheses and you hit a wall. Each new experiment returns less marginal information than the last, even if the hypothesis itself is novel. The paper calls this the "discovery bottleneck."

The key insight is that the bottleneck is not about how many hypotheses you can generate. It is about how many are geometrically independent. In high-dimensional spaces, most hypotheses are near-redundant with some combination of others. The effective number of independent tests a field can run is far smaller than the raw count of hypotheses on the table.

The authors derive a formal measure of hypothesis redundancy based on the angle between prediction vectors. Two hypotheses with a small angle between their predictions are nearly redundant. Testing one tells you most of what the other would. The discovery bottleneck tightens as the average pairwise angle shrinks.