From A Technical Explanation of Technical Explanation:

Visualize probability density or probability mass – probability as a lump of clay that you must distribute over possible outcomes.

Let’s say there’s a little light that can flash red, blue, or green each time you press a button. The light flashes one and only one color on each press of the button; the possibilities are mutually exclusive. You’re trying to predict the color of the next flash. On each try, you have a weight of clay, the probability mass, that you have to distribute over the possibilities red, green, and blue. You might put a fourth of your clay on the “green” possibility, a fourth of your clay on the “blue” possibility, and half your clay on the “red” possibility – like assigning a probability of green:25%, blue:25%, and red:50%.

The metaphor is that probability is a conserved resource, to dole out sparingly. If you think that “blue” is more likely to flash on the next experiment, you can assign a higher probability to blue, but you have to take the probability mass from the other hypotheses – maybe steal some clay from red and add it to blue. You can never get any more clay. Your probabilities can’t sum to more than 1.0 (100%). You can’t predict a 75% chance of seeing red, and an 80% chance of seeing blue.

I’ll ask you to imagine instead that your belief in a hypothesis is like sand, which you can spread out over different parts of (flattened-to-two-dimensional) “hypothesis space”. As above, you have a fixed total amount of sand for any given question to distribute between all the possibilities.

In the very, very beginning you have a flat prior, which is like all the sand being spread out evenly in a thin layer across a giant sheet of paper which represents all the possible worlds. (Or possibly you have some sort of complexity-weighted Kolmogorov-ish prior, which in this metaphor is best represented by “how large a circle” each hypothesis gets on the paper, with the sand remaining equally distributed.) Accumulating evidence lets you heap sand up on the parts of the paper that seem more likely, helping constrain your beliefs about what you expect to occur.

Fun fact about real sand: you can’t actually heap it up all that high. The pile can only get a certain amount of steep, and then no matter how you keep adding sand, it just sort of slides off to the side. The angle of repose is a measure of the steepest angle you can get the sides to pile up at. Primary determining factors include shape (smooth sand grains can’t pile as high as ones with rough edges) and wetness (sand castles fall apart once they dry).

(You can see this large-scale in places like the Grand Canyon: all those alternating layers of near-vertical and sloping are different kinds of rock that have different durability, and so have eroded more quickly or slowly than each other)

What’s interesting about this? Imagine the angle of repose in our hypothesis-space-sandbox as being like your model uncertainty (or rather, its inverse). Depending on which sandbox you’re in, or what you’re using to reason about the problem, you always have some degree of fundamental limitation in your ability to pile up the sand narrowly over one specific hypothesis. In order to get the pile higher, you have to swap it all out for better sand.

If I’ve seen 50 badly designed studies in favor of Supplement X, seeing 50 more won’t shift my beliefs much – I’ve already got the sand piled as high as I can over “Supplement X is good,” and the additional studies are like sand just sliding down the sides of the pile. In order to be more sure than that, I need to have a better model, or a better *kind* of evidence.

Contrast that with my sandbox of beliefs about apples and bowling balls falling on Earth, which is more like a drippy castle of narrowly concentrated credence.

I think lots of arguments (about cause prioritization within Effective Altruism, for example) boil down to disagreements about which methods of reasoning are like drippy castles, and which ones are like loose, dry sand. And people can get quite frustrated when it seems like their interlocuter is trying to use dry-sand arguments to make a very tall castle, and sort of using their hands to try and push it all together when it tries to settle down into its natural, more diffuse shape.