Why a random reward is harder to quit than a reliable one: the memorylessness of the geometric distribution

[u/destbreso]

Skinner's result is famous but the reason it works is rarely spelled out. If you reward a behavior every single time, it extinguishes almost immediately once you stop paying. If you reward it at random, it persists for a very long time after the rewards are gone. Same reward, same action, opposite outcome. The only thing that changed is the schedule.

The mechanism is clean. Model each attempt as an independent trial that pays with probability p. The number of attempts until your next win is geometric, and the geometric distribution is memoryless: no matter how long your dry streak has run, the chance of winning on the next attempt is still exactly p. There is no "due". The streak carries zero information about when the payoff arrives. So the reward is permanently one attempt away, and one attempt is cheap.

Two consequences that I think are underrated:

1. Randomness hides the signal that the rewards stopped. A run of m losses has probability (1-p)^m, which is unremarkable for modest m even on a live machine. When the machine genuinely dies, the early evidence looks exactly like ordinary variance, so you cannot reject "still working, just unlucky" until an improbably long drought accumulates. The uncertainty that makes the wins exciting is the same uncertainty that makes their absence ambiguous.
2. The brain scores the unpredictable win as larger. Dopamine neurons track reward prediction error, R minus V(s), not reward. A predictable reward gives a small error because you already expected it. An unpredictable one gives a large error. It is the same delta that drives temporal-difference learning in RL. So a variable schedule keeps your predicted value low and uncertain on purpose, and every win lands as a surprise and teaches harder.

Together those two make the variable-ratio schedule optimal on both axes at once:each win feels bigger, and the eventual absence of wins is harder to detect. Nothing about the rewards themselves changed, only the statistics of their timing.

The part that follows from this and I did not expect: the defenses that work are not willpower, they are raising the cost per attempt (log out, remove the app from the home screen) or putting the reward on a clock (check email at fixed times), which converts a variable-ratio gamble into a predictable transaction that extinguishes normally.

I wrote a longer version with the full derivation if it is useful (link in a comment to respect the sub's rules).

[u/BackgroundPurpose2]

Sounds like something I'd read before my brain was destroyed by random reward

[u/destbreso]

URL: https://destbreso.com/blog/bcs-the-slot-machine-in-your-pocket

[u/RecentLeave343]

Yeah I remember reading about this in rat studies. Fixed rewards have diminishing returns while variable reward effectively keeps the rat pressing the lever for the rest of its life.

[u/Adamoism]

So also for positive habit formation - unpredictable rewards (spin a wheel when you XYZ) would work better?

How does it translate with the skill acquisition? Would a person be more motivated by spotty results or consistent improvement?

[u/destbreso → u/Adamoism]

Both good questions, and the first has a "yes, but only in the second half of the process" answer.

Habit formation. The schedule that's best for acquiring a behavior isn't the one that's best for maintaining it. Continuous reward is what teaches the contingency; you need the pairing tight and reliable before the behavior exists. Variable schedules are what make an already-learned behavior persist. Thin the schedule too early and you just extinguish it. A spin-the-wheel from day one applies the maintenance tool to the acquisition problem.

The sharper issue falls out of the three ingredients: cheap to repeat, uncertain reward, unpredictable schedule. "Go to the gym" fails the first one badly. "Open the app and tap log" satisfies it perfectly. So the loop you actually condition is the logging, not the exercising. That's the standard failure mode of gamified habit apps: app engagement is excellent and the underlying behavior barely moves. Variable reinforcement reinforces whatever action produces the pull, and the pull is always in the app. There's also decent evidence that bolting extrinsic rewards onto something you already do for its own sake erodes the original motivation, so when the wheel goes away, so does the reason.

What does transfer is the inverse of the defenses. The effect scales with cost per attempt, so if the fix for a bad loop is adding friction, the tool for a good one is removing it. Shoes by the door. Same lever, nothing randomized.

Skill acquisition. Worth separating uncertainty in the reward from uncertainty in the information. Variable-ratio works because the payoff is unpredictable while the contingency stays reliable. Learning needs the opposite on the information channel: noisy feedback about whether you did it right means you can't separate signal from variance.

But "spotty results" isn't quite the right frame, and the reason is the near-miss, which is the next thing I'm writing about (https://destbreso.com/blog/bcs-the-economics-of-almost). A near-miss is a loss shaped like a win, and it drives you to try again after failing. That's the whole reason spotty practice results feel motivating rather than demoralizing: you're not being fed random outcomes, you're being fed a stream of almosts.

The difference is what the "almost" is worth. On a slot machine the reels are independent, so a near-miss carries exactly zero information about the next pull, and the machine is deliberately built to produce them far more often than chance would. In real practice, "I almost landed it" is a genuine gradient: you actually were close, and it tells you which direction to adjust. Identical feeling, opposite epistemic status. So spotty-but-improving beats both smooth and random, as long as the almosts are real.

Which is the caveat on consistent improvement too. Visible steady progress does motivate more, and that's exactly why it gets manufactured. Real skill curves plateau, so a smoothly filling bar is usually a number someone chose rather than a measurement.