statisticalrethinkin..

9 Big Entropy and the Generalized Linear ModelProbability theory is just counting. It looks fancy, and the mathematics can be opaque.But the trick I introduced in Chapter 3, where we converted a posterior probability calculationto a counting problem, works because probability theory really is just counting. Whatdoes it count? Probability counts the ways something can happen, given what we know (orthink we know). Remember, probability is a device for describing uncertainty, the relativeplausibility of different things, given certain constraints on how we believe things can happen.As a result, if you just write down everything that can happen, eliminate things thatare incompatible with the constraints, and then count up how many ways each thing canhappen, you are doing probability theory. Counting is not an approximation. Instead, it’sthe real thing.And that’s just about all Bayesian inference is, at least mechanically. e probability distributionswe routinely use in statistical modeling—the Gaussian, binomial, and others—areexactly the distributions that you arrive at when you just count up all the ways each thingcan happen, given a precise statement of prior information. As a result, these distributionsare the most honest expressions of the state of information that goes into defining the problem.ey are the unique MAXIMUM ENTROPY distributions that are consistent with ourassumptions. ere is no guarantee that what happened or will happen obeys the distribution.But there is a guarantee that no other distribution can do a better job of describing theuncertainty.Here’s an example. ink back to the globe tossing experiment from Chapter 2. Insteadof tossing the real globe, we’ll imagine tossing an inflatable ball that is half covered in blue,“water,” and half in green, “land.” As a result, there are as many ways to get a water result asthere are to get a green result. Remember, the physics of ball tossing are deterministic—anyrandomness in the outcome is a result of our lack of information, not of any true indeterminacyin the physics. So by saying the probability of water is one-half, we must mean thatthere are just about as many initial conditions that lead to observing water as there are initialconditions that lead to observing land.9.1. Generalized Linear Modelse most accessible way to resist the Tyranny of Gauss is to learn to construct, estimateand interpret generalized linear models (GLM’s). is label is unfortunate, because it referencesmathematical definition rather than purpose, and it is easy to confuse with generallinear models, which are quite different. Sadly, GLM is the standard terminology, so youhave to know it.267