Bayesian network - Caltech

Announcements ! Exam: ! Take home ! December 8 10am )ll December 9 10am ! Open book and lecture notes ! No other materials, no collabora)on ! Recita)on next Thursday ! Final project due December 7 2

Linear regression: squared loss x x x x x x x x x x x x x 4

Overfibng ! Min. training error ≠ min. generaliza)on error! error Model complexity 5

Regulariza)on ≈ Posterior inference ! A priori, assume weights should be small need fewer bits to describe, simpler model 6

Classifica)on ! In classifica)on, want to predict discrete label ! For example: binary linear classifica)on Spam Ham x 2 ++++++++--------x 1 f(x;w) = sign(w 0 + w 1 x 1 + w 2 x 2 ) 7

! Predict according to 0-­‐1 loss

Logis)c regression ! Key idea: Predict the probability of a label X 2 P(Y=1 | X) x 1 w T X 9

Logis)c regression ! Maximize (condi)onal) likelihood ln P (D Y | D X ,w)=Nln P (y i | x i ,w)=−i=1! Convex, differen)able! Ni=1! Can find op)mal weights w efficiently! log 1+exp(−y i w T x i )! Can regularize by pubng prior on weights w (exactly as in linear regression) 10

Summary supervised learning ! Want to learn func)on f from X to Y ! Typically f has parameters w (e.g., slope) ! Want to choose parameters to simultaneously minimize training error and model complexity ! Quan)fy training error using loss func)on ! E.g., squared loss for regression ! E.g., log-­‐loss for classifica)on ! Many other loss func)ons useful (e.g., hinge loss for support vector machines) ! Quan)fy model complexity using regularizer ! Choose regulariza)on parameter using cross valida)on

Reminder: Bayesian networks ! A Bayesian network (G,P) consists of ! A BN structure G and .. ! ..a set of condi)onal probability distribu)ons (CPTs) P(X s | Pa Xs ), where Pa Xs are the parents of node X s such that ! (G,P) defines joint distribu)on 12

Next topic: Learning BN from Data ! Two main parts: ! Learning structure (condi)onal independencies) ! Learning parameters (CPDs) E B A M J 0 0 0 0 0 1 0 1 1 0 1 0 1 1 1 0 1 1 0 1 0 1 0 0 1 … … … … … E J A B M 13

Maximum likelihood es)ma)on 15

Es)ma)ng CPDs P(F=cherry)θP(F=cherry)θFlavorFlavorFcherrylimeP(W=red | F)θ 1θ 2Wrapper(a)(b)16

MLE for general Bayes nets ! Given complete data D = {x (1) ,…,x (N) }, want maximum likelihood es)mate 17

! Given: Algorithm for Bayes net MLE ! Bayes Net structure G ! Data set D of complete observa)ons ! For each variable X i es)mate θ Xi |Pa i= Count(X i, Pa)Count(Pa i )! Results in globally op)mal maximum likelihood es)mate! 18

Regularizing Bayesian Networks ! With few samples, counts are “imprecise: θ Xi |Pa i= Count(X i, Pa)Count(Pa i )! Suppose we test only a single candy, and it has cherry flavor. ! What’s the MLE? 19

Pseudo-­‐counts ! Make prior assump)ons about parameters ! E.g.,: A priori, assume coin to be fair ! Prac)cal approach: Assume we’ve seen a certain number of heads / tails: ! Looks like a hack.. In fact, this is equivalent to assuming a Beta prior (Similar to the Gaussian prior for weights in regression) 20

Beta prior over parameters ! Beta distribu)on 2Beta(1,1)2Beta(2,3)6Beta(20,30)6Beta(0.2,0.3)1.51.55454P(θ)1P(θ)1P(θ)3P(θ)30.50.5212100 0.5 1θ00 0.5 1θ00 0.5 1θ00 0.5 1θ21

Bayesian parameter es)ma)on ΘFlavor 1 Flavor 2 Flavor 3Wrapper 1 Wrapper 2 Wrapper 3Θ 1 Θ 222

Learning parameters for dynamical models P(R 0 )0.7Rtf0P(R 1)0.70.3Rtf1P(R 2)0.70.3Rtf2P(R 3)0.70.3tfP(R 4)0.70.3Rain 0Rain 1Rain 2Rain 3R 3Umbrella 4Rain 41Umbrella 1Umbrella 2Umbrella 31 )92R 1tfP(U 1 )0.90.2R 2tfP(U 2 )0.90.2R 3tfP(U 3 )0.90.2R 4tfP(U 4 )0.90.223

Structure learning ! So far: Discussed how to learn parameters for given Bayes Net structure ! Where does the structure come from? 24

Score based structure learning ! Define scoring func)on S(G;D) ! Quan)fies, for each structure G, the fit to the data D ! Search over Bayes net structures G G ∗ ∈ arg minGS(G; D)! Key ques(on: how should we score a Bayes Net? 25

MLE score ! Idea: For fixed structure G, already know how to calculate MLE for parameters θ G ∈ arg maxθ! Can use maximum likelihood of data to score a Bayes Net S(G; D) = maxθ! Turns out, this scoring func)on has a simple form: nnlog P (D | θ G ,G)=m Î(X i , Pa i ) − m Ĥ(X i )i=1log P (D | θ, G)log P (D | θ, G)i=126

Example: Likelihood score E J A B M E B A M J 0 0 0 0 0 1 0 1 1 0 1 0 1 1 1 0 1 1 0 1 0 1 0 0 1 … … … … … 27

Maximizing the MLE score ! Score of a BN: arg maxGS(G) = arg maxGnÎ(X i , Pa i )i=1! What’s the op)mal solu)on to this problem?? 28

Finding the op)mal MLE structure ! Op)mal solu)on for MLE is always the fully connected graph!!! Non-­‐compact representa)on; Overfibng!! ! Solu)ons: ! Priors over parameters / structures ! Constraint op)miza)on (e.g., bound #parents) 29

! Prefer “simpler” models Regularizing a BN ! Bayesian Informa)on Criterion (BIC) score: S BIC (G) =ni=1Î(X i , Pa i ) − log N2N |G|! Hereby ! |G| is the number of parameters of G ! n is number of variables ! N is number of training examples ! Theorem: BIC is consistent (will iden)fy correct structure as N ∞) 30

Finding the best structure ! Finding the BN that maximizes the BIC score is NP hard ! Instead: use search techniques (see first part of this course) ! Start with no edges ! Add or remove edges, while preserving acyclicity ! However: Can solve one interes)ng special case op)mally: 31

Finding the op)mal tree BN ! Scoring func)on ! Scoring a tree 32

Finding the op)mal tree ! Given graph G = (V,E), and nonnega)ve weights w e for each edge e=(X i ,X j ) ! In our case: w e = I(X i ,X j ) ! Want to find tree T that maximizes ∑ e w e ! Maximum spanning tree problem! ! Can solve in )me O(|E| log |E|)! 33

Chow-­‐Liu algorithm ! For each pair X i , X j of variables compute ! Compute mutual informa)on ! Define complete graph with weight of edge (X i ,X i ) given by the mutual informa)on ! Find maximum spanning tree undirected tree ! Orient the edges using breadth-­‐first search 34

Example E B A M J 0 0 0 0 0 1 0 1 1 0 1 0 1 1 1 0 1 1 0 1 0 1 0 0 1 … … … … … 35

Summary ! To learn a Bayes net, need to ! Learn structure ! Learn parameters ! If all variables are observed ! Get maximum likelihood parameter es)mate by coun)ng ! Use pseudo-­‐counts (Beta prior) to avoid overfibng ! Search for structure by maximizing score func)on ! Score = Likelihood for best choice of parameters ! Can find op)mal trees efficiently! 36