Information Theory, Inference, and Learning ... - MAELabs UCSD

More documents

Recommendations

Info

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981 You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links. 116 6 — Stream Codes a b ✷ aa ab a✷ ba bb b✷ aaa aab aa✷ aba abb ab✷ baa bab ba✷ bba bbb bb✷ aaaa aaab aaba aabb abaa abab abba abbb baaa baab baba babb bbaa bbab bbba bbbb 00000 0000 00001 000 00010 0001 00011 00100 0010 00101 001 00110 0011 00111 00 01000 0100 01001 010 01010 0101 01011 01100 0110 01101 011 01110 0111 01111 01 10000 1000 10001 100 10010 1001 10011 10100 1010 10101 101 10110 1011 10111 10 11000 1100 11001 110 11010 1101 11011 11100 1110 11101 111 11110 1111 11111 11 naturally be produced by a Bayesian model. Figure 6.4 was generated using a simple model that always assigns a probability of 0.15 to ✷, and assigns the remaining 0.85 to a and b, divided in proportion to probabilities given by Laplace’s rule, PL(a | x1, . . . , xn−1) = Fa + 1 , (6.7) Fa + Fb + 2 where Fa(x1, . . . , xn−1) is the number of times that a has occurred so far, and Fb is the count of bs. These predictions correspond to a simple Bayesian model that expects and adapts to a non-equal frequency of use of the source symbols a and b within a file. Figure 6.5 displays the intervals corresponding to a number of strings of length up to five. Note that if the string so far has contained a large number of bs then the probability of b relative to a is increased, and conversely if many as occur then as are made more probable. Larger intervals, remember, require fewer bits to encode. Details of the Bayesian model Having emphasized that any model could be used – arithmetic coding is not wedded to any particular set of probabilities – let me explain the simple adaptive 0 1 Figure 6.5. Illustration of the intervals defined by a simple Bayesian probabilistic model. The size of an intervals is proportional to the probability of the string. This model anticipates that the source is likely to be biased towards one of a and b, so sequences having lots of as or lots of bs have larger intervals than sequences of the same length that are 50:50 as and bs.
Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981 You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links. 6.2: Arithmetic codes 117 probabilistic model used in the preceding example; we first encountered this model in exercise 2.8 (p.30). Assumptions The model will be described using parameters p✷, pa and pb, defined below, which should not be confused with the predictive probabilities in a particular context, for example, P (a | s = baa). A bent coin labelled a and b is tossed some number of times l, which we don’t know beforehand. The coin’s probability of coming up a when tossed is pa, and pb = 1 − pa; the parameters pa, pb are not known beforehand. The source string s = baaba✷ indicates that l was 5 and the sequence of outcomes was baaba. 1. It is assumed that the length of the string l has an exponential probability distribution P (l) = (1 − p✷) l p✷. (6.8) This distribution corresponds to assuming a constant probability p✷ for the termination symbol ‘✷’ at each character. 2. It is assumed that the non-terminal characters in the string are selected independently at random from an ensemble with probabilities P = {pa, pb}; the probability pa is fixed throughout the string to some unknown value that could be anywhere between 0 and 1. The probability of an a occurring as the next symbol, given pa (if only we knew it), is (1 − p✷)pa. The probability, given pa, that an unterminated string of length F is a given string s that contains {Fa, Fb} counts of the two outcomes is the Bernoulli distribution P (s | pa, F ) = p Fa a (1 − pa) Fb . (6.9) 3. We assume a uniform prior distribution for pa, P (pa) = 1, pa ∈ [0, 1], (6.10) and define pb ≡ 1 − pa. It would be easy to assume other priors on pa, with beta distributions being the most convenient to handle. This model was studied in section 3.2. The key result we require is the predictive distribution for the next symbol, given the string so far, s. This probability that the next character is a or b (assuming that it is not ‘✷’) was derived in equation (3.16) and is precisely Laplace’s rule (6.7). ⊲ Exercise 6.2. [3 ] Compare the expected message length when an ASCII file is compressed by the following three methods. Huffman-with-header. Read the whole file, find the empirical frequency of each symbol, construct a Huffman code for those frequencies, transmit the code by transmitting the lengths of the Huffman codewords, then transmit the file using the Huffman code. (The actual codewords don’t need to be transmitted, since we can use a deterministic method for building the tree given the codelengths.) Arithmetic code using the Laplace model. Fa + 1 PL(a | x1, . . . , xn−1) = a ′(Fa ′ + 1). (6.11) Arithmetic code using a Dirichlet model. This model’s predictions are: Fa + α PD(a | x1, . . . , xn−1) = a ′(Fa ′ + α), (6.12)
Page 1 and 2:
Copyright Cambridge University Pres
Page 3 and 4:
Page 5 and 6:
Page 7 and 8:
Page 9 and 10:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17 and 18:
Page 19 and 20:
Page 21 and 22:
Page 23 and 24:
Page 25 and 26:
Page 27 and 28:
Page 29 and 30:
Page 31 and 32:
Page 33 and 34:
Page 35 and 36:
Page 37 and 38:
Page 39 and 40:
Page 41 and 42:
Page 43 and 44:
Page 45 and 46:
Page 47 and 48:
Page 49 and 50:
Page 51 and 52:
Page 53 and 54:
Page 55 and 56:
Page 57 and 58:
Page 59 and 60:
Page 61 and 62:
Page 63 and 64:
Page 65 and 66:
Page 67 and 68:
Page 69 and 70:
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
Page 77 and 78: Copyright Cambridge University Pres
Page 127: Copyright Cambridge University Pres
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185 and 186:
Page 187 and 188:
Page 189 and 190:
Page 191 and 192:
Page 193 and 194:
Page 195 and 196:
Page 197 and 198:
Page 199 and 200:
Page 201 and 202:
Page 203 and 204:
Page 205 and 206:
Page 207 and 208:
Page 209 and 210:
Page 211 and 212:
Page 213 and 214:
Page 215 and 216:
Page 217 and 218:
Page 219 and 220:
Page 221 and 222:
Page 223 and 224:
Page 225 and 226:
Page 227 and 228:
Page 229 and 230:
Page 231 and 232:
Page 233 and 234:
Page 235 and 236:
Page 237 and 238:
Page 239 and 240:
Page 241 and 242:
Page 243 and 244:
Page 245 and 246:
Page 247 and 248:
Page 249 and 250:
Page 251 and 252:
Page 253 and 254:
Page 255 and 256:
Page 257 and 258:
Page 259 and 260:
Page 261 and 262:
Page 263 and 264:
Page 265 and 266:
Page 267 and 268:
Page 269 and 270:
Page 271 and 272:
Page 273 and 274:
Page 275 and 276:
Page 277 and 278:
Page 279 and 280:
Page 281 and 282:
Page 283 and 284:
Page 285 and 286:
Page 287 and 288:
Page 289 and 290:
Page 291 and 292:
Page 293 and 294:
Page 295 and 296:
Page 297 and 298:
Page 299 and 300:
Page 301 and 302:
Page 303 and 304:
Page 305 and 306:
Page 307 and 308:
Page 309 and 310:
Page 311 and 312:
Page 313 and 314:
Page 315 and 316:
Page 317 and 318:
Page 319 and 320:
Page 321 and 322:
Page 323 and 324:
Page 325 and 326:
Page 327 and 328:
Page 329 and 330:
Page 331 and 332:
Page 333 and 334:
Page 335 and 336:
Page 337 and 338:
Page 339 and 340:
Page 341 and 342:
Page 343 and 344:
Page 345 and 346:
Page 347 and 348:
Page 349 and 350:
Page 351 and 352:
Page 353 and 354:
Page 355 and 356:
Page 357 and 358:
Page 359 and 360:
Page 361 and 362:
Page 363 and 364:
Page 365 and 366:
Page 367 and 368:
Page 369 and 370:
Page 371 and 372:
Page 373 and 374:
Page 375 and 376:
Page 377 and 378:
Page 379 and 380:
Page 381 and 382:
Page 383 and 384:
Page 385 and 386:
Page 387 and 388:
Page 389 and 390:
Page 391 and 392:
Page 393 and 394:
Page 395 and 396:
Page 397 and 398:
Page 399 and 400:
Page 401 and 402:
Page 403 and 404:
Page 405 and 406:
Page 407 and 408:
Page 409 and 410:
Page 411 and 412:
Page 413 and 414:
Page 415 and 416:
Page 417 and 418:
Page 419 and 420:
Page 421 and 422:
Page 423 and 424:
Page 425 and 426:
Page 427 and 428:
Page 429 and 430:
Page 431 and 432:
Page 433 and 434:
Page 435 and 436:
Page 437 and 438:
Page 439 and 440:
Page 441 and 442:
Page 443 and 444:
Page 445 and 446:
Page 447 and 448:
Page 449 and 450:
Page 451 and 452:
Page 453 and 454:
Page 455 and 456:
Page 457 and 458:
Page 459 and 460:
Page 461 and 462:
Page 463 and 464:
Page 465 and 466:
Page 467 and 468:
Page 469 and 470:
Page 471 and 472:
Page 473 and 474:
Page 475 and 476:
Page 477 and 478:
Page 479 and 480:
Page 481 and 482:
Page 483 and 484:
Page 485 and 486:
Page 487 and 488:
Page 489 and 490:
Page 491 and 492:
Page 493 and 494:
Page 495 and 496:
Page 497 and 498:
Page 499 and 500:
Page 501 and 502:
Page 503 and 504:
Page 505 and 506:
Page 507 and 508:
Page 509 and 510:
Page 511 and 512:
Page 513 and 514:
Page 515 and 516:
Page 517 and 518:
Page 519 and 520:
Page 521 and 522:
Page 523 and 524:
Page 525 and 526:
Page 527 and 528:
Page 529 and 530:
Page 531 and 532:
Page 533 and 534:
Page 535 and 536:
Page 537 and 538:
Page 539 and 540:
Page 541 and 542:
Page 543 and 544:
Page 545 and 546:
Page 547 and 548:
Page 549 and 550:
Page 551 and 552:
Page 553 and 554:
Page 555 and 556:
Page 557 and 558:
Page 559 and 560:
Page 561 and 562:
Page 563 and 564:
Page 565 and 566:
Page 567 and 568:
Page 569 and 570:
Page 571 and 572:
Page 573 and 574:
Page 575 and 576:
Page 577 and 578:
Page 579 and 580:
Page 581 and 582:
Page 583 and 584:
Page 585 and 586:
Page 587 and 588:
Page 589 and 590:
Page 591 and 592:
Page 593 and 594:
Page 595 and 596:
Page 597 and 598:
Page 599 and 600:
Page 601 and 602:
Page 603 and 604:
Page 605 and 606:
Page 607 and 608:
Page 609 and 610:
Page 611 and 612:
Page 613 and 614:
Page 615 and 616:
Page 617 and 618:
Page 619 and 620:
Page 621 and 622:
Page 623 and 624:
Page 625 and 626:
Page 627 and 628:
Page 629 and 630:
Page 631 and 632:
Page 633 and 634:
Page 635 and 636:
Page 637 and 638:
Page 639 and 640:
show all

Information Theory, Inference, and Learning ... - MAELabs UCSD

Create successful ePaper yourself

Delete template?

Save as template?