Information Theory, Inference, and Learning ... - Inference Group

More documents

Recommendations

Info

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.160 9 — Communication over a Noisy ChannelUsing our skills picked up from exercise 2.17 (p.36), we rewrite this in theform1P (x = 1 | y, α, σ) = . (9.52)1 + e−a(y) The optimal decoder selects the most probable hypothesis; this can be donesimply by looking at the sign of a(y). If a(y) > 0 then decode as ˆx = 1.The probability of error isp b =∫ 0−∞Random codingdy Q(y | x = 1, α, σ) =∫ −xα−∞dyy21√ e− 2σ 22πσ 2(= Φ − xα ). (9.53)σSolution to exercise 9.20 (p.156). The probability that S = 24 people whosebirthdays are drawn at random from A = 365 days all have distinct birthdaysisA(A − 1)(A − 2) . . . (A − S + 1)A S . (9.54)The probability that two (or more) people share a birthday is one minus thisquantity, which, for S = 24 and A = 365, is about 0.5. This exact way ofanswering the question is not very informative since it is not clear for whatvalue of S the probability changes from being close to 0 to being close to 1.The number of pairs is S(S − 1)/2, and the probability that a particularpair shares a birthday is 1/A, so the expected number of collisions isS(S − 1)21A . (9.55)This answer is more instructive. The expected number of collisions is tiny ifS ≪ √ A and big if S ≫ √ A.We can also approximate the probability that all birthdays are distinct,for small S, thus:A(A − 1)(A − 2) . . . (A − S + 1)A S = (1)(1 − 1/ A)(1 − 2/ A) . . . (1 − (S−1)/ A)≃ exp(0) exp(−1/A) exp(−2/A) . . . exp(−(S−1)/A) (9.56)≃( )exp − 1 S−1∑i = expAi=1(−S(S − 1)/2A). (9.57)
Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.About Chapter 10Before reading Chapter 10, you should have read Chapters 4 and 9. Exercise9.14 (p.153) is especially recommended.Cast of charactersQthe noisy channelCthe capacity of the channelX N an ensemble used to create a random codeCa random codeNthe length of the codewordsx (s) a codeword, the sth in the codes the number of a chosen codeword (mnemonic: the sourceselects s)S = 2 K the total number of codewords in the codeK = log 2 S the number of bits conveyed by the choice of one codewordfrom S, assuming it is chosen with uniform probabilitysa binary representation of the number sR = K/N the rate of the code, in bits per channel use (sometimes calledR ′ instead)ŝthe decoder’s guess of s161
Page 1 and 2:
Copyright Cambridge University Pres
Page 3 and 4:
Page 5 and 6:
Page 7 and 8:
Page 9 and 10:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17 and 18:
Page 19 and 20:
Page 21 and 22:
Page 23 and 24:
Page 25 and 26:
Page 27 and 28:
Page 29 and 30:
Page 31 and 32:
Page 33 and 34:
Page 35 and 36:
Page 37 and 38:
Page 39 and 40:
Page 41 and 42:
Page 43 and 44:
Page 45 and 46:
Page 47 and 48:
Page 49 and 50:
Page 51 and 52:
Page 53 and 54:
Page 55 and 56:
Page 57 and 58:
Page 59 and 60:
Page 61 and 62:
Page 63 and 64:
Page 65 and 66:
Page 67 and 68:
Page 69 and 70:
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
Page 77 and 78:
Page 80 and 81:
Page 82 and 83:
Page 84 and 85:
Page 86 and 87:
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
Page 94 and 95:
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Page 102 and 103:
Page 104 and 105:
Page 106 and 107:
Page 108 and 109:
¥¡¥¡¥¡¥¡¥¡¥¡¥¡¥¡¥
Page 110 and 111:
Page 112 and 113:
Page 114 and 115:
Page 116 and 117:
Page 118 and 119:
Page 120 and 121:
Page 122 and 123: Copyright Cambridge University Pres
Page 222 and 223:
Page 224 and 225:
Page 226 and 227:
Page 228 and 229:
Page 230 and 231:
Page 232 and 233:
Page 234 and 235:
Page 236 and 237:
Page 238 and 239:
Page 240 and 241:
Page 242 and 243:
Page 244 and 245:
Page 246 and 247:
Page 248 and 249:
Page 250 and 251:
Page 252 and 253:
Page 254 and 255:
Page 256 and 257:
Page 258 and 259:
Page 260 and 261:
Page 262 and 263:
Page 264 and 265:
Page 266 and 267:
Page 268 and 269:
Page 270 and 271:
Page 272 and 273:
Page 274 and 275:
Page 276 and 277:
Page 278 and 279:
Page 280 and 281:
Page 282 and 283:
Page 284 and 285:
Page 286 and 287:
Page 288 and 289:
Page 290 and 291:
Page 292 and 293:
Page 294 and 295:
Page 296 and 297:
Page 298 and 299:
Page 300 and 301:
Page 302 and 303:
Page 304 and 305:
Page 306 and 307:
Page 308 and 309:
Page 310 and 311:
Page 312 and 313:
Page 314 and 315:
Page 316 and 317:
Page 318 and 319:
Page 320 and 321:
Page 322 and 323:
Page 324 and 325:
Page 326 and 327:
Page 328 and 329:
Page 330 and 331:
Page 332 and 333:
Page 334 and 335:
Page 336 and 337:
Page 338 and 339:
Page 340 and 341:
Page 342 and 343:
Page 344 and 345:
Page 346 and 347:
Page 348 and 349:
Page 350 and 351:
Page 352 and 353:
Page 354 and 355:
Page 356 and 357:
Page 358 and 359:
Page 360 and 361:
Page 362 and 363:
Page 364 and 365:
Page 366 and 367:
Page 368 and 369:
Page 370 and 371:
Page 372 and 373:
Page 374 and 375:
Page 376 and 377:
Page 378 and 379:
Page 380 and 381:
Page 382 and 383:
Page 384 and 385:
Page 386 and 387:
Page 388 and 389:
Page 390 and 391:
Page 392 and 393:
Page 394 and 395:
Page 396 and 397:
Page 398 and 399:
Page 400 and 401:
Page 402 and 403:
Page 404 and 405:
Page 406 and 407:
Page 408 and 409:
Page 410 and 411:
Page 412 and 413:
Page 414 and 415:
Page 416 and 417:
Page 418 and 419:
Page 420 and 421:
Page 422 and 423:
Page 424 and 425:
Page 426 and 427:
Page 428 and 429:
Page 430 and 431:
Page 432 and 433:
Page 434 and 435:
Page 436 and 437:
Page 438 and 439:
Page 440 and 441:
Page 442 and 443:
Page 444 and 445:
Page 446 and 447:
Page 448 and 449:
Page 450 and 451:
Page 452 and 453:
Page 454 and 455:
Page 456 and 457:
Page 458 and 459:
Page 460 and 461:
Page 462 and 463:
Page 464 and 465:
Page 466 and 467:
Page 468 and 469:
Page 470 and 471:
Page 472 and 473:
Page 474 and 475:
Page 476 and 477:
Page 478 and 479:
Page 480 and 481:
Page 482 and 483:
Page 484 and 485:
Page 486 and 487:
Page 488 and 489:
Page 490 and 491:
Page 492 and 493:
Page 494 and 495:
Page 496 and 497:
Page 498 and 499:
Page 500 and 501:
Page 502 and 503:
Page 504 and 505:
Page 506 and 507:
Page 508 and 509:
Page 510 and 511:
Page 512 and 513:
Page 514 and 515:
¡¤¢¢¤¨©¢£¡¢£¨¢£ ©Co
Page 516 and 517:
Page 518 and 519:
Page 520 and 521:
Page 522 and 523:
Page 524 and 525:
Page 526 and 527:
Page 528 and 529:
Page 530 and 531:
Page 532 and 533:
Page 534 and 535:
Page 536 and 537:
Page 538 and 539:
Page 540 and 541:
Page 542 and 543:
Page 544 and 545:
Page 546 and 547:
Page 548 and 549:
Page 550 and 551:
Page 552 and 553:
Page 554 and 555:
ttttCopyright Cambridge University
Page 556 and 557:
Page 558 and 559:
Page 560 and 561:
Page 562 and 563:
Page 564 and 565:
Page 566 and 567:
Page 568 and 569:
Page 570 and 571:
Page 572 and 573:
Page 574 and 575:
Page 576 and 577:
Page 578 and 579:
Page 580 and 581:
Page 582 and 583:
Page 584 and 585:
Page 586 and 587:
Page 588 and 589:
Page 590 and 591:
Page 592 and 593:
Page 594 and 595:
Page 596 and 597:
Page 598 and 599:
Page 600 and 601:
Page 602 and 603:
Page 604 and 605:
Page 606 and 607:
Page 608 and 609:
Page 610 and 611:
Page 612 and 613:
Page 614 and 615:
Page 616 and 617:
Page 618 and 619:
Page 620 and 621:
Page 622 and 623:
Page 624 and 625:
Page 626 and 627:
Page 628 and 629:
Page 630 and 631:
Page 632 and 633:
Page 634 and 635:
Page 636 and 637:
Page 638 and 639:
Page 640:
show all

Information Theory, Inference, and Learning ... - Inference Group

Create successful ePaper yourself

Delete template?

Save as template?