PROBABILITY, RANDOM VARIABLES, & RANDOM PROCESSES
CONTENTS PART I PROBABILITY 1 1 BASIC CONCEPTS OF PROBABILITY THEORY 3 1 1 STATISTICAL REGULARITY AND RELATIVE FREQUENCY 3 1 2 SET THEORY AND ITS APPLICATIONS TO PROBABILITY 5 1 3 THE AXIOMS AND COROLLARIES OF PROBABILITY 12 1 4 JOINT PROBABILITY AND CONDITIONAL PROBABILITY 18 1 5 STATISTICALLY INDEPENDENT EVENTS AND MUTUALLY EXCLUSIVE EVENTS 21 1 6 LAW OF TOTAL PROBABILITY AND BAYES THEOREM 28 1 7 SUMMARY 32 PROBLEMS 32 2 APPLICATIONS IN PROBABILITY 37 2 1 ODDS AND RISK 37 2 2 GAMBLER S RUIN PROBLEM 41 2 3 SYSTEMS RELIABILITY 43 2 4 MEDICAL DIAGNOSTIC TESTING 47 2 5 BAYESIAN SPAM FILTERING 50 2 6 MONTY HALL PROBLEM 51 2 7 DIGITAL TRANSMISSION ERROR 54 2 8 HOW TO MAKE THE BEST CHOICE PROBLEM 56 2 9 THE VITERBI ALGORITHM 59 2 10 ALL EGGS IN ONE BASKET 61 2 11 SUMMARY 63 PROBLEMS 63 3 COUNTING METHODS AND APPLICATIONS 67 3 1 BASIC RULES OF COUNTING 67 3 2 PERMUTATIONS AND COMBINATIONS 72 3 2 1 PERMUTATIONS WITHOUT REPLACEMENT 73 3 2 2 COMBINATIONS WITHOUT REPLACEMENT 73 3 2 3 PERMUTATIONS WITH REPLACEMENT 74 3 2 4 COMBINATIONS WITH REPLACEMENT 74 3 3 MULTINOMIAL COUNTING 77 3 4 SPECIAL ARRANGEMENTS AND SELECTIONS 79 3 5 APPLICATIONS 81 3 5 1 GAME OF POKER 81 3 5 2 BIRTHDAY PARADOX 83 3 5 3 QUALITY CONTROL 86 3 5 4 BEST OF SEVEN CHAMPIONSHIP SERIES 86 3 5 5 LOTTERY 89 3 6 SUMMARY 90 PROBLEMS 90 PART II RANDOM VARIABLES 95 4 ONE RANDOM VARIABLE FUNDAMENTALS 97 4 1 TYPES OF RANDOM VARIABLES 97 4 2 THE CUMULATIVE DISTRIBUTION FUNCTION 99 4 3 THE PROBABILITY MASS FUNCTION 102 4 4 THE PROBABILITY DENSITY FUNCTION 104 4 5 EXPECTED VALUES 107 4 5 1 MEAN OF A RANDOM VARIABLE 107 4 5 2 VARIANCE OF A RANDOM VARIABLE 110 4 5 3 MOMENTS OF A RANDOM VARIABLE 113 4 5 4 MODE AND MEDIAN OF A RANDOM VARIABLE 114 4 6 CONDITIONAL DISTRIBUTIONS 116 4 7 FUNCTIONS OF A RANDOM VARIABLE 120 4 7 1 PDF OF A FUNCTION OF A CONTINUOUS RANDOM VARIABLE 121 4 7 2 PMF OF A FUNCTION OF A DISCRETE RANDOM VARIABLE 123 4 7 3 COMPUTER GENERATION OF RANDOM VARIABLES 124 4 8 TRANSFORM METHODS 125 4 8 1 MOMENT GENERATING FUNCTION OF A RANDOM VARIABLE 125 4 8 2 CHARACTERISTIC FUNCTION OF A RANDOM VARIABLE 126 4 9 UPPER BOUNDS ON PROBABILITY 127 4 9 1 MARKOV BOUND 127 4 9 2 CHEBYSHEV BOUND 128 4 9 3 CHERNO BOUND 128 4 10 SUMMARY 131 PROBLEMS 131 5 SPECIAL PROBABILITY DISTRIBUTIONS AND APPLICATIONS 137 5 1 SPECIAL DISCRETE RANDOM VARIABLES 137 5 1 1 THE BERNOULLI DISTRIBUTION 137 5 1 2 THE BINOMIAL DISTRIBUTION 138 5 1 3 THE GEOMETRIC DISTRIBUTION 140 5 1 4 THE PASCAL DISTRIBUTION 142 5 1 5 THE HYPERGEOMETRIC DISTRIBUTION 143 5 1 6 THE POISSON DISTRIBUTION 144 5 1 7 THE DISCRETE UNIFORM DISTRIBUTION 146 5 1 8 THE ZIPF ZETA DISTRIBUTION 147 5 2 SPECIAL CONTINUOUS RANDOM VARIABLES 148 5 2 1 THE CONTINUOUS UNIFORM DISTRIBUTION 148 5 2 2 THE EXPONENTIAL DISTRIBUTION 149 5 2 3 THE GAMMA DISTRIBUTION 151 5 2 4 THE ERLANG DISTRIBUTION 152 5 2 5 THE WEIBULL DISTRIBUTION 152 5 2 6 THE BETA DISTRIBUTION 153 5 2 7 THE LAPLACE DISTRIBUTION 154 5 2 8 THE PARETO DISTRIBUTION 155 5 3 APPLICATIONS 156 5 3 1 DIGITAL TRANSMISSION REGENERATIVE REPEATERS 156 5 3 2 SYSTEM RELIABILITY FAILURE RATE 157 5 3 3 QUEUING THEORY SERVICING CUSTOMERS 158 5 3 4 RANDOM ACCESS SLOTTED ALOHA 159 5 3 5 ANALOG TO DIGITAL CONVERSION QUANTIZATION 160 5 4 SUMMARY 161 PROBLEMS 162 6 MULTIPLE RANDOM VARIABLES 165 6 1 PAIRS OF RANDOM VARIABLES 165 6 2 THE JOINT CUMULATIVE DISTRIBUTION FUNCTION OF TWO RANDOM VARIABLES 167 6 2 1 MARGINAL CUMULATIVE DISTRIBUTION FUNCTION 169 6 3 THE JOINT PROBABILITY MASS FUNCTION OF TWO RANDOM VARIABLES 170 6 3 1 MARGINAL PROBABILITY MASS FUNCTION 170 6 4 THE JOINT PROBABILITY DENSITY FUNCTION OF TWO RANDOM VARIABLES 171 6 4 1 MARGINAL PROBABILITY DENSITY FUNCTION 172 6 5 EXPECTED VALUES OF FUNCTIONS OF TWO RANDOM VARIABLES 173 6 5 1 JOINT MOMENTS 174 6 6 INDEPENDENCE OF TWO RANDOM VARIABLES 175 6 7 CORRELATION BETWEEN TWO RANDOM VARIABLES 178 6 8 CONDITIONAL DISTRIBUTIONS 185 6 8 1 CONDITIONAL EXPECTATIONS 186 6 9 DISTRIBUTIONS OF FUNCTIONS OF TWO RANDOM VARIABLES 188 6 9 1 JOINT DISTRIBUTION OF TWO FUNCTIONS OF TWO RANDOM VARIABLES 191 6 10 RANDOM VECTORS 192 6 11 SUMMARY 197 PROBLEMS 198 7 THE GAUSSIAN DISTRIBUTION 201 7 1 THE GAUSSIAN RANDOM VARIABLE 201 7 2 THE STANDARD GAUSSIAN DISTRIBUTION 204 7 3 BIVARIATE GAUSSIAN RANDOM VARIABLES 210 7 3 1 LINEAR TRANSFORMATIONS OF BIVARIATE GAUSSIAN RANDOM VARIABLES 213 7 4 JOINTLY GAUSSIAN RANDOM VECTORS 215 7 5 SUMS OF RANDOM VARIABLES 217 7 5 1 MEAN AND VARIANCE OF SUM OF RANDOM VARIABLES 217 7 5 2 MEAN AND VARIANCE OF SUM OF INDEPENDENT IDENTICALLY DISTRIBUTED RANDOM VARIABLES 218 7 5 3 DISTRIBUTION OF SUM OF INDEPENDENT RANDOM VARIABLES 218 7 5 4 SUM OF A RANDOM NUMBER OF INDEPENDENT IDENTICALLY DISTRIBUTED RANDOM VARIABLES 219 7 6 THE SAMPLE MEAN 220 7 6 1 LAWS OF LARGE NUMBERS 222 7 7 APPROXIMATING DISTRIBUTIONS WITH THE GAUSSIAN DISTRIBUTION 223 7 7 1 RELATION BETWEEN THE GAUSSIAN AND BINOMIAL DISTRIBUTIONS 223 7 7 2 RELATION BETWEEN THE GAUSSIAN AND POISSON DISTRIBUTIONS 225 7 7 3 THE CENTRAL LIMIT THEOREM 226 7 8 PROBABILITY DISTRIBUTIONS RELATED TO THE GAUSSIAN DISTRIBUTION 230 7 8 1 THE RAYLEIGH DISTRIBUTION 230 7 8 2 THE RICEAN DISTRIBUTION 231 7 8 3 THE LOG NORMAL DISTRIBUTION 231 7 8 4 THE CHI SQUARE DISTRIBUTION 232 7 8 5 THE MAXWELL BOLTZMANN DISTRIBUTION 232 7 8 6 THE STUDENT S T DISTRIBUTION 233 7 8 7 THE F DISTRIBUTION 234 7 8 8 THE CAUCHY DISTRIBUTION 234 7 9 SUMMARY 234 PROBLEMS 235 PART III STATISTICS 239 8 DESCRIPTIVE STATISTICS 241 8 1 OVERVIEW OF STATISTICS 241 8 2 DATA DISPLAYS 244 8 3 MEASURES OF LOCATION 249 8 4 MEASURES OF DISPERSION 250 8 5 MEASURES OF SHAPE 255 8 6 SUMMARY 257 PROBLEMS 257 9 ESTIMATION 259 9 1 PARAMETER ESTIMATION 259 9 2 PROPERTIES OF POINT ESTIMATORS 260 9 3 MAXIMUM LIKELIHOOD ESTIMATORS 265 9 4 BAYESIAN ESTIMATORS 270 9 5 CON DENCE INTERVALS 272 9 6 ESTIMATION OF A RANDOM VARIABLE 274 9 7 MAXIMUM A POSTERIORI PROBABILITY ESTIMATION 275 9 8 MINIMUM MEAN SQUARE ERROR ESTIMATION 277 9 9 LINEAR MINIMUM MEAN SQUARE ERROR ESTIMATION 279 9 10 LINEAR MMSE ESTIMATION USING A VECTOR OF OBSERVATIONS 282 9 11 SUMMARY 285 PROBLEMS 285 10 HYPOTHESIS TESTING 287 10 1 SIGNI CANCE TESTING 287 10 2 HYPOTHESIS TESTING FOR MEAN 291 10 2 1 P VALUE 294 10 3 DECISION TESTS 300 10 4 BAYESIAN TEST 303 10 4 1 MINIMUM COST TEST 304 10 4 2 MAXIMUM A POSTERIORI PROBABILITY MAP TEST 305 10 4 3 MAXIMUM LIKELIHOOD ML TEST 305 10 4 4 MINIMAX TEST 307 10 5 NEYMAN PEARSON TEST 307 10 6 SUMMARY 309 PROBLEMS 309 PART IV RANDOM PROCESSES 311 11 INTRODUCTION TO RANDOM PROCESSES 313 11 1 CLASSI CATION OF RANDOM PROCESSES 313 11 1 1 STATE SPACE 314 11 1 2 INDEX TIME PARAMETER 314 11 2 CHARACTERIZATION OF RANDOM PROCESSES 318 11 2 1 JOINT DISTRIBUTIONS OF TIME SAMPLES 318 11 2 2 INDEPENDENT IDENTICALLY DISTRIBUTED RANDOM PROCESS 319