ELEMENTS OF DISTRIBUTION THEORY
CONTENTS 1 PROPERTIES OF PROBABILITY DISTRIBUTIONS INTRODUCTION BASIC FRAMEWORK RANDOM VARIABLES DISTRIBUTION FUNCTIONS QUANTILE FUNCTIONS DENSITY AND FREQUENCY FUNCTIONS INTEGRATION WITH RESPECT TO A DISTRIBUTION FUNCTION EXPECTATION EXERCISES 1 10 SUGGESTIONS FOR FURTHER READING 2 CONDITIONAL DISTRIBUTIONS AND EXPECTATION MARGINAL DISTRIBUTIONS AND INDEPENDENCE CONDITIONAL DISTRIBUTIONS CONDITIONAL EXPECTATION EXCHANGEABILITY MARTINGALES SUGGESTIONS FOR FURTHER READING 3 CHARACTERISTIC FUNCTIONS BASIC PROPERTIES FURTHER PROPERTIES OF CHARACTERISTIC FUNCTIONS 4 MOMENTS AND CUMULANTS MOMENTS AND CENTRAL MOMENTS LAPLACE TRANSFORMS AND MOMENT-GENERATING FUNCTIONS CUMULANTS MOMENTS AND CUMULANTS OF THE SAMPLE MEAN CONDITIONAL MOMENTS AND CUMULANTS 5 PARAMETRIC FAMILIES OF DISTRIBUTIONS PARAMETERS AND IDENTI?ABILITY EXPONENTIAL FAMILY MODELS HIERARCHICAL MODELS REGRESSION MODELS MODELS WITH A GROUP STRUCTURE 6 STOCHASTIC PROCESSES DISCRETE-TIME STATIONARY PROCESSES MOVING AVERAGE PROCESSES MARKOV PROCESSES COUNTING PROCESSES WIENER PROCESSES 7 DISTRIBUTION THEORY FOR FUNCTIONS OF RANDOM VARIABLES FUNCTIONS OF A REAL-VALUED RANDOM VARIABLE FUNCTIONS OF A RANDOM VECTOR SUMS OF RANDOM VARIABLES ORDER STATISTICS RANKS MONTE CARLO METHODS 8 NORMAL DISTRIBUTION THEORY MULTIVARIATE NORMAL DISTRIBUTION QUADRATIC FORMS SAMPLING DISTRIBUTIONS 9 APPROXIMATION OF INTEGRALS SOME USEFUL FUNCTIONS ASYMPTOTIC EXPANSIONS CONTENTS 9 4 WATSON’S LEMMA 9 5 LAPLACE’S METHOD 9 6 UNIFORM ASYMPTOTIC APPROXIMATIONS 9 7 APPROXIMATION OF SUMS 9 8 EXERCISES 9 9 SUGGESTIONS FOR FURTHER READING 10 ORTHOGONAL POLYNOMIALS 10 1 INTRODUCTION 10 2 GENERAL SYSTEMS OF ORTHOGONAL POLYNOMIALS 10 3 CLASSICAL ORTHOGONAL POLYNOMIALS 10 4 GAUSSIAN QUADRATURE 10 5 EXERCISES 10 6 SUGGESTIONS FOR FURTHER READING 11 APPROXIMATION OF PROBABILITY DISTRIBUTIONS 11 1 INTRODUCTION 11 2 BASIC PROPERTIES OF CONVERGENCE IN DISTRIBUTION 11 3 CONVERGENCE IN PROBABILITY 11 4 CONVERGENCE IN DISTRIBUTION OF FUNCTIONS OF RANDOM VECTORS 11 5 CONVERGENCE OF EXPECTED VALUES 11 6 OP AND OP NOTATION 11 7 EXERCISES 11 8 SUGGESTIONS FOR FURTHER READING 12 CENTRAL LIMIT THEOREMS 12 1 INTRODUCTION 12 2 INDEPENDENT, IDENTICALLY DISTRIBUTED RANDOM VARIABLES 12 3 TRIANGULAR ARRAYS 12 4 RANDOM VECTORS 12 5 RANDOM VARIABLES WITH A PARAMETRIC DISTRIBUTION 12 6 DEPENDENT RANDOM VARIABLES 12 7 EXERCISES 12 8 SUGGESTIONS FOR FURTHER READING 13 APPROXIMATIONS TO THE DISTRIBUTIONS OF MORE GENERAL STATISTICS 13 1 INTRODUCTION 13 2 NONLINEAR FUNCTIONS OF SAMPLE MEANS 13 3 ORDER STATISTICS 13 4 U-STATISTICS 13 5 RANK STATISTICS 13 6 EXERCISES 13 7 SUGGESTIONS FOR FURTHER READING 14 HIGHER-ORDER ASYMPTOTIC APPROXIMATIONS 14 1 INTRODUCTION 14 2 EDGEWORTH SERIES APPROXIMATIONS CONTENTS 14 3 SADDLEPOINT APPROXIMATIONS 14 4 STOCHASTIC ASYMPTOTIC EXPANSIONS 14 5 APPROXIMATION OF MOMENTS 14 6 EXERCISES 14 7 SUGGESTIONS FOR FURTHER READING BASIC FRAMEWORK THE STARTING POINT FOR PROBABILITY THEORY AND, HENCE, DISTRIBUTION THEORY IS THE CONCEPT OF AN EXPERIMENT THE TERM EXPERIMENT MAY ACTUALLY REFER TO A PHYSICAL EXPERIMENT IN THE USUAL SENSE, BUT MORE GENERALLY WE WILL REFER TO SOMETHING AS AN EXPERIMENT WHEN IT HAS THE FOLLOWING PROPERTIES: THERE IS A WELL-DE?NED SET OF POSSIBLE OUTCOMES OF THE EXPERIMENT, EACH TIME THE EXPERIMENT IS PERFORMED EXACTLY ONE OF THE POSSIBLE OUTCOMES OCCURS, AND THE OUTCOME THAT OCCURS IS GOVERNED BY SOME CHANCE MECHANISM LET DENOTE THE SAMPLE SPACE OF THE EXPERIMENT, THE SET OF POSSIBLE OUTCOMES OF THE EXPERIMENT; A SUBSET A OF IS CALLED AN EVENT ASSOCIATED WITH EACH EVENT A IS A PROBABILITY P(A) HENCE, P IS A FUNCTION DE?NED ON SUBSETS OF AND TAKING VALUES IN THE INTERVAL [0, 1] THE FUNCTION P IS REQUIRED TO HAVE CERTAIN PROPERTIES: (P1) P() = (P2) IF A AND B ARE DISJOINT SUBSETS OF , THEN P(A ?B) = P(A) + P(B) AC N = LIM N??AC N RANDOM VARIABLES LET ? DENOTE THE OUTCOME OF AN EXPERIMENT; THAT IS, LET ? DENOTE AN ELEMENT OF IN MANY APPLICATIONS WE ARE CONCERNED PRIMARILY WITH CERTAIN NUMERICAL CHARACTERISTICS OF ?, RATHER THAN WITH ? ITSELF LET X : ?X , WHERE X IS A SUBSET OF RD FOR SOME D = 1, 2, , DENOTE A RANDOM VARIABLE; THE SET X IS CALLED THE RANGE OF X OR, SOMETIMES, THE SAMPLE SPACE OF X FOR A GIVEN OUTCOME ? ?, THE CORRESPONDING VALUE OF X IS X = X(?) PROBABILITIES REGARDING X MAY BE OBTAINED FROM THE PROBABILITY FUNCTION P FOR THE ORIGINAL EXPERIMENT LET PX DENOTE A FUNCTION SUCH THAT FOR ANY SET A ?X , PX(A) DENOTES THE PROBABILITY THAT X ?A THEN PX IS A PROBABILITY FUNCTION DE?NED ON SUBSETS OF X AND PX(A) = P({? ?: X(?) ?A}) WEWILLGENERALLYUSEALESSFORMALNOTATIONINWHICHPR(X ?A)DENOTESPX(A) FORINSTANCE, THE PROBABILITY THAT X ?1 MAY BE WRITTEN AS EITHER PR(X ?1) OR PX{(??, 1]} IN THIS BOOK, WE WILL GENERALLY FOCUS ON PROBABILITIES ASSOCIATED WITH RANDOM VARIABLES, WITHOUT EXPLICIT REFERENCE TO THE UNDERLYING EXPERIMENTS AND ASSOCIATED PROBABILITY FUNCTIONS NOTE THAT SINCE PX DE?NES A PROBABILITY FUNCTION ON THE SUBSETS OF X , IT MUST SATISFY CONDITIONS (P1)–(P3) ALSO, THE ISSUES REGARDING MEASURABILITY DISCUSSED IN THE PREVIOUS SECTION APPLY HERE AS WELL WHEN THE RANGE X OF A RANDOM VARIABLE X IS A SUBSET OF RD FOR SOME D = 1, 2, , IT IS OFTEN CONVENIENT TO PROCEED AS IF PROBABILITY FUNCTION PX IS DE?NED ON THE ENTIRE SPACE RD THEN THE PROBABILITY OF ANY SUBSET OF X C IS 0 AND, FOR ANY SET A ?RD, PX(A) ?PR(X ?A) = PR(X ?A ?X ) IT IS WORTH NOTING THAT SOME AUTHORS DISTINGUISH BETWEEN RANDOM VARIABLES AND RANDOM VECTORS, THE LATTER TERM REFERRING TO RANDOM VARIABLES X FOR WHICH X IS A SUBSET OF RD FOR D > 1 HERE WE WILL NOT MAKE THIS DISTINCTION THE TERM RANDOM VARIABLE WILL REFER TO EITHER A SCALAR OR VECTOR; IN THOSE CASES IN WHICH IT IS IMPORTANT TO DISTINGUISH BETWEEN REAL-VALUED AND VECTOR RANDOM VARIABLES, THE TERMS REAL-VALUED RANDOM VARIABLE AND SCALAR RANDOM VARIABLE WILL BE USED TO DENOTE A RANDOM VARIABLE WITH X ?R AND THE TERM VECTOR RANDOM VARIABLE AND RANDOM VECTOR WILL BE USED TO DENOTE A RANDOM VARIABLE WITH X ?RD, D > 1 RANDOM VECTORS WILL ALWAYS BE TAKEN TO BE COLUMN VECTORS SO THAT A D-DIMENSIONAL RANDOM THE MINIMAL SUPPORT OF THE DISTRIBUTION IS THE SMALLEST CLOSED SET X0 ?RD SUCH THAT PR(X ?X0) = 1 THAT IS, THE MINIMAL SUPPORT OF X IS A CLOSED SET X0 THAT IS A SUPPORT OF X, AND IF X1 IS ANOTHER CLOSED SET THAT IS A SUPPORT OF X, THEN X0 ?X1 THE DISTRIBUTION OF A REAL-VALUED RANDOM VARIABLE X IS SAID TO BE DEGENERATE IF THERE EXISTS A CONSTANT C SUCH THAT PR(X = C) = 1 FOR A RANDOM VECTOR X, WITH DIMENSION GREATER THAN 1, THE DISTRIBUTION OF X IS SAID TO BE DEGENERATE IF THERE EXISTS A VECTOR A ?= 0, WITH THE SAME DIMENSION AS X, SUCH THAT AT X IS EQUAL TO A CONSTANT WITH PROBABILITY 1 FOR EXAMPLE, A TWO-DIMENSIONAL RANDOM VECTOR X = (X1, X2) HAS A DEGENERATE DISTRIBUTION IF, AS IN THE CASE OF A REAL-VALUED RANDOM VARIABLE, IT IS EQUAL TO A CONSTANT WITH PROBABILITY 1 HOWEVER, IT ALSO HAS A DEGENERATE DISTRIBUTION IF PR(A1X1 + A2X2 = C) = X J, J THEN PR((2, ?1)T Y = 0) = 1 HENCE, Y HAS A DEGENERATE DISTRIBUTION ? DISTRIBUTION FUNCTIONS CONSIDER A REAL-VALUED RANDOM VARIABLE X THE PROPERTIES OF X ARE DESCRIBED BY ITS PROBA- BILITY FUNCTION PX, WHICH GIVES THE PROBABILITY THAT X ?A FOR ANY SET A ?R HOWEVER, IT IS ALSO POSSIBLE TO SPECIFY THE DISTRIBUTION OF A RANDOM VARIABLE BY CONSIDERING PR(X ?A) FOR A LIMITED CLASS OF SETS A; THIS APPROACH HAS THE ADVANTAGE THAT THE FUNCTION GIVING SUCH PROBA- BILITIES MAY BE EASIER TO USE IN COMPUTATIONS FOR INSTANCE, CONSIDER SETS OF THE FORM (??, X], FOR X ?R, SO THAT PX{(??, X]} GIVES PR(X ?X) THE DISTRIBUTION FUNCTION OF THE DISTRIBU- TION OF X OR, SIMPLY, THE DISTRIBUTION FUNCTION OF X, IS THE FUNCTION F ?FX : R ?[0, 1] GIVEN BY F(X) = PR(X ?X), ??< X < ? EXAMPLE 1 9 (UNIFORM DISTRIBUTION) SUPPOSE THAT X IS A REAL-VALUED RANDOM VARIABLE SUCH THAT PR(X ?A) = A?(0,1) DX, A ?R; X IS SAID TO HAVE A UNIFORM DISTRIBUTION ON (0, 1) THE DISTRIBUTION FUNCTION OF THIS DISTRIBUTION IS GIVEN BY F(X) = PR{X ?(??, X]} = (??,X]?(0,1) DX = IF X > X F (X) ? ? FIGURE 1 2 DISTRIBUTION FUNCTION IN EXAMPLE 1 10 NOTE THAT WHEN GIVING THE FORM OF A DISTRIBUTION FUNCTION, IT IS CONVENIENT TO ONLY GIVE THE VALUE OF THE FUNCTION IN THE RANGE OF X FOR WHICH F(X) VARIES BETWEEN 0 AND 1 FOR INSTANCE, IN THE PREVIOUS EXAMPLE, WE MIGHT SAY THAT F(X) = X, 0 < X < 1; IN THIS CASE IT IS UNDERSTOOD THAT F(X) = 0 FOR X ?0 AND F(X) = 1 FOR X ?1 EXAMPLE 1 10 (BINOMIAL DISTRIBUTION) LET X DENOTE A RANDOM VARIABLE WITH A BINOMIAL DISTRIBUTION WITH PARAMETERS N AND ?, AS DESCRIBED IN EXAMPLE 1 4 THEN PR(X = X) = N X ? X(1 ??)N?X, X = 0, 1, , N AND, HENCE, THE DISTRIBUTION FUNCTION OF X IS F(X) = J=0,1,; J?X N J ? J(1 ??)N?J THUS, F IS A STEP FUNCTION, WITH JUMPS AT 0, 1, 2, , N; FIGURE 1 2 GIVES A PLOT OF F FOR THE CASE N = 2, ? = 1/4 ? CLEARLY, THERE ARE SOME BASIC PROPERTIES WHICH ANY DISTRIBUTION FUNCTION F MUST POSSESS FOR INSTANCE, AS NOTED ABOVE, F MUST TAKE VALUES IN [0, 1]; ALSO, F MUST BE NONDECREASING THE PROPERTIES OF A DISTRIBUTION FUNCTION ARE SUMMARIZED IN THE FOLLOWING THEOREM THEOREM 1 2 A DISTRIBUTION FUNCTION F OF A DISTRIBUTION ON R HAS THE FOLLOWING PROPERTIES: (DF1) LIMX??F(X) = 1; LIMX???F(X) =