NEURAL NETWORKS
CONTENTS INTRODUCTION I ARTIFICIAL NEURAL NETWORKS (ANNS) 1 BASICS ON ANNS 2 ANN CALCULUS II APPROXIMATION 3 ONE-DIMENSIONAL ANN APPROXIMATION RESULTS 4 MULTI-DIMENSIONAL ANN APPROXIMATION RESULTS III OPTIMIZATION 5 OPTIMIZATION THROUGH GRADIENT FLOW (GF) TRAJECTORIES 6 DETERMINISTIC GRADIENT DESCENT (GD) OPTIMIZATION METHODS 7 STOCHASTIC GRADIENT DESCENT (SGD) OPTIMIZATION METHODS 8 BACKPROPAGATION 9 KURDYKA–?OJASIEWICZ (KL) INEQUALITIES 10 ANNS WITH BATCH NORMALIZATION 11 OPTIMIZATION THROUGH RANDOM INITIALIZATIONS IV GENERALIZATION 12 PROBABILISTIC GENERALIZATION ERROR ESTIMATES 13 STRONG GENERALIZATION ERROR ESTIMATES V COMPOSED ERROR ANALYSIS 14 OVERALL ERROR DECOMPOSITION 15 COMPOSED ERROR ESTIMATES 16 PHYSICS-INFORMED NEURAL NETWORKS (PINNS) VI DEEP LEARNING FOR PARTIAL DIFFERENTIAL EQUATIONS (PDES) 17 DEEP KOLMOGOROV METHODS (DKMS) 18 FURTHER DEEP LEARNING METHODS FOR PDES INTRODUCTION VERY ROUGHLY SPEAKING, THE FIELD DEEP LEARNING CAN BE DIVIDED INTO THREE SUBFIELDS, DEEP SUPERVISED LEARNING, DEEP UNSUPERVISED LEARNING, AND DEEP REINFORCEMENT LEARNING ALGORITHMS IN DEEP SUPERVISED LEARNING OFTEN SEEM TO BE MOST ACCESSIBLE FOR A MATHEMATICAL ANALYSIS IN THE FOLLOWING WE BRIEFLY SKETCH IN A SIMPLIFIED SITUATION SOME IDEAS OF DEEP SUPERVISED LEARNING LET D, M ?N = {1, 2, 3, }, E ?C(RD, R), X1, X2, , XM+1 ?RD, Y1, Y2, , YM ?R SATISFY FOR ALL M ?{1, 2, , M} THAT YM = E(XM) (1) IN THE FRAMEWORK DESCRIBED IN THE PREVIOUS SENTENCE WE THINK OF M ?N AS THE NUMBER OF AVAILABLE KNOWN INPUT-OUTPUT DATA PAIRS, WE THINK OF D ?N AS THE DIMENSION OF THE INPUT DATA, WE THINK OF E : RD ?R AS AN UNKNOWN FUNCTION WHICH RELATES INPUT AND OUTPUT DATA THROUGH (1), WE THINK OF X1, X2, , XM+1 ?RD AS THE AVAILABLE KNOWN INPUT DATA, AND WE THINK OF Y1, Y2, , YM ?R AS THE AVAILABLE KNOWN OUTPUT DATA IN THE CONTEXT OF A LEARNING PROBLEM OF THE TYPE (1) THE OBJECTIVE THEN IS TO APPROXIMATELY COMPUTE THE OUTPUT E(XM+1) OF THE (M + 1)-TH INPUT DATA XM+1 WITHOUT USING EXPLICIT KNOWLEDGE OF THE FUNCTION E : RD ?R BUT INSTEAD BY USING THE KNOWLEDGE OF THE M INPUT-OUTPUT DATA PAIRS (X1, Y1) = (X1, E(X1)), (X2, Y2) = (X2, E(X2)), , (XM, YM) = (XM, E(XM)) ?RD × R (2) TO ACCOMPLISH THIS, ONE CONSIDERS THE OPTIMIZATION PROBLEM OF COMPUTING APPROXIMATE MINIMIZERS OF THE FUNCTION L: C(RD, R) ?[0, ?) WHICH SATISFIES FOR ALL ? ?C(RD, R) THAT L(?) = # (3) OBSERVE THAT (1) ENSURES THAT L(E) = 0 AND, IN PARTICULAR, WE HAVE THAT THE UNKNOWN FUNCTION E : RD ?R IN (1) ABOVE IS A MINIMIZER OF THE FUNCTION L: C(RD [0, ?) (4) ?K+1XK (8) OR ONE COULD THINK OF (7) AS THE PARAMETRIZATION ASSOCIATED TO TRIGONOMETRIC POLYNOMIALS HOWEVER, IN THE CONTEXT OF DEEP SUPERVISED LEARNING ONE NEITHER CHOOSES (7) AS PARAMETRIZATION OF POLYNOMIALS NOR AS PARAMETRIZATION OF TRIGONOMETRIC POLYNOMIALS, BUT INSTEAD ONE CHOOSES (7) AS A PARAMETRIZATION ASSOCIATED TO DEEP ANNS IN CHAPTER 1 IN PART I WE PRESENT DIFFERENT TYPES OF SUCH DEEP ANN PARAMETRIZATION FUNCTIONS IN ALL MATHEMATICAL DETAILS TAKING THE SET IN (6) AND ITS PARAMETRIZATION FUNCTION IN (7) INTO ACCOUNT, WE THEN INTEND TO COMPUTE APPROXIMATE MINIMIZERS OF THE FUNCTION L RESTRICTED TO THE SET {?? : ? ?RD}, THAT IS, WE CONSIDER THE OPTIMIZATION PROBLEM OF COMPUTING APPROXIMATE MINIMIZERS OF THE FUNCTION ?? : ? ?RD ?? 7?L(?) = # ?[0, ?) (9) EMPLOYING THE PARAMETRIZATION FUNCTION IN (7), ONE CAN ALSO REFORMULATE THE OPTIMIZATION PROBLEM IN (9) AS THE OPTIMIZATION PROBLEM OF COMPUTING APPROXIMATE MINIMIZERS OF THE FUNCTION RD ?? 7?L(?) = L(??) = # ?[0, ?) (10)