INTERNETWORKING TECHNOLOGIES HANDBOOK
1 INTRODUCTION TO NEURAL NETWORKS FOR SIGNAL PROCESSING 1 2 SIGNAL PROCESSING USING THE MULTILAYER PERCEPTRON 31 3 RADIAL BASIS FUNCTIONS 60 4 AN INTRODUCTION TO KERNEL BASED LEARNING ALGORITHMS 83 5 COMMITTEE MACHINES 123 6 DYNAMIC NEURAL NETWORKS AND OPTIMAL SIGNAL PROCESSING 141 7 BLIND SIGNAL SEPARATION AND BLIND DECONVOLUTION 169 8 NEURAL NETWORKS AND PRINCIPAL COMPONENT ANALYSIS 200 9 APPLICATIONS OF ARTIFICIAL NEURAL NETWORKS TO TIME SERIES PREDICTION 238 10 APPLICATIONS OF ARTIFICIAL NEURAL NETWORKS ANNS TO SPEECH PROCESSING 256 11 LEARNING AND ADAPTIVE CHARACTERIZATION OF VISUAL CONTENTS IN IMAGE RETRIEVAL SYSTEMS 280 12 APPLICATIONS OF NEURAL NETWORKS TO IMAGE PROCESSING 309 CONTENTS 1 INTRODUCTION TO NEURAL NETWORKS FOR SIGNAL PROCESSING 1 1 INTRODUCTION 1 2 ARTI CIAL NEURAL NETWORK ANN MODELS AN OVERVIEW BASIC NEURAL NETWORK COMPONENTS MULTILAYER PERCEPTRON MLP MODEL RADIAL BASIS NETWORKS COMPETITIVE LEARNING NETWORKS COMMITTEE MACHINES SUPPORT VECTOR MACHINES SVMS 1 3 NEURAL NETWORK SOLUTIONS TO SIGNAL PROCESSING PROBLEMS DIGITAL SIGNAL PROCESSING 1 4 OVERVIEW OF THE HANDBOOK REFERENCES 1 1 INTRODUCTION THE THEORY AND DESIGN OF ARTI CIAL NEURAL NETWORKS HAVE ADVANCED SIGNI CANTLY DURING THE PAST 20 YEARS MUCH OF THAT PROGRESS HAS A DIRECT BEARING ON SIGNAL PROCESSING IN PARTICULAR THE NON LINEAR NATURE OF NEURAL NETWORKS THE ABILITY OF NEURAL NETWORKS TO LEARN FROM THEIR ENVIRONMENTS IN SUPERVISED AS WELL AS UNSUPERVISED WAYS AS WELL AS THE UNIVERSAL APPROXIMATION PROPERTY OF NEURAL NETWORKS MAKE THEM HIGHLY SUITED FOR SOLVING DIF CULT SIGNAL PROCESSING PROBLEMS FROM A SIGNAL PROCESSING PERSPECTIVE IT IS IMPERATIVE TO DEVELOP A PROPER UNDERSTANDING OF BASIC NEURAL NETWORK STRUCTURES AND HOW THEY IMPACT SIGNAL PROCESSING ALGORITHMS AND APPLICATIONS A CHALLENGE IN SURVEYING THE ELD OF NEURAL NETWORK PARADIGMS IS TO IDENTIFY THOSE NEURAL NETWORK STRUCTURES THAT HAVE BEEN SUCCESSFULLY APPLIED TO SOLVE REAL WORLD PROBLEMS FROM THOSE THAT ARE STILL UNDER DEVELOPMENT OR HAVE DIF CULTY SCALING UP TO SOLVE REALISTIC PROBLEMS WHEN DEALING WITH SIGNAL PROCESSING APPLICATIONS IT IS CRITICAL TO UNDERSTAND THE NATURE OF THE PROBLEM FORMULATION SO THAT THE MOST APPROPRIATE NEURAL NETWORK PARADIGM CAN BE APPLIED IN ADDITION IT IS ALSO IMPORTANT TO ASSESS THE IMPACT OF NEURAL NETWORKS ON THE PERFORMANCE ROBUSTNESS AND COST EFFECTIVENESS OF SIGNAL PROCESSING SYSTEMS AND DEVELOP METHODOLOGIES FOR INTEGRATING NEURAL NETWORKS WITH OTHER SIGNAL PROCESSING ALGORITHMS ANOTHER IMPORTANT ISSUE IS HOW TO EVALUATE NEURAL NETWORK PARADIGMS LEARNING ALGORITHMS AND NEURAL NETWORK STRUCTURES AND IDENTIFY THOSE THAT DO AND DO NOT WORK RELIABLY FOR SOLVING SIGNAL PROCESSING PROBLEMS THIS CHAPTER PROVIDES AN OVERVIEW OF THE TOPIC OF THIS HANDBOOK NEURAL NETWORKS FOR SIGNAL PROCESSING THE CHAPTER RST DISCUSSES THE DE NITION OF A NEURAL NETWORK FOR SIGNAL PROCESSING AND WHY IT IS IMPORTANT IT THEN SURVEYS SEVERAL MODERN NEURAL NETWORK MODELS THAT HAVE FOUND SUCCESSFUL SIGNAL PROCESSING APPLICATIONS EXAMPLES ARE CITED RELATING TO HOW TO APPLY THESE NONLINEAR 1 COMPUTATION PARADIGMS TO SOLVE SIGNAL PROCESSING PROBLEMS FINALLY THIS CHAPTER HIGHLIGHTS THE REMAINING CONTENTS OF THIS BOOK 1 2 ARTI CIAL NEURAL NETWORK ANN MODELS AN OVERVIEW 1 2 1 BASIC NEURAL NETWORK COMPONENTS A NEURAL NETWORK IS A GENERAL MATHEMATICAL COMPUTING PARADIGM THAT MODELS THE OPERATIONS OF BIO LOGICAL NEURAL SYSTEMS IN 1943 MCCULLOCH A NEUROBIOLOGIST AND PITTS A STATISTICIAN PUBLISHED A SEMINAL PAPER TITLED A LOGICAL CALCULUS OF IDEAS IMMINENT IN NERVOUS ACTIVITY IN BULLETIN OF MATHE MATICAL BIOPHYSICS 1 THIS PAPER INSPIRED THE DEVELOPMENT OF THE MODERN DIGITAL COMPUTER OR THE ELECTRONIC BRAIN AS JOHN VON NEUMANN CALLED IT AT APPROXIMATELY THE SAME TIME FRANK ROSENBLATT WAS ALSO MOTIVATED BY THIS PAPER TO INVESTIGATE THE COMPUTATION OF THE EYE WHICH EVENTUALLY LED TO THE RST GENERATION OF NEURAL NETWORKS KNOWN AS THE PERCEPTRON 2 THIS SECTION PROVIDES A BRIEF OVERVIEW OF ANN MODELS MANY OF THESE TOPICS WILL BE TREATED IN GREATER DETAIL IN LATER CHAPTERS THE PURPOSE OF THIS CHAPTER THEREFORE IS TO HIGHLIGHT THE BASIC CONCEPT OF THESE NEURAL NETWORK MODELS TO PREPARE THE READERS FOR LATER CHAPTERS 1 2 1 1 MCCULLOCH AND PITTS NEURON MODEL AMONG NUMEROUS NEURAL NETWORK MODELS THAT HAVE BEEN PROPOSED OVER THE YEARS ALL SHARE A COMMON BUILDING BLOCK KNOWN AS A NEURON AND A NETWORKED INTERCONNECTION STRUCTURE THE MOST WIDELY USED NEURON MODEL IS BASED ON MCCULLOCH AND PITTS WORK AND IS ILLUSTRATED IN FIGURE 1 1 1 1 MCCULLOCH AND PITTS NEURON MODEL IN FIGURE 1 1 EACH NEURON CONSISTS OF TWO PARTS THE NET FUNCTION AND THE ACTIVATION FUNCTION THE NET FUNCTION DETERMINES HOW THE NETWORK INPUTS YJ 1 J N ARE COMBINED INSIDE THE NEURON IN THIS GURE A WEIGHTED LINEAR COMBINATION IS ADOPTED U N J 1 WJYJ 1 1 WJ 1 J N ARE PARAMETERS KNOWN AS SYNAPTIC WEIGHTS THE QUANTITY IS CALLED THE BIAS OR THRESHOLD AND IS USED TO MODEL THE THRESHOLD IN THE LITERATURE OTHER TYPES OF NETWORK INPUT COMBINATION METHODS HAVE BEEN PROPOSED AS WELL THEY ARE SUMMARIZED IN TABLE 1 1 2 INTERNETWORKING TECHNOLOGIES HANDBOOK TABLE 1 1 SUMMARY OF NET FUNCTIONS NET FUNCTIONS FORMULA COMMENTS LINEAR U N J 1 WJ YJ MOST COMMONLY USED HIGHER ORDER 2ND ORDER FORMULA EXHIBITED U N J 1 N K 1 WJKYJ YK UI IS A WEIGHTED LINEAR COMBINATION OF HIGHER ORDER POLYNOMIAL TERMS OF INPUT VARIABLE THE NUMBER OF INPUT TERMS EQUALS ND WHERE D IS THE ORDER OF THE POLYNOMIAL DELTA U N J 1 WJ YJ SELDOM USED THE OUTPUT OF THE NEURON DENOTED BY AI IN THIS GURE IS RELATED TO THE NETWORK INPUT UI VIA A LINEAR OR NONLINEAR TRANSFORMATION CALLED THE ACTIVATION FUNCTION A F U 1 2 IN VARIOUS NEURAL NETWORK MODELS DIFFERENT ACTIVATION FUNCTIONS HAVE BEEN PROPOSED THE MOST COMMONLY USED ACTIVATION FUNCTIONS ARE SUMMARIZED IN TABLE 1 2 TABLE 1 2 NEURON ACTIVATION FUNCTIONS ACTIVATION FUNCTION FORMULA A F U DERIVATIVES DF U DU COMMENTS SIGMOID F U 1 1 E U T F U 1 F U T COMMONLY USED DERIVATIVE CAN BE COMPUTED FROM F U DIRECTLY HYPERBOLIC TANGENT F U TANH U T 1 F U 2 T T TEMPERATURE PARAMETER INVERSE TANGENT F U 2 TAN 1 U T 2 T 1 1 U T 2 LESS FREQUENTLY USED THRESHOLD F U 1 U 0 1 U 0 DERIVATIVES DO NOT EXIST AT U 0 GAUSSIAN RADIAL BASIS F U EXP U M 2 2 2 U M F U 2 USED FOR RADIAL BASIS NEURAL NETWORK M AND 2 ARE PARAMETERS TO BE SPECI ED LINEAR F U AU B A TABLE 1 2 LISTS BOTH THE ACTIVATION FUNCTIONS AS WELL AS THEIR DERIVATIVES PROVIDED THEY EXIST IN BOTH SIGMOID AND HYPERBOLIC TANGENT ACTIVATION FUNCTIONS DERIVATIVES CAN BE COMPUTED DIRECTLY FROM THE KNOWLEDGE OF F U 1 2 1 2 NEURAL NETWORK TOPOLOGY IN A NEURAL NETWORK MULTIPLE NEURONS ARE INTERCONNECTED TO FORM A NETWORK TO FACILITATE DIS TRIBUTED COMPUTING THE CON GURATION OF THE INTERCONNECTIONS CAN BE DESCRIBED EF CIENTLY WITH A DIRECTED GRAPH A DIRECTED GRAPH CONSISTS OF NODES IN THE CASE OF A NEURAL NETWORK NEURONS AS WELL AS EXTERNAL INPUTS AND DIRECTED ARCS IN THE CASE OF A NEURAL NETWORK SYNAPTIC LINKS THE TOPOLOGY OF THE GRAPH CAN BE CATEGORIZED AS EITHER ACYCLIC OR CYCLIC REFER TO FIGURE 1 2A A NEURAL NETWORK WITH ACYCLIC TOPOLOGY CONSISTS OF NO FEEDBACK LOOPS SUCH AN ACYCLIC NEURAL NETWORK IS OFTEN USED TO APPROXIMATE A NONLINEAR MAPPING BETWEEN ITS INPUTS AND OUTPUTS AS SHOWN IN FIGURE 1 2B A NEURAL NETWORK WITH CYCLIC TOPOLOGY CONTAINS AT LEAST ONE CYCLE FORMED BY DIRECTED ARCS SUCH A NEURAL NETWORK IS ALSO KNOWN AS A RECURRENT NETWORK DUE TO THE FEEDBACK LOOP A RECURRENT NETWORK LEADS TO A NONLINEAR DYNAMIC SYSTEM MODEL THAT CONTAINS INTERNAL MEMORY RECURRENT NEURAL NETWORKS OFTEN EXHIBIT COMPLEX BEHAVIORS AND REMAIN AN ACTIVE RESEARCH TOPIC IN THE ELD OF ARTI CIAL NEURAL NETWORKS 3 INTERNETWORKING TECHNOLOGIES HANDBOOK