NATURAL LANGUAGE UNDERSTANDING
CONTENTS PART WORDY MACHINES 1 PACKETS OF THOUGHT (NLP OVERVIEW) NATURAL LANGUAGE VS PROGRAMMING LANGUAGE THE MAGIC MACHINES THAT CONVERSE ?THE MATH PRACTICAL APPLICATIONS LANGUAGE THROUGH A COMPUTER’S “EYES” THE LANGUAGE OF LOCKS ?REGULAR EXPRESSIONS A SIMPLE CHATBOT ?ANOTHER WAY A BRIEF OVERFLIGHT OF HYPERSPACE WORD ORDER AND GRAMMAR A CHATBOT NATURAL LANGUAGE PIPELINE PROCESSING IN DEPTH NATURAL LANGUAGE IQ CONTENTS 2 BUILD YOUR VOCABULARY (WORD TOKENIZATION) CHALLENGES (A PREVIEW OF STEMMING) BUILDING YOUR VOCABULARY WITH A TOKENIZER DOT PRODUCT ?MEASURING BAG-OF-WORDS OVERLAP A TOKEN IMPROVEMENT ?EXTENDING YOUR VOCABULARY WITH N-GRAMS ?NORMALIZING YOUR VOCABULARY SENTIMENT VADER—A RULE-BASED SENTIMENT ANALYZER ?NAIVE BAYES 3 MATH WITH WORDS (TF-IDF VECTORS) BAG OF WORDS VECTORIZING VECTOR SPACES ZIPF’S LAW TOPIC MODELING RETURN OF ZIPF ?RELEVANCE RANKING ?TOOLS ALTERNATIVES ?WHAT’S NEXT 4 FINDING MEANING IN WORD COUNTS (SEMANTIC ANALYSIS) FROM WORD COUNTS TO TOPIC SCORES TF-IDF VECTORS AND LEMMATIZATION ?TOPIC VECTORS THOUGHT EXPERIMENT ?AN ALGORITHM FOR SCORING TOPICS AN LDA CLASSIFIER LATENT SEMANTIC ANALYSIS YOUR THOUGHT EXPERIMENT MADE REAL SINGULAR VALUE DECOMPOSITION U—LEFT SINGULAR VECTORS ?S—SINGULAR VALUES VT—RIGHT SINGULAR VECTORS ?SVD MATRIX ORIENTATION TRUNCATING THE TOPICS PRINCIPAL COMPONENT ANALYSIS PCA ON 3D VECTORS ?STOP HORSING AROUND AND GET BACK TO NLP ?USING PCA FOR SMS MESSAGE SEMANTIC ANALYSIS USING TRUNCATED SVD FOR SMS MESSAGE SEMANTIC ANALYSIS HOW WELL DOES LSA WORK FOR SPAM CLASSIFICATION? LATENT DIRICHLET ALLOCATION (LDIA) THE LDIA IDEA ?LDIA TOPIC MODEL FOR SMS MESSAGES LDIA + LDA = SPAM CLASSIFIER ?A FAIRER COMPARISON: 32 LDIA TOPICS DISTANCE AND SIMILARITY STEERING WITH FEEDBACK LINEAR DISCRIMINANT ANALYSIS TOPIC VECTOR POWER SEMANTIC SEARCH ?IMPROVEMENTS PART DEEPER LEARNING (NEURAL NETWORKS) 5 BABY STEPS WITH NEURAL NETWORKS (PERCEPTRONS AND BACKPROPAGATION) NEURAL NETWORKS, THE INGREDIENT LIST PERCEPTRON ?A NUMERICAL PERCEPTRON ?DETOUR THROUGH BIAS ?LET’S GO SKIING—THE ERROR SURFACE OFF THE CHAIR LIFT, ONTO THE SLOPE ?LET’S SHAKE THINGS UP A BIT ?KERAS: NEURAL NETWORKS IN PYTHON ?ONWARD AND DEEPWARD ?NORMALIZATION: INPUT WITH STYLE 6 REASONING WITH WORD VECTORS (WORD2VEC) SEMANTIC QUERIES AND ANALOGIES ANALOGY QUESTIONS WORD VECTORS VECTOR-ORIENTED REASONING ?HOW TO COMPUTE WORD2VEC REPRESENTATIONS ?HOW TO USE THE GENSIM WORD2VEC MODULE ?HOW TO GENERATE YOUR OWN WORD VECTOR REPRESENTATIONS ?WORD2VEC VS GLOVE (GLOBAL VECTORS) FASTTEXT ?WORD2VEC VS LSA ?VISUALIZING WORD RELATIONSHIPS ?UNNATURAL WORDS ?DOCUMENT SIMILARITY WITH DOC2VEC 7 GETTING WORDS IN ORDER WITH CONVOLUTIONAL NEURAL NETWORKS (CNNS) LEARNING MEANING TOOLKIT CONVOLUTIONAL NEURAL NETS BUILDING BLOCKS ?STEP SIZE (STRIDE) ?FILTER COMPOSITION ?PADDING ?LEARNING NARROW WINDOWS INDEED IMPLEMENTATION IN KERAS: PREPPING THE DATA ?CONVOLUTIONAL NEURAL NETWORK ARCHITECTURE ?POOLING DROPOUT ?THE CHERRY ON THE SUNDAE ?LET’S GET TO CONTENTS ASSEMBLE THE MODEL FOR SEQUENCE GENERATION ?PREDICTING A SEQUENCE ?GENERATING A RESPONSE ?CONVERSE WITH YOUR CHATBOT ENHANCEMENTS REDUCE TRAINING COMPLEXITY WITH BUCKETING ?PAYING ATTENTION IN THE REAL WORLD GETTING REAL (REAL-WORLD NLP CHALLENGES) 11 INFORMATION EXTRACTION (NAMED ENTITY EXTRACTION AND QUESTION ANSWERING) NAMED ENTITIES AND RELATIONS A KNOWLEDGE BASE ?INFORMATION EXTRACTION REGULAR PATTERNS REGULAR EXPRESSIONS ?INFORMATION EXTRACTION AS ML FEATURE EXTRACTION INFORMATION WORTH EXTRACTING EXTRACTING GPS LOCATIONS ?EXTRACTING DATES EXTRACTING RELATIONSHIPS (RELATIONS) PART-OF-SPEECH (POS) TAGGING ?ENTITY NAME NORMALIZATION RELATION NORMALIZATION AND EXTRACTION ?WORD PATTERNS SEGMENTATION ?WHY WON’T SPLIT(' !?') WORK? SENTENCE SEGMENTATION WITH REGULAR EXPRESSIONS 12 GETTING CHATTY (DIALOG ENGINES) LANGUAGE SKILL MODERN APPROACHES ?A HYBRID APPROACH PATTERN-MATCHING APPROACH A PATTERN-MATCHING CHATBOT WITH AIML ?A NETWORK VIEW OF PATTERN MATCHING GROUNDING RETRIEVAL (SEARCH) THE CONTEXT CHALLENGE ?EXAMPLE RETRIEVAL-BASED CHATBOT ?A SEARCH-BASED CHATBOT GENERATIVE MODELS CHAT ABOUT NLPIA ?PROS AND CONS OF EACH APPROACH FOUR-WHEEL DRIVE THE WILL TO SUCCEED DESIGN PROCESS TRICKERY ASK QUESTIONS WITH PREDICTABLE ANSWERS ?BE ENTERTAINING WHEN ALL ELSE FAILS, SEARCH ?BEING POPULAR ?BE A CONNECTOR ?GETTING EMOTIONAL 13 SCALING UP (OPTIMIZATION, PARALLELIZATION, AND BATCH PROCESSING) TOO MUCH OF A GOOD THING (DATA) OPTIMIZING NLP ALGORITHMS ?ADVANCED INDEXING ?ADVANCED INDEXING WITH ANNOY ?WHY USE APPROXIMATE INDEXES AT ALL? AN INDEXING WORKAROUND: DISCRETIZING CONSTANT RAM ALGORITHMS GENSIM ?GRAPH COMPUTING PARALLELIZING YOUR NLP COMPUTATIONS TRAINING NLP MODELS ON GPUS ?RENTING VS BUYING GPU RENTAL OPTIONS ?TENSOR PROCESSING UNITS REDUCING THE MEMORY FOOTPRINT DURING MODEL TRAINING GAINING MODEL INSIGHTS WITH TENSORBOARD HOW TO VISUALIZE WORD EMBEDDINGS