THE DATA SCIENCE DESIGN MANUAL
CONTENTS WHAT IS DATA SCIENCE? COMPUTER SCIENCE, DATA SCIENCE, AND REAL SCIENCE ASKING INTERESTING QUESTIONS FROM DATA THE BASEBALL ENCYCLOPEDIA THE INTERNET MOVIE DATABASE (IMDB) GOOGLE NGRAMS NEW YORK TAXI RECORDS PROPERTIES OF DATA STRUCTURED VS UNSTRUCTURED DATA QUANTITATIVE VS CATEGORICAL DATA BIG DATA VS LITTLE DATA CLASSI?CATION AND REGRESSION DATA SCIENCE TELEVISION: THE QUANT SHOP KAGGLE CHALLENGES ABOUT THE WAR STORIES WAR STORY: ANSWERING THE RIGHT QUESTION CHAPTER NOTES EXERCISES MATHEMATICAL PRELIMINARIES PROBABILITY PROBABILITY VS STATISTICS COMPOUND EVENTS AND INDEPENDENCE CONDITIONAL PROBABILITY PROBABILITY DISTRIBUTIONS DESCRIPTIVE STATISTICS CENTRALITY MEASURES VARIABILITY MEASURES INTERPRETING VARIANCE CHARACTERIZING DISTRIBUTIONS CORRELATION ANALYSIS CORRELATION COE?CIENTS: PEARSON AND SPEARMAN RANK THE POWER AND SIGNI?CANCE OF CORRELATION CORRELATION DOES NOT IMPLY CAUSATION! DETECTING PERIODICITIES BY AUTOCORRELATION LOGARITHMS LOGARITHMS AND MULTIPLYING PROBABILITIES LOGARITHMS AND RATIOS LOGARITHMS AND NORMALIZING SKEWED DISTRIBUTIONS WAR STORY: FITTING DESIGNER GENES DATA MUNGING LANGUAGES FOR DATA SCIENCE THE IMPORTANCE OF NOTEBOOK ENVIRONMENTS STANDARD DATA FORMATS COLLECTING DATA HUNTING SCRAPING LOGGING CLEANING DATA ERRORS VS ARTIFACTS DATA COMPATIBILITY DEALING WITH MISSING VALUES OUTLIER DETECTION WAR STORY: BEATING THE MARKET CROWDSOURCING THE PENNY DEMO WHEN IS THE CROWD WISE? MECHANISMS FOR AGGREGATION CROWDSOURCING SERVICES GAMI?CATION SCORES AND RANKINGS THE BODY MASS INDEX (BMI) DEVELOPING SCORING SYSTEMS GOLD STANDARDS AND PROXIES SCORES VS RANKINGS RECOGNIZING GOOD SCORING FUNCTIONS Z-SCORES AND NORMALIZATION ADVANCED RANKING TECHNIQUES ELO RANKINGS MERGING RANKINGS DIGRAPH-BASED RANKINGS PAGERANK WAR STORY: CLYDE’S REVENGE ARROW’S IMPOSSIBILITY THEOREM WAR STORY: WHO’S BIGGER? STATISTICAL ANALYSIS STATISTICAL DISTRIBUTIONS THE BINOMIAL DISTRIBUTION THE NORMAL DISTRIBUTION IMPLICATIONS OF THE NORMAL DISTRIBUTION POISSON DISTRIBUTION POWER LAW DISTRIBUTIONS SAMPLING FROM DISTRIBUTIONS RANDOM SAMPLING BEYOND ONE DIMENSION STATISTICAL SIGNI?CANCE THE SIGNI?CANCE OF SIGNI?CANCE THE T-TEST: COMPARING POPULATION MEANS THE KOLMOGOROV-SMIRNOV TEST THE BONFERRONI CORRECTION FALSE DISCOVERY RATE WAR STORY: DISCOVERING THE FOUNTAIN OF YOUTH? PERMUTATION TESTS AND P-VALUES GENERATING RANDOM PERMUTATIONS DIMAGGIO’S HITTING STREAK BAYESIAN REASONING VISUALIZING DATA EXPLORATORY DATA ANALYSIS CONFRONTING A NEW DATA SET SUMMARY STATISTICS AND ANSCOMBE’S QUARTET VISUALIZATION TOOLS DEVELOPING A VISUALIZATION AESTHETIC MAXIMIZING DATA-INK RATIO MINIMIZING THE LIE FACTOR MINIMIZING CHARTJUNK PROPER SCALING AND LABELING E?ECTIVE USE OF COLOR AND SHADING THE POWER OF REPETITION CHART TYPES TABULAR DATA DOT AND LINE PLOTS SCATTER PLOTS BAR PLOTS AND PIE CHARTS HISTOGRAMS DATA MAPS GREAT VISUALIZATIONS MAREY’S TRAIN SCHEDULE SNOW’S CHOLERA MAP NEW YORK’S WEATHER YEAR READING GRAPHS THE OBSCURED DISTRIBUTION OVERINTERPRETING VARIANCE INTERACTIVE VISUALIZATION WAR STORY: TEXTMAPPING THE WORLD MATHEMATICAL MODELS PHILOSOPHIES OF MODELING OCCAM’S RAZOR BIAS–VARIANCE TRADE-O?S WHAT WOULD NATE SILVER DO? A TAXONOMY OF MODELS LINEAR VS NON-LINEAR MODELS BLACKBOX VS DESCRIPTIVE MODELS FIRST-PRINCIPLE VS DATA-DRIVEN MODELS STOCHASTIC VS DETERMINISTIC MODELS FLAT VS HIERARCHICAL MODELS BASELINE MODELS BASELINE MODELS FOR CLASSI?CATION BASELINE MODELS FOR VALUE PREDICTION EVALUATING MODELS EVALUATING CLASSI?ERS RECEIVER-OPERATOR CHARACTERISTIC (ROC) CURVES EVALUATING MULTICLASS SYSTEMS EVALUATING VALUE PREDICTION MODELS EVALUATION ENVIRONMENTS DATA HYGIENE FOR EVALUATION AMPLIFYING SMALL EVALUATION SETS WAR STORY: 100% ACCURACY SIMULATION MODELS WAR STORY: CALCULATED BETS 7 10 EXERCISES LINEAR ALGEBRA THE POWER OF LINEAR ALGEBRA INTERPRETING LINEAR ALGEBRAIC FORMULAE GEOMETRY AND VECTORS VISUALIZING MATRIX OPERATIONS MATRIX ADDITION MATRIX MULTIPLICATION APPLICATIONS OF MATRIX MULTIPLICATION IDENTITY MATRICES AND INVERSION MATRIX INVERSION AND LINEAR SYSTEMS MATRIX RANK FACTORING MATRICES WHY FACTOR FEATURE MATRICES? LU DECOMPOSITION AND DETERMINANTS EIGENVALUES AND EIGENVECTORS PROPERTIES OF EIGENVALUES COMPUTING EIGENVALUES EIGENVALUE DECOMPOSITION SINGULAR VALUE DECOMPOSITION PRINCIPAL COMPONENTS ANALYSIS WAR STORY: THE HUMAN FACTORS LINEAR AND LOGISTIC REGRESSION LINEAR REGRESSION LINEAR REGRESSION AND DUALITY ERROR IN LINEAR REGRESSION FINDING THE OPTIMAL FIT BETTER REGRESSION MODELS REMOVING OUTLIERS FITTING NON-LINEAR FUNCTIONS FEATURE AND TARGET SCALING DEALING WITH HIGHLY-CORRELATED FEATURES WAR STORY: TAXI DERIVER REGRESSION AS PARAMETER FITTING CONVEX PARAMETER SPACES GRADIENT DESCENT SEARCH WHAT IS THE RIGHT LEARNING RATE? STOCHASTIC GRADIENT DESCENT SIMPLIFYING MODELS THROUGH REGULARIZATION RIDGE REGRESSION LASSO REGRESSION TRADE-O?S BETWEEN FIT AND COMPLEXITY CLASSI?CATION AND LOGISTIC REGRESSION REGRESSION FOR CLASSI?CATION DECISION BOUNDARIES LOGISTIC REGRESSION ISSUES IN LOGISTIC CLASSI?CATION BALANCED TRAINING CLASSES MULTI-CLASS CLASSI?CATION HIERARCHICAL CLASSI?CATION PARTITION FUNCTIONS AND MULTINOMIAL REGRESSION 10 DISTANCE AND NETWORK METHODS 10 1 MEASURING DISTANCES 10 1 1 DISTANCE METRICS 10 1 2 THE LK DISTANCE METRIC 10 1 3 WORKING IN HIGHER DIMENSIONS 10 1 4 DIMENSIONAL EGALITARIANISM 10 1 5 POINTS VS VECTORS 10 1 6 DISTANCES BETWEEN PROBABILITY DISTRIBUTIONS 10 2 NEAREST NEIGHBOR CLASSI?CATION 10 2 1 SEEKING GOOD ANALOGIES 10 2 2 K-NEAREST NEIGHBORS 10 2 3 FINDING NEAREST NEIGHBORS 10 2 4 LOCALITY SENSITIVE HASHING 10 3 GRAPHS, NETWORKS, AND DISTANCES 10 3 1 WEIGHTED GRAPHS AND INDUCED NETWORKS 10 3 2 TALKING ABOUT GRAPHS 10 3 3 GRAPH THEORY 10 4 PAGERANK 10 5 CLUSTERING 10 5 1 K-MEANS CLUSTERING 10 5 2 AGGLOMERATIVE CLUSTERING 10 5 3 COMPARING CLUSTERINGS 10 5 4 SIMILARITY GRAPHS AND CUT-BASED CLUSTERING 10 6 WAR STORY: CLUSTER BOMBING 10 7 CHAPTER NOTES 10 8 EXERCISES 11 MACHINE LEARNING 11 1 NAIVE BAYES 11 1 1 FORMULATION 11 1 2 DEALING WITH ZERO COUNTS (DISCOUNTING) 11 2 DECISION TREE CLASSI?ERS 11 2 1 CONSTRUCTING DECISION TREES 11 2 2 REALIZING EXCLUSIVE OR 11 2 3 ENSEMBLES OF DECISION TREES 11 3 BOOSTING AND ENSEMBLE LEARNING 11 3 1 VOTING WITH CLASSI?ERS 11 3 2 BOOSTING ALGORITHMS 11 4 SUPPORT VECTOR MACHINES 11 4 1 LINEAR SVMS 11 4 2 NON-LINEAR SVMS 11 4 3 KERNELS 11 5 DEGREES OF SUPERVISION 11 5 1 SUPERVISED LEARNING 11 5 2 UNSUPERVISED LEARNING 11 5 3 SEMI-SUPERVISED LEARNING 11 5 4 FEATURE ENGINEERING 11 6 DEEP LEARNING 11 6 1 NETWORKS AND DEPTH 11 6 2 BACKPROPAGATION 11 6 3 WORD AND GRAPH EMBEDDINGS 11 7 WAR STORY: THE NAME GAME 11 8 CHAPTER NOTES 11 9 EXERCISES 12 BIG DATA: ACHIEVING SCALE 12 1 WHAT IS BIG DATA? 12 1 1 BIG DATA AS BAD DATA 12 1 2 THE THREE VS 12 2 WAR STORY: INFRASTRUCTURE MATTERS 12 3 ALGORITHMICS FOR BIG DATA 12 3 1 BIG OH ANALYSIS 12 3 2 HASHING 12 3 3 EXPLOITING THE STORAGE HIERARCHY 12 3 4 STREAMING AND SINGLE-PASS ALGORITHMS 12 4 FILTERING AND SAMPLING 12 4 1 DETERMINISTIC SAMPLING ALGORITHMS 12 4 2 RANDOMIZED AND STREAM SAMPLING 12 5 PARALLELISM 12 5 1 ONE, TWO, MANY 12 5 2 DATA PARALLELISM 12 5 3 GRID SEARCH 12 5 4 CLOUD COMPUTING SERVICES 12 6 MAPREDUCE 12 6 1 MAP-REDUCE PROGRAMMING 12 6 2 MAPREDUCE UNDER THE HOOD 12 7 SOCIETAL AND ETHICAL IMPLICATIONS 12 8 CHAPTER NOTES 12 9 EXERCISES 13 CODA 13 1 GET A JOB! 13 2 GO TO GRADUATE SCHOOL! 13 3 PROFESSIONAL CONSULTING SERVICES