THE MYTHICAL MAN-MONTH: ESSAYS ON SOFTWARE ENGINEERING

By EMMA LOPEZ

TABLE OF CONTENTS PART I THESIS 1 WHAT IS SOFTWARE ENGINEERING? TIME AND CHANGE HYRUM’S LAW EXAMPLE: HASH ORDERING WHY NOT JUST AIM FOR “NOTHING CHANGES”? SCALE AND EFFICIENCY POLICIES THAT DON’T SCALE POLICIES THAT SCALE WELL EXAMPLE: COMPILER UPGRADE SHIFTING LEFT TRADE-OFFS AND COSTS EXAMPLE: MARKERS INPUTS TO DECISION MAKING EXAMPLE: DISTRIBUTED BUILDS EXAMPLE: DECIDING BETWEEN TIME AND SCALE REVISITING DECISIONS, MAKING MISTAKES SOFTWARE ENGINEERING VERSUS PROGRAMMING CONCLUSION TL;DRS PART II CULTURE 2 HOW TO WORK WELL ON TEAMS HELP ME HIDE MY CODE THE GENIUS MYTH HIDING CONSIDERED HARMFUL EARLY DETECTION THE BUS FACTOR PACE OF PROGRESS IN SHORT, DON’T HIDE IT’S ALL ABOUT THE TEAM THE THREE PILLARS OF SOCIAL INTERACTION WHY DO THESE PILLARS MATTER? HUMILITY, RESPECT, AND TRUST IN PRACTICE BLAMELESS POST-MORTEM CULTURE BEING GOOGLEY 3 KNOWLEDGE SHARING CHALLENGES TO LEARNING PHILOSOPHY SETTING THE STAGE: PSYCHOLOGICAL SAFETY MENTORSHIP PSYCHOLOGICAL SAFETY IN LARGE GROUPS GROWING YOUR KNOWLEDGE ASK QUESTIONS UNDERSTAND CONTEXT SCALING YOUR QUESTIONS: ASK THE COMMUNITY GROUP CHATS MAILING LISTS YAQS: QUESTION-AND-ANSWER PLATFORM SCALING YOUR KNOWLEDGE: YOU ALWAYS HAVE SOMETHING TO TEACH OFFICE HOURS TECH TALKS AND CLASSES DOCUMENTATION CODE SCALING YOUR ORGANIZATION’S KNOWLEDGE CULTIVATING A KNOWLEDGE-SHARING CULTURE ESTABLISHING CANONICAL SOURCES OF INFORMATION STAYING IN THE LOOP READABILITY: STANDARDIZED MENTORSHIP THROUGH CODE REVIEW WHAT IS THE READABILITY PROCESS? WHY HAVE THIS PROCESS? 4 ENGINEERING FOR EQUITY BIAS IS THE DEFAULT UNDERSTANDING THE NEED FOR DIVERSITY BUILDING MULTICULTURAL CAPACITY MAKING DIVERSITY ACTIONABLE REJECT SINGULAR APPROACHES CHALLENGE ESTABLISHED PROCESSES VALUES VERSUS OUTCOMES STAY CURIOUS, PUSH FORWARD 5 HOW TO LEAD A TEAM MANAGERS AND TECH LEADS (AND BOTH) THE ENGINEERING MANAGER THE TECH LEAD THE TECH LEAD MANAGER MOVING FROM AN INDIVIDUAL CONTRIBUTOR ROLE TO A LEADERSHIP ROLE THE ONLY THING TO FEAR IS…WELL, EVERYTHING SERVANT LEADERSHIP MANAGER IS A FOUR-LETTER WORD TODAY’S ENGINEERING MANAGER ANTIPATTERNS ANTIPATTERN: HIRE PUSHOVERS ANTIPATTERN: IGNORE LOW PERFORMERS ANTIPATTERN: IGNORE HUMAN ISSUES ANTIPATTERN: BE EVERYONE’S FRIEND ANTIPATTERN: COMPROMISE THE HIRING BAR ANTIPATTERN: TREAT YOUR TEAM LIKE CHILDREN POSITIVE PATTERNS LOSE THE EGO BE A ZEN MASTER BE A CATALYST REMOVE ROADBLOCKS BE A TEACHER AND A MENTOR SET CLEAR GOALS BE HONEST TRACK HAPPINESS THE UNEXPECTED QUESTION OTHER TIPS AND TRICKS PEOPLE ARE LIKE PLANTS INTRINSIC VERSUS EXTRINSIC MOTIVATION 6 LEADING AT SCALE ALWAYS BE DECIDING THE PARABLE OF THE AIRPLANE IDENTIFY THE BLINDERS IDENTIFY THE KEY TRADE-OFFS DECIDE, THEN ITERATE ALWAYS BE LEAVING YOUR MISSION: BUILD A “SELF-DRIVING” TEAM DIVIDING THE PROBLEM SPACE ALWAYS BE SCALING THE CYCLE OF SUCCESS IMPORTANT VERSUS URGENT LEARN TO DROP BALLS PROTECTING YOUR ENERGY 7 MEASURING ENGINEERING PRODUCTIVITY WHY SHOULD WE MEASURE ENGINEERING PRODUCTIVITY? TRIAGE: IS IT EVEN WORTH MEASURING? SELECTING MEANINGFUL METRICS WITH GOALS AND SIGNALS GOALS SIGNALS METRICS USING DATA TO VALIDATE METRICS TAKING ACTION AND TRACKING RESULTS PART III PROCESSES 8 STYLE GUIDES AND RULES WHY HAVE RULES? CREATING THE RULES GUIDING PRINCIPLES THE STYLE GUIDE CHANGING THE RULES THE PROCESS THE STYLE ARBITERS EXCEPTIONS GUIDANCE APPLYING THE RULES ERROR CHECKERS CODE FORMATTERS 9 CODE REVIEW CODE REVIEW FLOW HOW CODE REVIEW WORKS AT GOOGLE CODE REVIEW BENEFITS CODE CORRECTNESS COMPREHENSION OF CODE CODE CONSISTENCY PSYCHOLOGICAL AND CULTURAL BENEFITS KNOWLEDGE SHARING CODE REVIEW BEST PRACTICES BE POLITE AND PROFESSIONAL WRITE SMALL CHANGES WRITE GOOD CHANGE DESCRIPTIONS KEEP REVIEWERS TO A MINIMUM AUTOMATE WHERE POSSIBLE TYPES OF CODE REVIEWS GREENFIELD CODE REVIEWS BEHAVIORAL CHANGES, IMPROVEMENTS, AND OPTIMIZATIONS BUG FIXES AND ROLLBACKS REFACTORINGS AND LARGE-SCALE CHANGES 10 DOCUMENTATION WHAT QUALIFIES AS DOCUMENTATION? WHY IS DOCUMENTATION NEEDED? DOCUMENTATION IS LIKE CODE KNOW YOUR AUDIENCE TYPES OF AUDIENCES DOCUMENTATION TYPES REFERENCE DOCUMENTATION DESIGN DOCS TUTORIALS CONCEPTUAL DOCUMENTATION LANDING PAGES DOCUMENTATION REVIEWS DOCUMENTATION PHILOSOPHY WHO, WHAT, WHEN, WHERE, AND WHY THE BEGINNING, MIDDLE, AND END THE PARAMETERS OF GOOD DOCUMENTATION DEPRECATING DOCUMENTS WHEN DO YOU NEED TECHNICAL WRITERS? 11 TESTING OVERVIEW WHY DO WE WRITE TESTS? THE STORY OF GOOGLE WEB SERVER TESTING AT THE SPEED OF MODERN DEVELOPMENT WRITE, RUN, REACT BENEFITS OF TESTING CODE DESIGNING A TEST SUITE TEST SIZE TEST SCOPE THE BEYONCÉ RULE A NOTE ON CODE COVERAGE TESTING AT GOOGLE SCALE THE PITFALLS OF A LARGE TEST SUITE HISTORY OF TESTING AT GOOGLE ORIENTATION CLASSES TEST CERTIFIED TESTING ON THE TOILET TESTING CULTURE TODAY THE LIMITS OF AUTOMATED TESTING 12 UNIT TESTING THE IMPORTANCE OF MAINTAINABILITY PREVENTING BRITTLE TESTS STRIVE FOR UNCHANGING TESTS TEST VIA PUBLIC APIS TEST STATE, NOT INTERACTIONS WRITING CLEAR TESTS MAKE YOUR TESTS COMPLETE AND CONCISE TEST BEHAVIORS, NOT METHODS DON’T PUT LOGIC IN TESTS WRITE CLEAR FAILURE MESSAGES TESTS AND CODE SHARING: DAMP, NOT DRY SHARED VALUES SHARED SETUP SHARED HELPERS AND VALIDATION DEFINING TEST INFRASTRUCTURE 13 TEST DOUBLES THE IMPACT OF TEST DOUBLES ON SOFTWARE DEVELOPMENT TEST DOUBLES AT GOOGLE BASIC CONCEPTS AN EXAMPLE TEST DOUBLE SEAMS MOCKING FRAMEWORKS TECHNIQUES FOR USING TEST DOUBLES FAKING STUBBING INTERACTION TESTING REAL IMPLEMENTATIONS PREFER REALISM OVER ISOLATION HOW TO DECIDE WHEN TO USE A REAL IMPLEMENTATION WHY ARE FAKES IMPORTANT? WHEN SHOULD FAKES BE WRITTEN? THE FIDELITY OF FAKES FAKES SHOULD BE TESTED WHAT TO DO IF A FAKE IS NOT AVAILABLE THE DANGERS OF OVERUSING STUBBING WHEN IS STUBBING APPROPRIATE? PREFER STATE TESTING OVER INTERACTION TESTING WHEN IS INTERACTION TESTING APPROPRIATE? BEST PRACTICES FOR INTERACTION TESTING 14 LARGER TESTING WHAT ARE LARGER TESTS? FIDELITY COMMON GAPS IN UNIT TESTS WHY NOT HAVE LARGER TESTS? LARGER TESTS AT GOOGLE LARGER TESTS AND TIME LARGER TESTS AT GOOGLE SCALE STRUCTURE OF A LARGE TEST THE SYSTEM UNDER TEST TEST DATA VERIFICATION TYPES OF LARGER TESTS FUNCTIONAL TESTING OF ONE OR MORE INTERACTING BINARIES BROWSER AND DEVICE TESTING PERFORMANCE, LOAD, AND STRESS TESTING DEPLOYMENT CONFIGURATION TESTING EXPLORATORY TESTING A/B DIFF REGRESSION TESTING UAT PROBERS AND CANARY ANALYSIS DISASTER RECOVERY AND CHAOS ENGINEERING USER EVALUATION LARGE TESTS AND THE DEVELOPER WORKFLOW AUTHORING LARGE TESTS RUNNING LARGE TESTS OWNING LARGE TESTS 15 DEPRECATION WHY DEPRECATE? WHY IS DEPRECATION SO HARD? DEPRECATION DURING DESIGN TYPES OF DEPRECATION ADVISORY DEPRECATION COMPULSORY DEPRECATION DEPRECATION WARNINGS MANAGING THE DEPRECATION PROCESS PROCESS OWNERS MILESTONES DEPRECATION TOOLING PART IV TOOLS 16 VERSION CONTROL AND BRANCH MANAGEMENT WHAT IS VERSION CONTROL? WHY IS VERSION CONTROL IMPORTANT? CENTRALIZED VCS VERSUS DISTRIBUTED VCS SOURCE OF TRUTH VERSION CONTROL VERSUS DEPENDENCY MANAGEMENT BRANCH MANAGEMENT WORK IN PROGRESS IS AKIN TO A BRANCH DEV BRANCHES RELEASE BRANCHES VERSION CONTROL AT GOOGLE ONE VERSION SCENARIO: MULTIPLE AVAILABLE VERSIONS THE “ONE-VERSION” RULE (NEARLY) NO LONG-LIVED BRANCHES WHAT ABOUT RELEASE BRANCHES? MONOREPOS FUTURE OF VERSION CONTROL 17 CODE SEARCH THE CODE SEARCH UI HOW DO GOOGLERS USE CODE SEARCH? WHERE? WHAT? HOW? WHY? WHO AND WHEN? WHY A SEPARATE WEB TOOL? SCALE ZERO SETUP GLOBAL CODE VIEW SPECIALIZATION INTEGRATION WITH OTHER DEVELOPER TOOLS API EXPOSURE IMPACT OF SCALE ON DESIGN SEARCH QUERY LATENCY INDEX LATENCY GOOGLE’S IMPLEMENTATION SEARCH INDEX RANKING SELECTED TRADE-OFFS COMPLETENESS: REPOSITORY AT HEAD COMPLETENESS: ALL VERSUS MOST-RELEVANT RESULTS COMPLETENESS: HEAD VERSUS BRANCHES VERSUS ALL HISTORY VERSUS WORKSPACES EXPRESSIVENESS: TOKEN VERSUS SUBSTRING VERSUS REGEX 18 BUILD SYSTEMS AND BUILD PHILOSOPHY PURPOSE OF A BUILD SYSTEM WHAT HAPPENS WITHOUT A BUILD SYSTEM? BUT ALL I NEED IS A COMPILER! SHELL SCRIPTS TO THE RESCUE? MODERN BUILD SYSTEMS IT’S ALL ABOUT DEPENDENCIES TASK-BASED BUILD SYSTEMS ARTIFACT-BASED BUILD SYSTEMS DISTRIBUTED BUILDS TIME, SCALE, TRADE-OFFS DEALING WITH MODULES AND DEPENDENCIES USING FINE-GRAINED MODULES AND THE 1:1:1 RULE MINIMIZING MODULE VISIBILITY MANAGING DEPENDENCIES 19 CRITIQUE: GOOGLE’S CODE REVIEW TOOL CODE REVIEW TOOLING PRINCIPLES NOTIFICATIONS STAGE 1: CREATE A CHANGE DIFFING ANALYSIS RESULTS TIGHT TOOL INTEGRATION STAGE 2: REQUEST REVIEW STAGES 3 AND 4: UNDERSTANDING AND COMMENTING ON A CHANGE COMMENTING UNDERSTANDING THE STATE OF A CHANGE STAGE 5: CHANGE APPROVALS (SCORING A CHANGE) STAGE 6: COMMITING A CHANGE AFTER COMMIT: TRACKING HISTORY 20 STATIC ANALYSIS CHARACTERISTICS OF EFFECTIVE STATIC ANALYSIS SCALABILITY USABILITY KEY LESSONS IN MAKING STATIC ANALYSIS WORK FOCUS ON DEVELOPER HAPPINESS MAKE STATIC ANALYSIS A PART OF THE CORE DEVELOPER WORKFLOW EMPOWER USERS TO CONTRIBUTE TRICORDER: GOOGLE’S STATIC ANALYSIS PLATFORM INTEGRATED TOOLS INTEGRATED FEEDBACK CHANNELS SUGGESTED FIXES PER-PROJECT CUSTOMIZATION PRESUBMITS COMPILER INTEGRATION ANALYSIS WHILE EDITING AND BROWSING CODE 21 DEPENDENCY MANAGEMENT WHY IS DEPENDENCY MANAGEMENT SO DIFFICULT? CONFLICTING REQUIREMENTS AND DIAMOND DEPENDENCIES IMPORTING DEPENDENCIES COMPATIBILITY PROMISES CONSIDERATIONS WHEN IMPORTING HOW GOOGLE HANDLES IMPORTING DEPENDENCIES DEPENDENCY MANAGEMENT, IN THEORY NOTHING CHANGES (AKA THE STATIC DEPENDENCY MODEL) SEMANTIC VERSIONING BUNDLED DISTRIBUTION MODELS LIVE AT HEAD THE LIMITATIONS OF SEMVER SEMVER MIGHT OVERCONSTRAIN SEMVER MIGHT OVERPROMISE MOTIVATIONS MINIMUM VERSION SELECTION SO, DOES SEMVER WORK? DEPENDENCY MANAGEMENT WITH INFINITE RESOURCES EXPORTING DEPENDENCIES 22 LARGE-SCALE CHANGES WHAT IS A LARGE-SCALE CHANGE? WHO DEALS WITH LSCS? BARRIERS TO ATOMIC CHANGES TECHNICAL LIMITATIONS MERGE CONFLICTS NO HAUNTED GRAVEYARDS HETEROGENEITY TESTING CODE REVIEW LSC INFRASTRUCTURE POLICIES AND CULTURE CODEBASE INSIGHT CHANGE MANAGEMENT LANGUAGE SUPPORT THE LSC PROCESS AUTHORIZATION CHANGE CREATION SHARDING AND SUBMITTING CLEANUP 23 CONTINUOUS INTEGRATION CI CONCEPTS FAST FEEDBACK LOOPS AUTOMATION CONTINUOUS TESTING CI CHALLENGES HERMETIC TESTING CI AT GOOGLE CI CASE STUDY: GOOGLE TAKEOUT BUT I CAN’T AFFORD CI 24 CONTINUOUS DELIVERY IDIOMS OF CONTINUOUS DELIVERY AT GOOGLE VELOCITY IS A TEAM SPORT: HOW TO BREAK UP A DEPLOYMENT INTO MANAGEABLE PIECES EVALUATING CHANGES IN ISOLATION: FLAG-GUARDING FEATURES STRIVING FOR AGILITY: SETTING UP A RELEASE TRAIN NO BINARY IS PERFECT MEET YOUR RELEASE DEADLINE QUALITY AND USER-FOCUS: SHIP ONLY WHAT GETS USED SHIFTING LEFT: MAKING DATA-DRIVEN DECISIONS EARLIER CHANGING TEAM CULTURE: BUILDING DISCIPLINE INTO DEPLOYMENT 25 COMPUTE AS A SERVICE TAMING THE COMPUTE ENVIRONMENT AUTOMATION OF TOIL CONTAINERIZATION AND MULTITENANCY SUMMARY WRITING SOFTWARE FOR MANAGED COMPUTE ARCHITECTING FOR FAILURE BATCH VERSUS SERVING MANAGING STATE CONNECTING TO A SERVICE ONE-OFF CODE CAAS OVER TIME AND SCALE CONTAINERS AS AN ABSTRACTION ONE SERVICE TO RULE THEM ALL SUBMITTED CONFIGURATION CHOOSING A COMPUTE SERVICE CENTRALIZATION VERSUS CUSTOMIZATION LEVEL OF ABSTRACTION: SERVERLESS PUBLIC VERSUS PRIVATE PART V CONCLUSION AFTERWORD