BEOWULF CLUSTER COMPUTING WITH LINUX
CONTENTS INTRODUCTION— DE?NITIONS AND TAXONOMY OPPORTUNITIES AND ADVANTAGES A SHORT HISTORY ELEMENTS OF A CLUSTER DESCRIPTION OF THE BOOK I ENABLING TECHNOLOGIES AN OVERVIEW OF CLUSTER COMPUTING— A TAXONOMY OF PARALLEL COMPUTING HARDWARE SYSTEM STRUCTURE BEOWULF COMPUTE NODES INTERCONNECTION NETWORKS NODE SOFTWARE RESOURCE MANAGEMENT DISTRIBUTED PROGRAMMING CONCLUSIONS NODE HARDWARE— OVERVIEW OF A BEOWULF NODE PRINCIPAL SPECI?CATIONS BASIC ELEMENTS PROCESSORS INTEL PENTIUM FAMILY AMD ATHLON COMPAQ ALPHA MOTHERBOARD MEMORY MEMORY CAPACITY MEMORY SPEED MEMORY TYPES MEMORY HIERARCHY AND CACHES PACKAGE STYLES BIOS SECONDARY STORAGE PCI BUS EXAMPLE OF A BEOWULF NODE BOXES, SHELVES, PILES, AND RACKS NODE ASSEMBLY MOTHERBOARD PREASSEMBLY THE CASE MINIMAL PERIPHERALS BOOTING THE SYSTEM INSTALLING THE OTHER COMPONENTS TROUBLESHOOTING LINUX— WHAT IS LINUX? WHY USE LINUX FOR A BEOWULF? A KERNEL AND A DISTRIBUTION OPEN SOURCE AND FREE SOFTWARE A LINUX DISTRIBUTION VERSION NUMBERS AND DEVELOPMENT METHODS THE LINUX KERNEL COMPILING A KERNEL LOADABLE KERNEL MODULES THE BEOWULF KERNEL DIET DISKLESS OPERATION DOWNLOADING AND COMPILING A NEW KERNEL LINUX FILE SYSTEMS PRUNING YOUR BEOWULF NODE INETD CONF /ETC/RC D/INIT D OTHER PROCESSES AND DAEMONS OTHER CONSIDERATIONS TCP MESSAGING HARDWARE PERFORMANCE COUNTERS FINAL TUNING WITH /PROC NETWORK HARDWARE— INTERCONNECT TECHNOLOGIES THE ETHERNETS MYRINET CLAN SCALABLE COHERENT INTERFACE QSNET IN?NIBAND A DETAILED LOOK AT ETHERNET PACKET FORMAT NIC ARCHITECTURE HUBS AND SWITCHES NETWORK PRACTICALITIES: INTERCONNECT CHOICE IMPORTANCE OF THE INTERCONNECT DI?ERENCES BETWEEN THE INTERCONNECT CHOICES STRATEGIES TO IMPROVE PERFORMANCE OVER ETHERNET CLUSTER NETWORK PITFALLS AN EXAMPLE OF AN ETHERNET INTERCONNECTED BEOWULF AN EXAMPLE OF A MYRINET INTERCONNECTED CLUSTER NETWORK SOFTWARE— TCP/IP IP ADDRESSES ZERO-COPY PROTOCOLS SOCKETS HIGHER-LEVEL PROTOCOLS REMOTE PROCEDURE CALLS DISTRIBUTED OBJECTS: CORBA AND JAVA RMI DISTRIBUTED FILE SYSTEMS NFS AFS AUTOFS: THE AUTOMOUNTER REMOTE COMMAND EXECUTION BSD R COMMANDS SSH—THE SECURE SHELL SETTING UP CLUSTERS: INSTALLATION AND CON?GURATION— SYSTEM ACCESS MODELS THE STANDALONE SYSTEM THE UNIVERSALLY ACCESSIBLE MACHINE THE GUARDED BEOWULF ASSIGNING NAMES STATISTICALLY ASSIGNED ADDRESSES DYNAMICALLY ASSIGNED ADDRESSES INSTALLING NODE SOFTWARE CREATING TAR IMAGES SETTING UP A CLONE ROOT PARTITION SETTING UP BOOTP BUILDING A CLONE BOOT FLOPPY BASIC SYSTEM ADMINISTRATION BOOTING AND SHUTTING DOWN THE NODE FILE SYSTEM ACCOUNT MANAGEMENT RUNNING UNIX COMMANDS IN PARALLEL AVOIDING SECURITY COMPROMISES SYSTEM CON?GURATION RESTRICTING HOST ACCESS SECURE SHELL IP MASQUERADING JOB SCHEDULING SOME ADVICE ON UPGRADING YOUR SOFTWARE HOW FAST IS MY BEOWULF?— METRICS PING-PONG TEST THE LINPACK BENCHMARK THE NAS PARALLEL BENCHMARK SUITE II PARALLEL PROGRAMMING PARALLEL PROGRAMMING WITH MPI— HELLO WORLD IN MPI COMPILING AND RUNNING MPI PROGRAMS ADDING COMMUNICATION TO HELLO WORLD MANAGER/WORKER EXAMPLE TWO-DIMENSIONAL JACOBI EXAMPLE WITH ONE-DIMENSIONAL DECOMPOSITION COLLECTIVE OPERATIONS PARALLEL MONTE CARLO COMPUTATION INSTALLING MPICH UNDER LINUX OBTAINING AND INSTALLING MPICH RUNNING MPICH JOBS WITH THE CH P4 DEVICE STARTING AND MANAGING MPD RUNNING MPICH JOBS UNDER MPD DEBUGGING MPI PROGRAMS OTHER COMPILERS TOOLS PRO?LING LIBRARIES VISUALIZING PARALLEL PROGRAM BEHAVIOR MPI IMPLEMENTATIONS FOR CLUSTERS MPI ROUTINE SUMMARY ADVANCED TOPICS IN MPI PROGRAMMING— DYNAMIC PROCESS MANAGEMENT IN MPI INTERCOMMUNICATORS SPAWNING NEW MPI PROCESSES REVISITING MATRIX-VECTOR MULTIPLICATION MORE ON DYNAMIC PROCESS MANAGEMENT FAULT TOLERANCE REVISITING MESH EXCHANGES BLOCKING AND NONBLOCKING COMMUNICATION COMMUNICATING NONCONTIGUOUS DATA IN MPI MOTIVATION FOR COMMUNICATORS MORE ON COLLECTIVE OPERATIONS PARALLEL I/O A SIMPLE EXAMPLE A MORE COMPLEX EXAMPLE REMOTE MEMORY ACCESS USING C++ AND FORTRAN MPI, OPENMP, AND THREADS MEASURING MPI PERFORMANCE 10 10 1 MPPTEST 10 10 2 SKAMPI 10 10 3 HIGH PERFORMANCE LINPACK MPI-2 STATUS PARALLEL PROGRAMMING WITH PVM— OVERVIEW PROGRAM EXAMPLES FORK/JOIN DOT PRODUCT MATRIX MULTIPLY ONE-DIMENSIONAL HEAT EQUATION USING PVM SETTING UP PVM STARTING PVM RUNNING PVM PROGRAMS PVM CONSOLE DETAILS HOST FILE OPTIONS XPVM 11 10 1 NETWORK VIEW 11 10 2 SPACE-TIME VIEW 11 10 3 OTHER VIEWS FAULT-TOLERANT AND ADAPTIVE PROGRAMS WITH PVM— CONSIDERATIONS FOR FAULT TOLERANCE BUILDING FAULT-TOLERANT PARALLEL APPLICATIONS ADAPTIVE PROGRAMS III MANAGING CLUSTERS CLUSTER WORKLOAD MANAGEMENT— GOAL OF WORKLOAD MANAGEMENT SOFTWARE WORKLOAD MANAGEMENT ACTIVITIES QUEUEING SCHEDULING MONITORING ACCOUNTING CONDOR: A DISTRIBUTED JOB SCHEDULER— INTRODUCTION TO CONDOR FEATURES OF CONDOR UNDERSTANDING CONDOR CLASSADS USING CONDOR ROADMAP TO USING CONDOR SUBMITTING A JOB OVERVIEW OF USER COMMANDS SUBMITTING DI?ERENT TYPES OF JOBS: ALTERNATIVE UNIVERSES GIVING YOUR JOB ACCESS TO ITS DATA FILES THE DAGMAN SCHEDULER CONDOR ARCHITECTURE THE CONDOR DAEMONS THE CONDOR DAEMONS IN ACTION INSTALLING CONDOR UNDER LINUX CON?GURING CONDOR LOCATION OF CONDOR’S CON?GURATION FILES RECOMMENDED CON?GURATION FILE LAYOUT FOR A CLUSTER CUSTOMIZING CONDOR’S POLICY EXPRESSIONS CUSTOMIZING CONDOR’S OTHER CON?GURATION SETTINGS ADMINISTRATION TOOLS REMOTE CON?GURATION AND CONTROL ACCOUNTING AND LOGGING USER PRIORITIES IN CONDOR CLUSTER SETUP SCENARIOS BASIC CON?GURATION: UNIFORMLY OWNED CLUSTER USING MULTIPROCESSOR COMPUTE NODES SCHEDULING A DISTRIBUTIVELY OWNED CLUSTER SUBMITTING TO THE CLUSTER FROM DESKTOP WORKSTATIONS EXPANDING THE CLUSTER TO NONDEDICATED (DESKTOP) COMPUTING RESOURCES CONCLUSION MAUI SCHEDULER: A MULTIFUNCTION CLUSTER SCHEDULER— INSTALLATION AND INITIAL CON?GURATION BASIC CON?GURATION SIMULATION AND TESTING PRODUCTION SCHEDULING ADVANCED CON?GURATION ASSIGNING VALUE: JOB PRIORITIZATION AND NODE ALLOCATION FAIRNESS: THROTTLING POLICIES AND FAIRSHARE MANAGING RESOURCE ACCESS: RESERVATIONS, ALLOCATION MANAGERS, AND QUALITY OF SERVICE OPTIMIZING USAGE: BACK?LL, NODE SETS, AND PREEMPTION EVALUATING SYSTEM PERFORMANCE: DIAGNOSTICS, PRO?LING, TESTING, AND SIMULATION STEERING WORKLOAD AND IMPROVING QUALITY OF INFORMATION PBS: PORTABLE BATCH SYSTEM— HISTORY OF PBS ACQUIRING PBS PBS FEATURES PBS ARCHITECTURE USING PBS CREATING A PBS JOB SUBMITTING A PBS JOB GETTING THE STATUS OF A PBS JOB PBS COMMAND SUMMARY USING THE PBS GRAPHICAL USER INTERFACE PBS APPLICATION PROGRAMMING INTERFACE INSTALLING PBS CON?GURING PBS NETWORK ADDRESSES AND PBS THE QMGR COMMAND NODES CREATING OR ADDING NODES DEFAULT CON?GURATION CON?GURING MOM SCHEDULER CON?GURATION MANAGING PBS STARTING PBS DAEMONS MONITORING PBS TRACKING PBS JOBS PBS ACCOUNTING LOGS CLIENTS UNABLE TO CONTACT SERVER NODES DOWN NONDELIVERY OF OUTPUT JOB CANNOT BE EXECUTED PVFS: PARALLEL VIRTUAL FILE SYSTEM— INTRODUCTION PARALLEL FILE SYSTEMS SETTING UP A PARALLEL FILE SYSTEM PROGRAMMING WITH A PARALLEL FILE SYSTEM USING PVFS WRITING PVFS PROGRAMS PVFS UTILITIES ADMINISTERING PVFS BUILDING THE PVFS COMPONENTS INSTALLATION STARTUP AND SHUTDOWN CON?GURATION DETAILS MISCELLANEA FINAL WORDS CHIBA CITY: THE ARGONNE SCALABLE CLUSTER— CHIBA CITY CON?GURATION NODE CON?GURATION LOGICAL CON?GURATION NETWORK CON?GURATION PHYSICAL CON?GURATION CHIBA CITY TIMELINE PHASE 1: MOTIVATION PHASE 2: DESIGN AND PURCHASE PHASE 3: INSTALLATION PHASE 4: FINAL DEVELOPMENT PHASE 5: EARLY USERS PHASE 6: FULL OPERATION CHIBA CITY SOFTWARE ENVIRONMENT THE COMPUTING ENVIRONMENT MANAGEMENT ENVIRONMENT CHIBA CITY USE FINAL THOUGHTS LESSONS LEARNED FUTURE DIRECTIONS CONCLUSIONS— FUTURE DIRECTIONS FOR HARDWARE COMPONENTS FUTURE DIRECTIONS FOR SOFTWARE COMPONENTS