REVERSING: SECRETS OF REVERSE ENGINEERING
PART I REVERSING CHAPTER FOUNDATIONS WHAT IS REVERSE ENGINEERING? SOFTWARE REVERSE ENGINEERING: REVERSING REVERSING APPLICATIONS SECURITY-RELATED REVERSING MALICIOUS SOFTWARE REVERSING CRYPTOGRAPHIC ALGORITHMS DIGITAL RIGHTS MANAGEMENT AUDITING PROGRAM BINARIES REVERSING IN SOFTWARE DEVELOPMENT ACHIEVING INTEROPERABILITY WITH PROPRIETARY SOFTWARE DEVELOPING COMPETING SOFTWARE EVALUATING SOFTWARE QUALITY AND ROBUSTNESS LOW-LEVEL SOFTWARE ASSEMBLY LANGUAGE COMPILERS VIRTUAL MACHINES AND BYTECODES OPERATING SYSTEMS CONTENTS THE REVERSING PROCESS SYSTEM-LEVEL REVERSING CODE-LEVEL REVERSING THE TOOLS SYSTEM-MONITORING TOOLS DISASSEMBLERS DEBUGGERS DECOMPILERS IS REVERSING LEGAL? INTEROPERABILITY COMPETITION COPYRIGHT LAW TRADE SECRETS AND PATENTS THE DIGITAL MILLENIUM COPYRIGHT ACT DMCA CASES LICENSE AGREEMENT CONSIDERATIONS CODE SAMPLES & TOOLS CONCLUSION HIGH-LEVEL PERSPECTIVES PROGRAM STRUCTURE MODULES COMMON CODE CONSTRUCTS DATA MANAGEMENT VARIABLES USER-DEFINED DATA STRUCTURES LISTS CONTROL FLOW HIGH-LEVEL LANGUAGES C++ JAVA LOW-LEVEL PERSPECTIVES LOW-LEVEL DATA MANAGEMENT REGISTERS THE STACK HEAPS EXECUTABLE DATA SECTIONS FLAGS INSTRUCTION FORMAT BASIC INSTRUCTIONS MOVING DATA ARITHMETIC COMPARING OPERANDS CONDITIONAL BRANCHES FUNCTION CALLS EXAMPLES A PRIMER ON COMPILERS AND COMPILATION DEFINING A COMPILER COMPILER ARCHITECTURE FRONT END INTERMEDIATE REPRESENTATIONS OPTIMIZER BACK END LISTING FILES SPECIFIC COMPILERS EXECUTION ENVIRONMENTS SOFTWARE EXECUTION ENVIRONMENTS (VIRTUAL MACHINES) BYTECODES INTERPRETERS JUST-IN-TIME COMPILERS REVERSING STRATEGIES HARDWARE EXECUTION ENVIRONMENTS IN MODERN PROCESSORS INTEL NETBURST µOPS (MICRO-OPS) PIPELINES BRANCH PREDICTION WINDOWS FUNDAMENTALS COMPONENTS AND BASIC ARCHITECTURE BRIEF HISTORY FEATURES SUPPORTED HARDWARE MEMORY MANAGEMENT VIRTUAL MEMORY AND PAGING PAGING PAGE FAULTS WORKING SETS KERNEL MEMORY AND USER MEMORY THE KERNEL MEMORY SPACE SECTION OBJECTS VAD TREES USER-MODE ALLOCATIONS MEMORY MANAGEMENT APIS OBJECTS AND HANDLES NAMED OBJECTS PROCESSES AND THREADS PROCESSES THREADS CONTEXT SWITCHING SYNCHRONIZATION OBJECTS PROCESS INITIALIZATION SEQUENCE APPLICATION PROGRAMMING INTERFACES THE WIN32 API THE NATIVE API SYSTEM CALLING MECHANISM EXECUTABLE FORMATS BASIC CONCEPTS IMAGE SECTIONS SECTION ALIGNMENT DYNAMICALLY LINKED LIBRARIES HEADERS IMPORTS AND EXPORTS DIRECTORIES INPUT AND OUTPUT THE I/O SYSTEM THE WIN32 SUBSYSTEM OBJECT MANAGEMENT STRUCTURED EXCEPTION HANDLING REVERSING TOOLS DIFFERENT REVERSING APPROACHES OFFLINE CODE ANALYSIS (DEAD-LISTING) LIVE CODE ANALYSIS IDA PRO ILDASM USER-MODE DEBUGGERS OLLYDBG USER DEBUGGING IN WINDBG PEBROWSE PROFESSIONAL INTERACTIVE KERNEL-MODE DEBUGGERS KERNEL DEBUGGING IN WINDBG NUMEGA SOFTICE KERNEL DEBUGGING ON VIRTUAL MACHINES PATCHING TOOLS HEX WORKSHOP MISCELLANEOUS REVERSING TOOLS EXECUTABLE-DUMPING TOOLS DUMPBIN PEVIEW PEBROWSE PROFESSIONAL PART II APPLIED REVERSING BEYOND THE DOCUMENTATION REVERSING AND INTEROPERABILITY LAYING THE GROUND RULES LOCATING UNDOCUMENTED APIS WHAT ARE WE LOOKING FOR? CASE STUDY: THE GENERIC TABLE API IN NTDLL DLL RTLINITIALIZEGENERICTABLE RTLNUMBERGENERICTABLEELEMENTS RTLISGENERICTABLEEMPTY RTLGETELEMENTGENERICTABLE SETUP AND INITIALIZATION LOGIC AND STRUCTURE SEARCH LOOP RECONSTRUCTING THE SOURCE CODE RTLINSERTELEMENTGENERICTABLE RTLLOCATENODEGENERICTABLE RTLREALINSERTELEMENTWORKER SPLAY TREES RTLLOOKUPELEMENTGENERICTABLE RTLDELETEELEMENTGENERICTABLE PUTTING THE PIECES TOGETHER DECIPHERING FILE FORMATS CRYPTEX USING CRYPTEX REVERSING CRYPTEX THE PASSWORD VERIFICATION PROCESS CATCHING THE “BAD PASSWORD” MESSAGE THE PASSWORD TRANSFORMATION ALGORITHM HASHING THE PASSWORD THE DIRECTORY LAYOUT ANALYZING THE DIRECTORY PROCESSING CODE ANALYZING A FILE ENTRY DUMPING THE DIRECTORY LAYOUT THE FILE EXTRACTION PROCESS SCANNING THE FILE LIST DECRYPTING THE FILE THE FLOATING-POINT SEQUENCE THE DECRYPTION LOOP VERIFYING THE HASH VALUE THE BIG PICTURE DIGGING DEEPER DEFINING THE PROBLEM VULNERABILITIES STACK OVERFLOWS A SIMPLE STACK VULNERABILITY INTRINSIC IMPLEMENTATIONS STACK CHECKING NONEXECUTABLE MEMORY HEAP OVERFLOWS STRING FILTERS INTEGER OVERFLOWS ARITHMETIC OPERATIONS ON USER-SUPPLIED INTEGERS TYPE CONVERSION ERRORS CASE-STUDY: THE IIS INDEXING SERVICE VULNERABILITY CVARIABLESET::ADDEXTENSIONCONTROLBLOCK DECODEURLESCAPES REVERSING MALWARE TYPES OF MALWARE VIRUSES WORMS TROJAN HORSES BACKDOORS MOBILE CODE ADWARE/SPYWARE STICKY SOFTWARE FUTURE MALWARE INFORMATION-STEALING WORMS BIOS/FIRMWARE MALWARE USES OF MALWARE MALWARE VULNERABILITY POLYMORPHISM METAMORPHISM ESTABLISHING A SECURE ENVIRONMENT THE BACKDOOR HACARMY D UNPACKING THE EXECUTABLE INITIAL IMPRESSIONS THE INITIAL INSTALLATION INITIALIZING COMMUNICATIONS CONNECTING TO THE SERVER JOINING THE CHANNEL COMMUNICATING WITH THE BACKDOOR RUNNING SOCKS4 SERVERS CLEARING THE CRIME SCENE THE BACKDOOR HACARMY D: A COMMAND REFERENCE PART III CRACKING PIRACY AND COPY PROTECTION COPYRIGHTS IN THE NEW WORLD THE SOCIAL ASPECT SOFTWARE PIRACY CLASS BREAKS REQUIREMENTS THE THEORETICALLY UNCRACKABLE MODEL TYPES OF PROTECTION MEDIA-BASED PROTECTIONS SERIAL NUMBERS CHALLENGE RESPONSE AND ONLINE ACTIVATIONS HARDWARE-BASED PROTECTIONS SOFTWARE AS A SERVICE ADVANCED PROTECTION CONCEPTS CRYPTO-PROCESSORS DRM MODELS THE WINDOWS MEDIA RIGHTS MANAGER SECURE AUDIO PATH WATERMARKING TRUSTED COMPUTING ATTACKING COPY PROTECTION TECHNOLOGIES ANTIREVERSING TECHNIQUES WHY ANTIREVERSING? BASIC APPROACHES TO ANTIREVERSING ELIMINATING SYMBOLIC INFORMATION CODE ENCRYPTION ACTIVE ANTIDEBUGGER TECHNIQUES DEBUGGER BASICS THE ISDEBUGGERPRESENT API SYSTEMKERNELDEBUGGERINFORMATION DETECTING SOFTICE USING THE SINGLE-STEP INTERRUPT THE TRAP FLAG CODE CHECKSUMS CONFUSING DISASSEMBLERS LINEAR SWEEP DISASSEMBLERS RECURSIVE TRAVERSAL DISASSEMBLERS APPLICATIONS CODE OBFUSCATION CONTROL FLOW TRANSFORMATIONS OPAQUE PREDICATES CONFUSING DECOMPILERS TABLE INTERPRETATION INLINING AND OUTLINING INTERLEAVING CODE ORDERING TRANSFORMATIONS DATA TRANSFORMATIONS MODIFYING VARIABLE ENCODING RESTRUCTURING ARRAYS BREAKING PROTECTIONS PATCHING KEYGENNING RIPPING KEY-GENERATION ALGORITHMS ADVANCED CRACKING: DEFENDER REVERSING DEFENDER’S INITIALIZATION ROUTINE ANALYZING THE DECRYPTED CODE SOFTICE’S DISAPPEARANCE REVERSING THE SECONDARY THREAD DEFEATING THE “KILLER” THREAD LOADING KERNEL32 DLL REENCRYPTING THE FUNCTION BACK AT THE ENTRY POINT PARSING THE PROGRAM PARAMETERS PROCESSING THE USERNAME VALIDATING USER INFORMATION UNLOCKING THE CODE BRUTE-FORCING YOUR WAY THROUGH DEFENDER PROTECTION TECHNOLOGIES IN DEFENDER LOCALIZED FUNCTION-LEVEL ENCRYPTION RELATIVELY STRONG CIPHER BLOCK CHAINING REENCRYPTING OBFUSCATED APPLICATION/OPERATING SYSTEM INTERFACE PROCESSOR TIME-STAMP VERIFICATION THREAD RUNTIME GENERATION OF DECRYPTION KEYS INTERDEPENDENT KEYS USER-INPUT-BASED DECRYPTION KEYS HEAVY INLINING PART IV BEYOND DISASSEMBLY REVERSING NET GROUND RULES NET BASICS MANAGED CODE NET PROGRAMMING LANGUAGES COMMON TYPE SYSTEM (CTS) INTERMEDIATE LANGUAGE (IL) THE EVALUATION STACK ACTIVATION RECORDS IL INSTRUCTIONS IL CODE SAMPLES COUNTING ITEMS A LINKED LIST SAMPLE OBFUSCATORS RENAMING SYMBOLS CONTROL FLOW OBFUSCATION BREAKING DECOMPILATION AND DISASSEMBLY REVERSING OBFUSCATED CODE XENOCODE OBFUSCATOR DOTFUSCATOR BY PREEMPTIVE SOLUTIONS REMOTESOFT OBFUSCATOR AND LINKER REMOTESOFT PROTECTOR PRECOMPILED ASSEMBLIES ENCRYPTED ASSEMBLIES DECOMPILATION NATIVE CODE DECOMPILATION: AN UNSOLVABLE PROBLEM? TYPICAL DECOMPILER ARCHITECTURE EXPRESSIONS AND EXPRESSION TREES CONTROL FLOW GRAPHS THE FRONT END SEMANTIC ANALYSIS GENERATING CONTROL FLOW GRAPHS CODE ANALYSIS DATA-FLOW ANALYSIS SINGLE STATIC ASSIGNMENT (SSA) DATA PROPAGATION REGISTER VARIABLE IDENTIFICATION DATA TYPE PROPAGATION TYPE ANALYSIS PRIMITIVE DATA TYPES COMPLEX DATA TYPES CONTROL FLOW ANALYSIS FINDING LIBRARY FUNCTIONS THE BACK END REAL-WORLD IA-32 DECOMPILATION