MINING THE WEB
TABLE OF CONTENTS INTRODUCTION 2 CRAW LING THE WE B 17 WE B SE A RCH AND INFORMA TION RE TRIE V A L 45 S I MILA RITY AND CLUSTE RING 79 SUPE RV IS E D LEA RNING 125 SE MIS UPE RV IS E D LEA RNING 177 SOCIA L NETW ORK ANA LY S IS 203 RE S OURCE DISCOV E RY 255 T HE FUTURE WE B MINING 289 CH APT ER 1 INTRODUCTION THE WORLD WIDE WEB IS THE LARGEST AND MOST WIDELY KNOWN REPOSITORY OF HYPERTEXT HYPERTEXT DOCUMENTS CONTAIN TEXT AND GENERALLY EMBED HYPERLINKS TO OTHER DOCUMENTS DISTRIBUTED ACROSS THE WEB TODAY THE WEB COMPRISES BILLIONS OF DOCUMENTS AUTHORED BY MILLIONS OF DIVERSE PEOPLE EDITED BY NO ONE IN PARTICULAR AND DISTRIBUTED OVER MILLIONS OF COMPUTERS THAT ARE CONNECTED BY TELEPHONE LINES OPTICAL BERS AND RADIO MODEMS IT IS A WONDER THAT THE WEB WORKS AT ALL YET IT IS RAPIDLY ASSISTING AND SUPPLEMENTING NEWSPAPERS RADIO TELEVISION AND TELEPHONE THE POSTAL SYSTEM SCHOOLS AND COLLEGES LIBRARIES PHYSICAL WORKPLACES AND EVEN THE SITES OF COMMERCE AND GOVERNANCE A BRIEF HISTORY OF HYPERTEXT AND THE WEB CITATION A FORM OF HYPERLINKING IS AS OLD AS WRITTEN LANGUAGE ITSELF THE TALMUD WITH ITS HEAVY USE OF ANNOTATIONS AND NESTED COMMENTARY AND THE RAMAYANA AND MAHABHARATA WITH THEIR BRANCHING NONLIN EAR DISCOURSE ARE ANCIENT EXAMPLES OF HYPERTEXT DICTIONARIES AND ENCYCLOPEDIAS CAN BE VIEWED AS A SELF CONTAINED NETWORK OF TEXTUAL NODES JOINED BY REFERENTIAL LINKS WORDS AND CONCEPTS ARE DESCRIBED BY APPEALING TO OTHER WORDS AND CON CEPTS IN MODERN TIMES 1945 VANNEVAR BUSH IS CREDITED WITH THE RST DESIGN OF A PHOTO ELECTRICAL MECHANICAL STORAGE AND COMPUTING DEVICE CALLED A MEMEX FOR MEMORY EXTENSION WHICH COULD CREATE AND HELP FOLLOW HYPERLINKS ACROSS DOC UMENTS DOUG ENGELBART AND TED NELSON WERE OTHER EARLY PIONEERS TED NELSON 1