INTELLIGENT DOCUMENT RETRIEVAL
CONTENTS 1 INTRODUCTION 1 1 1 INTRODUCTORY EXAMPLES 4 1 2 USING MARKUP TO EXTRACT KNOWLEDGE 8 1 3 APPLYING THE EXTRACTED KNOWLEDGE 15 1 4 STRUCTURE OF THE BOOK 17 PART I THE MODEL 2 RELATED WORK 23 2 1 INFORMATION RETRIEVAL 24 2 2 INFORMATION EXTRACTION 26 2 3 CLUSTERING 27 2 4 CLASSI CATION 29 2 5 WEB SEARCH TECHNIQUES 31 2 6 ONTOLOGIES 34 2 7 LAYOUT ANALYSIS 36 2 8 WEB SEARCH STUDIES 36 2 9 NAVIGATING CONCEPT HIERARCHIES 38 2 10 DIALOGUE SYSTEMS 41 2 11 USABILITY ISSUES 42 2 12 CONCLUDING REMARKS ON RELATED WORK 43 3 DATA ANALYSIS AND DOMAIN MODEL CONSTRUCTION 45 3 1 DOCUMENTS 45 3 2 CONCEPTS 47 3 3 A DOMAIN MODEL BASED ON CONCEPTS 51 3 4 MODEL STRUCTURE 53 3 5 MODEL CONSTRUCTION 54 3 6 USING THE MODEL FOR QUERY MODI CATION 58 3 7 IMPLEMENTATIONAL ISSUES 60 4 INCORPORATING ADDITIONAL KNOWLEDGE 63 4 1 INTERNAL KNOWLEDGE 63 4 2 EXTERNAL KNOWLEDGE 67 5 A DIALOGUE SYSTEM FOR PARTIALLY STRUCTURED DATA 69 5 1 DIALOGUE AS MOVEMENT IN SPACE 70 5 2 DIALOGUE EXAMPLE 71 5 3 STATIC VS DYNAMIC CLUSTERS 73 5 4 REAL USER QUERIES 73 5 5 PROPERTIES 75 5 5 1 DOCUMENT PROPERTIES 76 5 5 2 SYSTEM PROPERTIES 76 5 5 3 GOAL DESCRIPTION 77 5 6 DIALOGUE 78 5 6 1 HIGH LEVEL DIALOGUE STATES 78 5 6 2 LOW LEVEL DIALOGUE STATES 80 5 6 3 CONSTRUCTING POTENTIAL CHOICES 85 5 6 4 DIALOGUE STRATEGIES 89 5 6 5 CUSTOMIZATION 89 PART II PRACTICAL APPLICATIONS 6 UKSEARCH INTELLIGENT WEB SEARCH 93 6 1 INDEXING WEB PAGES 94 6 2 THE UKSEARCH SYSTEM 98 6 2 1 INDEXING AND MODEL CONSTRUCTION 100 6 2 2 DIALOGUE STRATEGY 102 6 3 SAMPLE DOMAIN 1 ESSEX UNIVERSITY 107 6 3 1 INDEX TABLES 108 6 3 2 DOMAIN MODEL 109 6 3 3 CONCEPTS VS REAL USER QUERIES 111 6 4 SAMPLE DOMAIN 2 BBC NEWS 112 6 4 1 INDEX TABLES 115 6 4 2 DOMAIN MODEL 116 6 4 3 ADJUSTED DIALOGUE STRATEGY 117 6 5 IMPLEMENTATIONAL ISSUES 117 7 UKSEARCH EVALUATION AND DISCUSSION 121 7 1 LOG ANALYSIS 121 7 1 1 SYSTEM SETUP 122 7 1 2 RESULTS 124 7 1 3 DISCUSSION 125 7 2 INVESTIGATING DOMAIN MODEL RELATIONS 125 7 2 1 TASK AND SETUP 125 7 2 2 RESULTS 127 7 2 3 DISCUSSION 128 7 3 TASK BASED EVALUATION ESSEX UNIVERSITY 129 7 3 1 SEARCH TASKS 129 7 3 2 EXPERIMENTAL SETUP 133 7 3 3 PROCEDURE 134 7 3 4 RESULTS 134 7 3 5 DISCUSSION 140 7 4 TASK BASED EVALUATION BBC NEWS 141 7 4 1 SEARCH TASKS 142 7 4 2 EXPERIMENTAL SETUP AND PROCEDURE 143 7 4 3 RESULTS 143 7 4 4 DISCUSSION 151 8 YPA SEARCHING CLASSI ED DIRECTORIES 157 8 1 SYSTEM OVERVIEW 158 8 2 INDEXING CLASSI ED ADVERTISEMENTS 159 8 2 1 STRUCTURE OF THE BACKEND 160 8 2 2 DOMAIN MODEL CONSTRUCTION 161 8 3 DIALOGUE STRATEGY IN THE YPA 162 8 3 1 PROPERTIES 165 8 3 2 DIALOGUE SETUP 166 8 3 3 DIALOGUE FUNCTION 168 8 3 4 CALCULATION OF POTENTIAL CHOICES 168 8 4 IMPLEMENTATIONAL ISSUES 171 9 FUTURE DIRECTIONS AND CONCLUSIONS 173 9 1 TOWARDS EVOLVING DOMAIN MODELS 173 9 2 DIALOGUE MANAGEMENT 176 9 3 AN OUTLOOK ON FUTURE EVALUATIONS 177 9 4 CONCLUSIONS 178 REFERENCES 181 1 INTRODUCTION THE ISSUE OF EXTRACTING THE STRUCTURE OF SOME TEXT IS A CHALLENGING ISSUE 1 WE ARE WITNESSING A MASSIVE GROWTH OF ELECTRONIC NATURAL LANGUAGE RE SOURCES MOST NOTICEABLE IS THE DEVELOPMENT OF THE WEB WITH ONLINE NEWSPA PERS PRODUCT CATALOGUES DATA ARCHIVES ETC MILLIONS OF USERS ACCESS THE WEB OR OTHER ELECTRONIC DOCUMENT COLLECTIONS EVERY DAY IN THIS BOOK WE LOOK AT A SINGLE ASPECT OF THIS RATHER COMPLEX AREA HOW CAN WE HELP A USER TO NAVIGATE A DOCUMENT COLLECTION EASILY AND HOW CAN WE ASSIST A USER WHO WANTS TO SEARCH A COLLECTION FOR DOCUMENTS THAT SATISFY SOME INFORMATION NEED WE WILL NOT LOOK AT GENERAL WEB SEARCH BUT INSTEAD WE WILL CONCENTRATE ON SMALLER COLLECTIONS SUCH AS WEB SITES OR COLLECTIONS OF CLASSI ED ADVERTISEMENTS THEY REPRESENT MUCH NARROWER DOMAINS UNLIKE THE BROAD COVERAGE OF THE WEB ONE REASON FOR CONSIDERING THIS AREA A WORTHWHILE RESEARCH ISSUE IS THE FACT THAT SEARCHES IN DOCUMENT COLLECTIONS OFTEN RETURN EITHER LARGE NUMBERS OF MATCHES OR NO SUITABLE MATCHES AT ALL WE ACKNOWLEDGE THAT WEB SEARCH ALGORITHMS HAVE MATURED SIGNI CANTLY OVER THE PAST FEW YEARS AND THAT A SEARCH REQUEST SUBMITTED TO GOOGLE1 TYPICALLY RETURNS EXCELLENT MATCHES FOR A USER QUERY NEVERTHELESS THIS IS NOT ALWAYS THE CASE IF THE COLLECTION IS ONLY A FRACTION THE SIZE OF THE WEB AND THE DOCUMENTS COVER A MUCH SMALLER RANGE OF TOPICS SUCH COLLECTIONS ARE VERY COMMON IN INSTITUTIONS UNIVERSITIES OR COMPANIES SOME EXPLICIT KNOWLEDGE ABOUT THE DOMAIN I E A DOMAIN MODEL COULD BE USEFUL TO HELP THE USER ND THE RIGHT DOCUMENTS A DOMAIN MODEL COULD ENCODE RELATIONS BETWEEN WORDS OR PHRASES THAT HAVE BEEN EXTRACTED BY ANALYSING THE DOCUMENT COLLECTION THE GRAPH IN FIG 1 1 GIVES AN IDEA OF HOW SUCH A MODEL MAY BE STRUCTURED WE SEE A TREE OF RELATED TERMS THAT CAN BE USED TO EITHER ASSIST A USER IN THE SEARCH PROCESS PERFORM AUTOMATIC QUERY RE NEMENTS OR ALLOW THE USER TO BROWSE THE COLLECTION NOTE THAT THE TYPES OF RELATION IN THE SAMPLE TREE ARE NOT FORMALLY SPECI ED THIS IS SIGNI CANTLY DI ERENT FROM 1