From Thinking Machines to thinking about the future of search
Neil Budde is Editor in chief of Yahoo! News, Finance and Sports, and a member of the iFOCOS Search Working Group.
I first became intrigued by search technology when I joined Dow Jones in 1987. The visionary leader of Dow Jones Information Services, Bill Dunn, had just convinced the company to purchase a pair of Connection Machine computers from Thinking Machines Corp. The promise: that the natural language searching and relevance feedback capabilities of the massively parallel computers could open up the vast databases of Dow Jones News/Retrieval to a huge audience of business information users and move the business outside the library and information specialist marketplace.
Of course, the timing was all wrong: the technology wasn’t capable of searching sizable collections of information fast enough and the dial-up command-line interfaces of the time weren’t suitable for this form of searching. Fast forward to today and you find much larger amounts of information being indexed and searched in milliseconds. Yet we still face many of the same problems: users who enter 1-2 word queries and then dig only so deep into the results as well as interfaces that still don’t make search that simple. What is the answer? I’m looking forward to searching for it.
tags:1 Comment so far
Leave a reply
Mr Budde’s comments are so apt. Quite probably, the future of search is oriented towards being domain specific. Already there are search engines dedicated to pharma and healthcare. Or there is knuru which specialises in business and management. And so on. Underlying this is possibly the perception that natural language processing will improve with better usage which can only emerge from sustained searches by many users, who understand what they are looking for and keep improving their search. Of course, I am relying on the notion of ‘wisdom of the crowds’ that is presumed to govern search, except that the crowd here has a sophisticated understanding of what it is seeking. And that goes into making the search more productive. Perhaps, the taxonomy is easier to manage since we will be dealing with a closed domain. Perhaps also, there is the notion of learning by the machine. Jeff Bezoz pointed to an amazing suggestion that their system made to people who were searching for books on Zen: it suggested books which dealt with how to keep a clutter free desk, a connection that no human editor would have seen. If such suggestions can be incorporated into search, we might see more appropriate results, allowing for the possibility that the system falls for spurious coincidences.
Meanwhile, in Google for example, even ‘exact phrase’ often yields many unconnected results, leading to the doubt that the search engine is not really capturing ‘exact phrase’. Years ago, when I used to actively search Euromoney.com’s exact phrase in their structured query, I invariably got very good results. Knuru, for instance, produces better results if you are looking for business information, but is still far from being good enough. Those who seek academic writings will find ‘jstor’ a more efficient option. As an active search user, I have seen vastly improved results when I modify my searches to accommodate many possibilities, leading me to think that it is partly a question of what connections we see in a topic.