7 Comments

Algorithms + Human Editors = Smarter Search

Oct 26, 2006
by

I’ve spent the last few days playing with the Google Custom Search Engine aka Google Coop, and it’s opened my eyes to a whole new world of possibilities. Vertical search engines powered by the collective intelligence of smart people will create a search experience that will make today’s search results look second rate.

Google has made it quick and easy to create your own custom search engine. Simply fill out a two page web form and you have your own customized search engine running high-octane Google under the hood. There are several featured examples on the Coop site. Affiliate Classroom has already developed a vertical search engine for affiliate marketing.

User generated content has been fueling the rapid growth of Web 2.0 companies over the last 2-3 years. The rise of community-based vertical search will add even more fuel to the fire. Google Coop could improve search the same way Firefox improved the browser by harnessing the power of an active and passionate user community.

Google is smart to let publishers monetize Google Coop through AdSense. The financial incentive combined with the desire to improve search results will continue to drive innovation. However, Google will have to take the necessary steps to detect and remove community spam and fraud, otherwise, there will be an unusually high ratio of sites based on keywords like “mesothelioma” and other expensive search phrases.

Google could make custom search even better by adding more social elements like user ratings and reviews. I would also like to see them add some more user-defined search options like Google Base, which would allow us to refine search results even further.

The underlying technology powering vertical search platforms will continue to remain proprietary for the foreseeable future. Building and maintaining a hardware infrastructure to support a search index costs millions of dollars, which will prevent an open source engine like Nutch from gaining mass appeal. However, the open source community is very innovate – I could see somebody building a distributed computing environment that could support such an endeavor. As soon as someone pairs a money-making business model with this idea, open source search could become feasible.

Filed under: Internet Strategy
  • Jonathan (Trust)

    Nutch? At first look, it looked like a domain parked page and at the bottom says Domain Parking.

    As far as custom search engines, it gives another toy for webmasters to play with but people will always use the main search engines for their primary searching. Algorithms + Human Editors = Problematic Search. I've seen what happens when you let human editors at it, usually problems.

  • http://www.quityourdayjob.com Jeremy Palmer

    I screwed up on the nutch link… I just updated it. Here is the correct link:

    http://lucene.apache.org/nutch/

    I spell checked, but I forgot to double-check the links. Thanks for letting me know.

    Best,

    Jeremy

  • http://www.quityourdayjob.com Jeremy Palmer

    "Algorithms + Human Editors = Problematic Search. I've seen what happens when you let human editors at it, usually problems."

    No doubt people will come up with ways to manipulate human edited search results (e.g. Wikipedia/ODP), but people do the same thing to algorithms unabashedly.

    The same open source community that built the tools you and I use every day could easily collaborate on quality vertical search engine projects. Humans are also better at detecting spam than algos.

  • http://www.developersllc.com Devendra

    its a neat tool – google needs to relax the formatting contraints though; the results look like the adsense for search results e.g.

    here is a video game search engine I quickly threw together.

    Video Game Search Engine

    One thing is for certain, if they make it more flexible it will blow up a lot of vertical search models that vcs have invested in. e.g. Wellington Partners portfolio company

    wazap

    which is about to launch in English.

    bet those guys arent happy.

  • http://www.seostraighttalk.com SEO Straight Talk

    Algorithms + Human Editors = Spam Alert too…

    If you give page ranking control to publishers, you give them incentive to place biased results on top of relevant web pages. Watch out for self serving verticals.

    What's really going to happen? Over the next 6 to 12 months we'll see a handful of Google customized vertical search engines rise to popularity with a search market share capping at about 2% (0.5% would be a huge success). If Google search 2.0 (search user generated content)expands beyond just the small popularity set, you can bet advertisers will be given an option to syndicate ads to selected search engine verticals (or sites) and pay a premium for doing so.

    Will be interesting.

  • http://www.seostraighttalk.com SEO Straight Tak

    Agreed.

    Let me clarify. By 2%, i'm referring to the collective sum of google verticals.

    That said – 0.5% would be phenomenal for all Google vertical search share…

  • http://www.bessed.com Adam Jusko

    Custom search is a neat tool for some, but in general it's just a way to slice the bigger Google engine into a niche of interest or to do a site search, something that Google already offered. But it's still Google, which means it's not really a different animal. What if you want to search a certain selection of sites but Google has cut off the number of pages that are indexed for a certain site? You will only get results that Google has deemed relevant, not a complete search of the site in question.

    Nice, but not revolutionary.