Vertical Search - Why?
So, apparently silicon valley is buzzing about vertical search. I have heard about quite a few start ups that do vertical search in some capacity or the other. I am listing a few here. Most of them seem to be focused on jobs. Indeed.com , simplyhired.com, Workzoo, NimbleCat, Fatlens are a few of the companies that are out of the “stealth mode”. There are couple of other companies that are in the stealth mode but tackling similar problems.
Let me tell you, i have had the opportunity to think about this problem in a specific business context for over an year. Ofcourse, thinking does not mean “solving” and i don’t claim to have solutions for any of the challenging problems involved with vertical search.
How do people find what they want on the internet?
Since the begining of internet, the one thing everyone recognized is how vast internet is, how quickly content was/is coming on to it, and how useful/powerful it can be. However, finding what you want is also very hard on the internet and it’s the quintessential problem that is being solved since 1995.
To provide a service that is useful to users, one has to discover all (meaninful portion of) the content on internet, store/index relevant information, answer queries with best information, build right supportive applications around it and all this should be done in a way that you answer the user in a split second.
At that time, crawl/search based solutions were yielding poor results for several reasons. A lot of them are because traditional search algorithms which were applied trusted content (produced by you or someone trusted) are not really useful when you apply them to un-trusted content (produced by various people some with commercial intent). Human review based solutions yielded better results at that time.
The general problem was how do i help users find “anything” on the internet. However, people realized pretty quickly that a lot of specific industries are moving online and they all have different needs. So a slew of applications (vertical) began to come online - shopping, jobs, classifieds, auctions, health, etc.
That generation of applications, let’s say web 1.0 apps - even though i don’t like web 2.0/1.0 thingy - generally solved problems in the same model as the overall internet search.
These vertical apps typically, collect information through feeds or submissions, build search applications focused on that business/vertical, build supportive applications around those. The land grab was on. The best business has the most submissions/feeds and drove most activity through matching finders with offerers.
The key thing i am getting at is - Web 1.0 vertical apps modelled their products/businesses around the web 1.0 core apps.
This web 1.0 model created silos/islands of applications each of them competing with the other for user’s attention.
Enter GOOGLE.
Google approached core web problem, in a search centric way and created algorithms that infer trust (and hence relevance) from web interlinks. They improved relevance of web results, quality of crawling (discovery) and come up with solid technology to serve this information blazingly fast. With google, users could search through more web pages than ever before and get relavant results.
This and quite a few other technological evolutions influence the web world. For the past few years user habits have been changing. Users are lot more search box friendly and search activity is on the rise. Businesses are more accustomed to search based advertising and the overall search application model is thriving. Big players woke to these trends somewhat early and now we have the big 3 competing in the search space.
Why Vertical Search
Knowing what we know now, if we looked at each of these vertical businesses we might solve the same problem differently. The siloed model of applications means that user’s do not have one single place to find anything and everything. People have to look at few (sometimes 10, sometimes 100s) of sites to find ALL the inventory. Websearch as it is right now, is meant to sort out documents and figure out which is the most relevant web document that matches your query. This is a very generic solution. It’s good for a lot of things, but when it comes to specific tasks i have at hand, it has it’s limitations.
For example, I am looking for honda civic 1999-2003 with in 50 miles of where i am. Well, Since car is a transient product, a web page that is listing this car for sale is in existence for a brief period of time (few weeks) and then it goes away. Using google’s PageRank or other similar methods is not ideal to identify most relavant cars for me. This example can be repeated for tons of other tasks, flight tickets, job search, digital cameras :), tickets search, etc.
So, Tom Evslin continues to think that vertical search is humbug. I remain convinced otherwise.
March 26th, 2005 at 6:35 pm
Nice post Ravi. Your comment about ’supportive applications’ is right on target, and your reasoning is very sound for why some content/data (tho not all) is better served by vertical search.
I work for SimplyHired.com, one of the companies you mentioned, and you’ve picked up on a lot of the issues we’ve been thinking about.
Indeed, Workzoo, & Nimblecat have all been doing pretty well on job search to date. We’ve probably got a little bit of catching up with them to do on a few features (though not on the breadth of data), but we hope to offer some unique stuff of our own in the next month or two.
One thing we do think is important and perhaps a little different than those guys is that we think the UI should be pretty simple, and advanced features should be layered on carefully. Also, we enjoy having a little fun with our site here & there, and we hope the voice we’ve chosen is an enjoyable one to read.
anyway, i agree with you that supportive apps are important…
(btw, interested in a job?
- dave mcclure
May 25th, 2005 at 9:52 pm
Ravi:
I agree with Dave’s comment - there is a lot of content out there that isn’t worthy of vertical search. Heck - it isn’t even worthy of inclusion in the search engines (Google’s sandbox effect is trying to weed the spammers out)
I also agree with what you said - it’s going to be nearly impossible for people to get a complete view of the inventory of content online if they were only to turn to the big search engines. This is the main focus of the Vertical Search market - providing a complete view.
The Vertical Job Search Engines out there are beginning to understanding how to gather, organize, and disseminate content from all their different providers. The end goal - to give job seekers a one stop shop for job searching.
But, I would argue that the crawling model which Google and other traditional search engines is applicable to the Vertical Job Search Market.
Our company, Fetchster.com, is a Minnesota Job Search Engine which uses crawling technology to spider the employment portals of Minnesota companies and provides those results in a web-searchable database.
In fact, you’ll find many of the same Minnesota listings on Fetchster as you do on SimplyHired and Indeed.
Despite showing that crawling technology works in this space, we still have a long way to go. We think our method provides the most accurate, interesting, and unique job results. But that’s not to say that we’re right, or that any other player in the industry is right or wrong. The VJSE market is very young, but one thing’s for sure - there are a lot of great innovators out there, and that’s going to benefit the job seeker in the long run.
Adam Gedde
Fetchster.com
November 10th, 2005 at 5:12 am
Hi Ravi,
I dont know if I am qualified enough to speak to ur mails but being a true passionate Computer science student, its my duty to let the knowledge flow properly and especially in the feild of search engines since i have given 4 years of my life to it.
I am surprised to see this euphoria over the invent of vertical search engines and the kind of facilities it provide to specific searches but one thing little thing that is missed by all of u is the bottle neck it brings to the inquisitiveness that researchers have in the feild of AI implementation.
An ideal search engine is neither a vertical-centric search engine(Indeed.com , simplyhired.com, Workzoo, NimbleCat, Fatlens) nor a pattern-centric search engine(google,yahoo,altavista etc) but a context-centric search engine.
Let talk technical now. What google does by its page-ranking is that it evaluates the confidence value that the site holds for the data it contains and displays it on the screen, which sometimes gives very relevent answers, but the success is not credited to the fact that the site had relevent data but to the fact that its linked more than others. CRAZY!!. Image yourself developing a website that has the most relevent data for a topic ever written but it would not be visible to the ranks until it been linked to enough sites. The context is lost.
Basically the flaw in all the search engines including the vertical ones is that its pattern centric. Relevent is drawn by every logic other than the most basic one, CONTEXT.
I have been working on context-search engines and hope would give u better insight asap.
Thanks anyways,
November 10th, 2005 at 8:14 pm
kruttik,
google, yahoo and other major search engines use context. However, the current context gleaning algorithms are very susceptible to spam. Thus, the refinement of pagerank. PageRank was innovative because it made it very difficult to spam. However, these days SEOs figured out how to spam PageRank. There are variations called TrustRank etc, that are used to associate authenticness to document content.
So, Every search engine starts with context. Unless you have a major break through in distinguishing spam from relevant content, just based on document info, I’d not be optimistic on relying document content alone.
September 2nd, 2007 at 9:02 am
Runescape Money Blog
I have written a followup to this on my blog. Feel free to check it out if interested. -Louis— This post was created by Comment Poster. Want to post thousands of comments to blogs instantly and increase your visitors exponentially? Check it out he…