Ravi’s Blog

I look at kayak.com, sidestep, mobissimo and wonder if really vertical search is just this. All these services are useful, but to me these sites are a lot closer to meta-search rather than a vertical search.

I wrote before that the vertical search apps are built based on the core search apps. Google has taught us quite a lot about what goes into a successful search app. Google validated that aggregation can be very successful. The siloed model of building proprietary networks and keeping people on the network is not the only way to be successful. In fact, Google proved that aggregation can not only be succesful, but can also be scalable and high margin.

However, to be a complete search app there need to be few other things.
First of these is Comprehensiveness. What good is aggregation, if i still need to go to few other sites? Comprehensiveness doesn’t really mean that you should have every possible item, but there should be enough coverage that 90%+ users donot need to use another site 99% of the time. These numbers are not really scientific, but you get the idea. None of the so called travel vertical searches are comprehensive. They may be aggregating all the booking engines, but booking is just part of my travel planning. They leave out bulk of the travel queries regarding reviews, location selection, hotel selection etc. I consider trip advisor more of a travel vertical search than any of the above.

Then there is Relevance. Since early on, the problem on the web hasn’t been about finding results. It’s always been about finding the ‘right’ results. Google’s competitive advantage has been PageRank more so than anything else. The key is to come up with scalable algorithms for relevance that work across a breadth of queries. Clearly, hiring harmonious editors to optimize search results manually for each term is not scalable.
Let me talk a bit more about relevance. When it comes to vertical search, none of the vertical search apps out there does relevance well. Product search is searching purely on text. I can refine based on other attributes, but the first set of results presented to me are nothing more than simple text match. For example, froogle shows “digital camera” as a sample query. When i click on it, it shows ‘canon powershot A95′ as the most ‘relevant’ result for my query. I can’t really tell what makes that more or less relevant than any other camera on that list. While there might be more real life queries, this illustrates the core issue, that the relevance is mostly text match.
The folks dealing in the search world always hear the words structured, semi-structured, unstructured etc. The general consensus is that structured data is better for searching and refining. However, structured data always leads us to data normalization problem. Every one has structured data, and they all structure it differently. This is by far the most painful problem i have encountered dealing with various feeds. It seems simple enough and easy enough to be solved by standards, yet it isn’t. It repeats in almost all verticals.

There are extensions to normalization problems - like identifying the “product” or “item” uniquely or deduping. For example, how can you tell it’s the same job that got posted on craigslist and hot jobs? Or it’s the same item by the same seller for sale on both yahoo! auctions and ebay?
I personally believe that if a vertical solves the relevance problem well, then they don’t really need to solve deduping.

So, apparently silicon valley is buzzing about vertical search. I have heard about quite a few start ups that do vertical search in some capacity or the other. I am listing a few here. Most of them seem to be focused on jobs. Indeed.com , simplyhired.com, Workzoo, NimbleCat, Fatlens are a few of the companies that are out of the “stealth mode”. There are couple of other companies that are in the stealth mode but tackling similar problems.

Let me tell you, i have had the opportunity to think about this problem in a specific business context for over an year. Ofcourse, thinking does not mean “solving” and i don’t claim to have solutions for any of the challenging problems involved with vertical search.

How do people find what they want on the internet?
Since the begining of internet, the one thing everyone recognized is how vast internet is, how quickly content was/is coming on to it, and how useful/powerful it can be. However, finding what you want is also very hard on the internet and it’s the quintessential problem that is being solved since 1995.

To provide a service that is useful to users, one has to discover all (meaninful portion of) the content on internet, store/index relevant information, answer queries with best information, build right supportive applications around it and all this should be done in a way that you answer the user in a split second.

At that time, crawl/search based solutions were yielding poor results for several reasons. A lot of them are because traditional search algorithms which were applied trusted content (produced by you or someone trusted) are not really useful when you apply them to un-trusted content (produced by various people some with commercial intent). Human review based solutions yielded better results at that time.
The general problem was how do i help users find “anything” on the internet. However, people realized pretty quickly that a lot of specific industries are moving online and they all have different needs. So a slew of applications (vertical) began to come online - shopping, jobs, classifieds, auctions, health, etc.

That generation of applications, let’s say web 1.0 apps - even though i don’t like web 2.0/1.0 thingy - generally solved problems in the same model as the overall internet search.
These vertical apps typically, collect information through feeds or submissions, build search applications focused on that business/vertical, build supportive applications around those. The land grab was on. The best business has the most submissions/feeds and drove most activity through matching finders with offerers.
The key thing i am getting at is - Web 1.0 vertical apps modelled their products/businesses around the web 1.0 core apps.

This web 1.0 model created silos/islands of applications each of them competing with the other for user’s attention.

Enter GOOGLE.
Google approached core web problem, in a search centric way and created algorithms that infer trust (and hence relevance) from web interlinks. They improved relevance of web results, quality of crawling (discovery) and come up with solid technology to serve this information blazingly fast. With google, users could search through more web pages than ever before and get relavant results.

This and quite a few other technological evolutions influence the web world. For the past few years user habits have been changing. Users are lot more search box friendly and search activity is on the rise. Businesses are more accustomed to search based advertising and the overall search application model is thriving. Big players woke to these trends somewhat early and now we have the big 3 competing in the search space.

Why Vertical Search
Knowing what we know now, if we looked at each of these vertical businesses we might solve the same problem differently. The siloed model of applications means that user’s do not have one single place to find anything and everything. People have to look at few (sometimes 10, sometimes 100s) of sites to find ALL the inventory. Websearch as it is right now, is meant to sort out documents and figure out which is the most relevant web document that matches your query. This is a very generic solution. It’s good for a lot of things, but when it comes to specific tasks i have at hand, it has it’s limitations.

For example, I am looking for honda civic 1999-2003 with in 50 miles of where i am. Well, Since car is a transient product, a web page that is listing this car for sale is in existence for a brief period of time (few weeks) and then it goes away. Using google’s PageRank or other similar methods is not ideal to identify most relavant cars for me. This example can be repeated for tons of other tasks, flight tickets, job search, digital cameras :), tickets search, etc.

So, Tom Evslin continues to think that vertical search is humbug. I remain convinced otherwise.

So Jupiter research came out with an expensive research report about ‘vertical search’. The conclusion is that there will be specialized search engines helping you do specific tasks very well. While I don’t neccessarily disagree that vertical search engines may be the thing of the future. I would love to read the analysis that gets Mr.Niki Scevak to that conclusion. My belief that most analysts really have little clue about what they “analyze” and i hope Mr. Scevak does not fall in that category.

I got the summary from Search Views which i found via tom evslin’s blog (which i found via GigaOM). With disclaimers that i haven’t read the actual report, here’s what I make out the content of the analysis to be. Search Big 3 (G, Y & M$) can be likened to TV big 3 (ABC, CBS, NBC) of early days. ABC, CBS, NBC are general purpose networks covering mainstream content. Just like we have sports network, we will get vertical search.

Uh-huh. Tom Evslin has a reasonable re-buttal of the above analogy. I think it makes sense. However, I am not sure if i agree with his conclusion.

To be honest, I think that this already played out on the web - with “Portals”. We started with big web players like AOL, MSN, Y!, Excite and quite a few other players wanting to become the portal of choice for the users. A lot of niche services came up because these portals are unable to focus on specific verticals (Monster, hotjobs, Auto trader, Imdb, Ebay etc). Eventually, the industry consolidated, and now we are left with few players again. Couple of new big players emerged from this shake out (EBay, Google), and some mid sized players (monster, auto trader, expedia etc). A lot of death and destruction as well

The scenario that will play out - IMHO -is very similar to the above. Big 3 are fighting against each other. The content on the web is exploding. The number of users online and time spent online is being relatively flat. The software development models are changing. In this environment, we the big 3 stretched thin and not able to a great job of organizing and present the content the way users need it. This is the opportunity.

How do you collate, organize the vast content on the web, and present in a fashion that makes meaningful sense for the task at hand?

The vertical search folks argue, that by including domain specific knowledge into the search algorithms you can make the experience better. I tend to agree with that.
However, I think the long term trend that plays is something like what happened before.

• A lot of these niche small players come along, focus on verticals and organize information better.
• Some of them get big enough and others fall apart
• The top ones get bought out by Y!, G, and M$.
• This causes excitement, inviting more entrepreneurs to try and solve other problems and VCs to fund companies that solve similar problems
• Eventually, this results in the frenzy that funds companies which do “vertical search for digital cameras”.
• Om Malik will write a blog post that says “I predicted this”
• Most of the vertical search apps will have smaller markets than the current top players, very few (1-2) might emerge with a big market cap. A lot of the small players will get wiped out.
• We will be left with 3-7 top players in the web space.

I definitely think that there is use for vertical search. The experience is really bad searching for jobs, searching for cars, searching the web for financial information, and so forth. Limited resources at G, Y, M$ to focus on all these problems

Quite a few of businesses crawling the web for job postings and building interesting applications. Seems like everyone feels that this space has a lot of room for improvement. Here are couple of crawl/index based job sites i have seen. If you know anymore please post in comments.