This blog post is written as an assignment for a Technology & e-Business class taught at the Nanyang MBA.

A lot of competition is going on in the search market. Google has dominated this industry over the last decade. Yahoo and Microsoft both tried to beat the company from Mountain View with their own offerings. Smaller companies such as Cuil have tried to capture a part of the enormously profitable market as well. Let’s have a look at a few search engines that everyone talks about to see if scale matters in search.

The largest user base

Google is today’s most successful search engine. The company won the first “search engine war” because it was able to deliver the best search results when it mattered.

With the help of PageRank and an uncluttered interface Google was able to deliver much better search results and a better overall search experience when it first became available. The knowledge about this new search engine with the silly name spread virally. And to remain at the top Google started a continuous improvement process. The software and platform powering Google’s search engine and many services are analyzed day in and day out and with the help of multivariate testing and other methods and improved based on the results. Google capitalizes in its user base to stay on top and grow even further.

Google was the best search engine when it mattered, it had a sufficiently large index, it assembled an incredible platform to power its systems, and it has grown an enormous user base. But will it be able to stay on top?

The largest Index

Cuil (pronounced like cool) was one of the more promising companies that tried to compete with Google. In 2008 the company was launched with a lot of hype. The buzz generated around the company, which was founded by former Google employees, promised a search engine that should have been much better than Google’s. The company touted that its index of website was much larger than Google’s and its search engine therefore delivers better results. The rest is history, the meager search result quality (in the beginning) resulted in no one using Cuil today.

Cuil wasn’t able to capitalize on its (allegedly) larger index and failed. A large index alone obviously isn’t enough to compete with current search engines. Also Cuil was able to build this index far cheaper than Google but it obviously impaired the quality of the results delivered

The largest competitor

Microsoft was online very early but it still missed the importance of this paradigm shift. The company grew large with Software created for the Desktop. Windows and Office made MS successful and rich. On the web it has been a slightly different story. Even though Microsoft’s web properties are fairly popular its search engines never could capitalize on this.

Microsoft has had many goes at creating a search engine that could beat Google’s and its newest try, Bing, is (at least partly) based on technology developed by Powerset which was acquired by Microsoft in 2008.

Bing shapes up to be Microsoft’s most successful search engine so far but it is nowhere near Google’s market share. It is an uphill battle for Microsoft. Microsoft just announced that it will be powering Yahoo’s search, a clear sign to Google, and the company hopes that the larger user base will help it to improve its search further to be able to continuously erode Google’s market share.

Being a humongous corporation and having almost unlimited funds so far hasn’t been enough. In general, Microsoft’s size definitely is an advantage because of the resources it can put into a project. But Microsoft’s size also has put it on the radar of competition watch dogs. They inhibit the company to use its size efficiently in promoting the engine through its other offerings (e.g. Windows or Internet Explorer), if this is a good or a bad thing isn’t part of this post.

The largest Portal

On February 1^st 2008 Microsoft offered to buy Yahoo for $44 billion and threw the smaller rival of its feet. Yahoo itself probably doesn’t know what it wants right now. First it was a link-list, then a search engine powered by Google, then a search engine powered by Yahoo itself, and now a search engine powered by Bing.

Yahoo had been at the forefront of development of the Internet by doing almost everything right before the offer. The exodus of key personnel and the ever increasing insecurity about Yahoo’s future made the company lose its game. One CEO later Yahoo tries to focus on its core competency which the company defines as Content. Yahoo’s portal always had been very successful and its website is one of the most visited on the net. Its search engine’s success on the other hand had been minor for a long time. Yahoo wasn’t able to catch a substantial part of the worldwide search market anymore.

Yahoo had and has the eyeballs but its search engine still wasn’t able to succeed. Just having a large user-base isn’t sufficient.

So, does scale in search matter?

Scale affects many parts of search. Scale is a factor in the quality of search results as well as the the amount of money that can be made.

As I just pointed out there is scale in:

• The user base
• The index size
• The resources available (infrastructure and people)
• etc.

Scale definitely is part of the search business. Only by indexing a large amount of websites is it possible to deliver good result.  Good results attract users and only if you have enough visitors can you make enough money to support an infrastructure which enables you to build a large index.

There now are different approaches such as a peer-to-peer based search engines that try to defy the paradigm presented before (see here & here) but it remains to be seen if taking out the infrastructure cost will be sufficient to compete with Google’s and Microsoft’s armies of engineers.

For now I have to disagree with Hal Varian! Scale is very important for a search engine to be successful.