When we let users loose to go anywhere they wanted on the Web, they went to a search engine 88 percent of the time. Only in 12 percent of cases did they go straight to a Web site that they hoped would help them with their problem. For example, when the task was to research vacation travel to Mexico, only a few users went to a travel site that they already knew and trusted, such as Expedia or Travelocity. The vast majority employed a search engine to help them find sites.

The fact that search engines have become the dominant tool for users looking for solutions is clearly why search advertising has become such a profitable business. Search ads are the main way you get in front of prospective customers at the exact moment they are looking for new vendors.

Increasingly, the Internet user experience is becoming one of dipping a toe into Web sites rather than truly "visiting" them. Using search engines as their Web interface, people often grab query-related nuggets from sites without engaging with the sites themselves.

The search engine has always been an important tool for users. In 1994, when we were trying to understand why people used the Web despite its lousy usability, we asked everyone who came by our lab two questions: What are you doing online? What are your favorite sites? Their answers were strikingly diverse. People's pursuits ranged from golf to knitting to Linux to military history, and their favorite sites varied just as widely. In fact, the only commonality among them was that all users named a search engine among their top sites.

The conclusion was clear: The Web's strength comes from narrowly targeted sites that provide users with highly specialized information that they need or care about passionately. It was also clear that Search was a hugely important general-interest service because even back when the Web had only 30,000 sites, locating specialized ones was nearly impossible without help.

The Rise of "Answer Engines"

The study we conducted for this book confirmed our early conclusion: Users pursue their own idiosyncratic goals and depend on a generic serviceSearchfor guidance. What has changed: Rather than looking for sites to explore and use in depth, users now hunt for specific answers. Search engines have essentially become "answer engines."

A major change over the years has been a decline in using Search to identify good sites as such. People are looking for answers. The Web as a whole has become one agglomerated resource for people who use search engines to dredge up specific pages related to specific needs, without caring which sites supply them. The job of search engines is no longer resource discovery but to answer questions.

It's a testament to the Web's growth that users now view it as integrated whole. This is good news for search engines and users but not for Web sites.

This changing behavior is explained by the theory of "information foraging": The easier it is to track down new resources, the less time users spend at each resource. Thus, improvement in Search quality over time is driving the trend toward answer engines. Always-on connections have a similar effect, because they encourage information snacking and shorter sessions. Finally, Web browsers' despicably weak support for bookmarks/favorites has contributed to the decline in users' interest in building a list of favorite sites.

It's a testament to the Web's growth that users now view it as integrated whole and don't bother with Web sites; they assume that anything they want to know is available somewhere. They just have to ask. "Web sites" weren't really a tangible concept until 1993 anyway. The pre-Mosaic Web in 1991 and 1992 was exactly that: a web of information where the fundamental unit was the article, not the server hosting a particular Web page. So this new user behavior is actually a reversion to the Web's original vision to some extentthough not completely because users still have some favorite sites that they treat as resources.

For search engines, becoming the user interface to the Web's embarrassment of riches is good news. It's also good news for users, who can find answers by visiting a few search hits rather than enduring the obscure design and poor navigation found on many sites. But is this good for Web sites? Unfortunately not. There is very little value in giving answers to users who don't know or care who provides the service.

E-commerce sites are something of an exception, because they often get a sale from users dipping a toe into their catalog. E-commerce sites differ from other Web sites in having confirmation and fulfillment stages that follow up on users' initial visits, and these steps can also grow the site's mindsharethe likelihood that the site will come to mind when the user is thinking about the type of products or services it offers. Thus, closing the first sale is one of the most important drivers of subsequent e-commerce sales.

It would be self-defeating for e-commerce sites to refuse shopbots (software that searches the Web for a product's lowest prices), prohibit deep links, or employ other tricks that require users to enter at the homepage and spend time navigating the site. Any barrier between the customer and the product translates into lost sales.

Four Ways to Grab Value from Search Engine Visitors

There are some tactics you can use to gain value for your company from users who dive into your site from a search engine. Unless you use at least one of these four, visitors may only see one or two pages on your site and never know anything about your company.

Offer flytrap content that attracts users by providing narrowly focused pages with clear answers to common problems. These pages should perform well in Search Engine Optimization (SEO), so remember to write clear headlines.

Embellish the answer with rich "See Also" links to related content and services. Global navigation won't do the trick because answer-seekers will ignore it. Remember, they are not interested in your site. But contextual links will make the most eager users dig deeperand the eager people are the ones you'll want to keep as prospects for your for-pay services. "See Also" links can be embedded or placed at the end of the article, where they serve as a follow-up call to action. The latter gives you the opportunity to let people know that you're actually selling something and not just handing out free information.

Go beyond pure information and provide analysis and insight, preferably from a unique perspective and with a striking personality that supports your positioning. A percentage of users will appreciate your perspective and want more, even after they've found the answer to their immediate question.

Publish a newsletter with additional tips and useful information. E-mail newsletters set up a relationship with users, offering a more personal experience than page viewing.

Even sites that don't sell must accept the trend toward users' answer-seeking behavior. Walling yourself off from the Web's web-like nature won't solve the problem. Tracking numbers of unique visitors is now irrelevant. Most such visitors are sampling a single page to get an answer, not engaging with your site. Instead of tracking them, count loyal users as a key measure for site success.

Organic vs. Sponsored Links

Search engines typically present two different kinds of hits on the SERP: organic listings and sponsored links. The organic listings are those that were naturally found on the Web as the best matches with the user's query, according to whatever algorithm is used by the search engine. Supposedly the organic listings are "pure" and noncommercial to the extent that the site at the other end of each listing didn't pay to be listed and won't be charged if the user clicks on an organic link.

The sponsored links are quite the opposite: They are nothing but advertisements, even though most search engines persist in employing the euphemism "sponsored" in referring to them. Sadly, many less technical users don't understand what "sponsored" means in this case and don't recognize that these links are simply ads.

How People Use the Search Engine Results Page

The search engine results page is usually referred to as a SERP. It's fitting that this term is rarely used in plural because most users don't see more than one SERP per query. In 93 percent of searches, the users in our study only visited the first SERP, which usually held ten search results plus a number of ads. In only seven percent of cases did users page on to a second SERP, and the number who visited three SERPs for a single query was too small to provide a firm estimate, but it was likely less than one percent.

Not only did most users make do with a single SERP; most of them didn't even bother reviewing the entire page. Only 47 percent of users scrolled the first SERP, which means that 53 percent saw only those search hits that were "above the fold." (Originally a newspaper term, "above the fold" refers to what part of a Web page is visible on a screen without scrolling.) On the most widely used search engine, Google, users can only see four or five results above the fold, if they view the page on the most common screen resolution of 1024-by-768 pixels, like those in our study. In addition to the four or five "organic" search results, there are typically six or seven advertisements above the folda total of ten or so items to choose from.

This table shows how often users clicked the search hits at different placements on the SERP list. Since the first SERP only displays ten organic links, users needed to go to the second SERP to click links number 11 or higher. Of the seven percent who actually visited the second page, only five percent clicked therea discrepancy explained by the fact that a few users who visited the second page returned to the first page. It's no big surprise that the top links get more clicks than bottom links, but it may be surprising to see just how dominant the No. 1 spot is, with more than half of all the clicks.

Number One Guideline for Search Engine Optimization

Aim for the top spot in the listings for all the important keywords your users are likely to search for. (Unfortunately, this is an easy guideline to state but a hard one to achieve.) If you don't get the top listing, second or third is certainly good as well. The farther down the list, the less likely your chance of being seen, but if you have a choice between being ninth or tenth, pick the bottom spot on the page, as there is some prominence to being the last element in a list.

If you can't get a decent organic ranking for important query terms, seriously consider running search ads for those terms.

Using Keyword Pricing to Estimate Usability Improvements

It is very expensive and time consuming to conduct broad usability studies like ours of a wide range of Web sites. We can't do it every year, so we can't accurately track usability improvements on a steady basis through direct measurements. Fortunately there is an indirect measurement that's more feasible to acquire and that can be used as a proxy for Web site usability: the prices paid for keyword advertising on major search engines.

Over time, the price for keyword advertising has shown a distinct upward trend. Future increases in keyword bids will be tightly related to improvements in Web site usability.

On most major search engines, keyword advertising works as follows: First a company must consider how much it is worth paying to try to attract a visitor who has searched for a certain keyword and is presumably interested in a product or service associated with that word. The company then bids for each keyword, and the search engine usually displays the ads from the top eight bidders. (There are many twists to search engine advertising, but the basic principle is an auction in which a limited number of slots are sold to the highest bidders.)

For example, our own company, Nielsen Norman Group, usually pays 31 cents per user who searched for "usability training" when that user clicks through from the search engine to the page for our annual usability conference. Similarly, a company that sells vacation packages to Mexico would be willing to pay a substantial amount for users who are searching for "vacation Mexico" or "hotels Yucatan."

Over time, the price for keyword advertising has shown a distinct upward trend. Over the long term, increases in keyword bids will be tightly related to improvements in Web site usability. During the next several years, keyword prices will increase substantially as more companies realize the benefits of search engine advertising. Search ads are the best way to promote a Web site because keyword matching attracts users who are looking for the exact thing you're offering.

Currently most Internet marketing managers are clueless about keyword advertising and spend most of their budgets on advertising techniques that were appropriate in old media but don't work in an interactive medium like the Web. But one of the beauties of the Web is its accountability: You can track the results of advertising expenditures in terms of both click-throughs and the extent to which clickers turn into spenders. Because of this, even clueless managers will gradually reallocate their budgets to the best performing promotions, which will most often be search advertising.

How To Determine the Optimal Bid for a Search Keyword Ad

We say in our discussion of keyword advertising that a company should increase its bids for search keywords as long as it can make more money from the average new visitor than it is paying per click. This is a nice simplification that makes a complicated argument easier to follow, but it's not the best way of actually determining what your bid should be.

In real life, you should maximize your profits, not simply do as much business as possible. For example, let's say that your Web site makes an average profit of $2.00 for each new visitor who arrives after clicking on a certain keyword. (Let's also say that you have a conversion rate of one percent, so the $2.00 profit per visitor was derived from a profit of $200 per buying customer, since only one of every hundred new visitors becomes a customer.)

Your earlier experimentation has established that you will gain the following traffic for each of three possible advertising bids for a keyword that's highly targeted for your product:

Bidding $1.00 places your ad third from the top and generates 500 visitors per month

Bidding $1.50 places yours as the second ad from the top and generates 1,500 visitors per month

Bidding $1.90 places your ad on top and generates 2,000 visitors per month

How much should you bid? Certainly, you could afford to bid $1.90 because you would still make 10 cents per visitor for a profit of $200 a month. On the other hand, bidding $1.00 would gain you a full dollar's profit per user, for a total monthly profit of $500 from the 500 users who were attracted by a third-place ad. Finally, bidding $1.50 will gain you monthly profits of $750. This is obviously the preferred outcome because it has the highest profit margin.

In this example, you are better off doing without the 500 extra users who would be attracted by the top ad but who would not click on a second-position ad. These users are simply not worth the higher cost of the top spot. Users who click on lower-positioned ads are often worth slightly more to companies than users who click on top ads because they tend to be more actively committed to getting a solution to their problem.

Of course, you must gather the detailed data for your customers from your own Web site. In general, though, the main guideline is simple: The higher the worth of a new visitor, the more you should bid to attract them to your site.

As more and more companies discover the high return on investment (ROI) of search advertising, keyword prices will be bid up because of the increased demand for a fixed supply of advertising positions. Of course, a search engine could put more ads on each page but only at the price of diluting the effectiveness of each ad. What search engines are really auctioning off is the attention of motivated users, and as long as motivation stays the same, there's a limit to how much a page can be subdivided and remain profitable.

How Much Is Improved Usability Worth?

A company can double the "conversion rate" (making a sale or getting a request for product information from a new user) of its Web site through a good usability project. We found in a study of 42 redesign projects that site measures improved 135 percent after redesigning for usability.

It's difficult to estimate when all companies will fully understand the benefits of search advertising, but 2010 might be a reasonable guess. When that happens, further growth in keyword pricing will be due to improvements in the target Web sites.

As we've said, each company should bid no higher than what it takes to get a positive ROI. So let's say that a company brings in an average profit of $2.00 for each new customer. It might bid up to $1.99 for a click, for marginal profits of one cent per new visitor. Bidding $2.01 would be too much and would result in losses.

Now let's assume that this company performed enough usability work on its Web site to double the conversion rate for new visitors, which is a likely result. The company now makes an average of $4.00 per new visitor, so it can bid up to $3.99 for each click on the keywords that generate the $4.00 visitors. Improving usability and doubling the conversion rate doubles the bid that the company can afford to pay the search engine.

Of course, our example company would hope that it wouldn't have to increase its search engine bids by quite as much. It would be preferable to pocket some of the increased profits from usability improvements instead of handing them over to a search engine. In the short term, the company would indeed be able to retain some of the profits because its site would be better than its competitors'. Until its competitors also increase the value of their visitors, they could not afford higher search bids. Sooner or later, however, they will conduct their own usability projects and improve their profits, allowing them to increase their bids. As soon as eight competitors have doubled their bids, it will be necessary for our example company to double its bid again. Since even one cent's profit is better than no profit, rational companies will increase their bids accordingly.

In the long term, keyword prices will tend to increase at about the same pace as Web site conversion rates. Since the conversion rate is a key measure of usability, bids for search engine advertising will continue to indicate the extent to which Web sites are improving their designs.

As Web sites improve, search engines will confiscate almost all the increased profits they gain from increased usability. In other words, search engines need do nothing but watch their incomes grow as mainstream Web sites do all the hard work of improving the Web. Is this fair? No. But the reality is that search engines drive much of the new traffic that a site can hope to attract.

Three Reasons to Improve Your Site

Why should a Web site bother with usability if most of the value of improvements accrues to search engines? There are three reasons:

If you don't improve, your keyword bids will gradually become insufficient to get your ads shown, and your Search-derived traffic will shrivel to nothing.

If you do improve, there will be a window of opportunity when you're better than the competition and you won't have to increase your keyword bids to the max. During this time, you get to keep the fruits of your labor, so you can hope that your competitors don't read this book and are slow to improve their sites.

Finally, you do get to keep the increased profits from customers who arrive through other means. Although search ads are a great way of driving traffic, they're not the only way. Links from other sites, word of mouth, offline advertising, and many other techniques can drive traffic as well. Also, most search engines allow so-called "organic results," which are free listings where your site is shown because it naturally scores well for the user's query, even if you are not running any ads.