Predicting Search Popularity

by Anthony Mills on February 8, 2010

Search volume is one of the most important metrics when analyzing organic and PPC data; however it is one of the least accurately recorded statistics. Part of the difficulty in reporting on exact search volume stems primarily from the various ways in which the engines associate the results pages (broad, phrase, and exact matching). The second problem may lie in the engine’s inability to maintain an active database of every combination of search words being referenced.

Google launched two products last year, Google Insights and Google Trends, that haven’t received a lot of attention and they offer an interesting work-around to the search volume problem. These products may assist us in our performance projections and competitive reporting.

They both provide a daily insight into what people are searching for by using a relative measurement of volume for any search query. It’s an index in of itself that measures average daily fluctuations around the terms popularity. By rolling individual terms into categories such as retail or restaurants we can begin seeing particular trends and patterns that follow that industry’s sales or economic indicators.

Google Research blog put some research together and an interesting subsequent analysis has been conducted by Hal Varian, Google’s Chief Economist. Below is a brief outline of the questions and conclusions from the experiment.

Questions Google was examining:

How many search queries have trends that are predictable?
Are some categories more predictable than others? How is the distribution of predictable trends between the various categories?
How predictable are the trends of aggregated search queries for different categories? Which categories are more predictable and which are less so?

Some highlights of their observations:

Over half of the most popular Google search queries are predictable in a 12 month ahead forecast, with a mean absolute prediction error of about 12%.
Nearly half of the most popular queries are not predictable (with respect to the model we have used).
Some categories have particularly high fraction of predictable queries; for instance, Health (74%), Food & Drink (67%) and Travel (65%).
Some categories have particularly low fraction of predictable queries; for instance, Entertainment (35%) and Social Networks & Online Communities (27%).
The trends of aggregated queries per categories are much more predictable: 88% of the aggregated category search trends of over 600 categories in Insights for Search are predictable, with a mean absolute prediction error of less than 6%.
There is a clear association between the existence of seasonality patterns and higher predictability, as well as an association between high levels of outliers and lower predictability. For the Entertainment category that has typically less seasonal search behavior as well as relatively higher number of singular spikes of interest, we have seen a predictability of 35%, where as the category of Travel with a very seasonal behavior and lower tendency for short spikes of interest had a predictability of 65%.
One should expect the actual search trends to deviate from forecast for many predictable queries, due to possible events and dynamic circumstances.
We show the forecasting vs. actual for trends of a few categories, including some that were used recently for predicting the present of various economic indicators. This demonstrates how forecasting can serve as a good baseline for identifying interesting deviations in actual search traffic.

The benefit that these products have for Sitewire is that it allows us to plan in advance of search volume spikes in order to optimize the number of visits and conversions to our clients’ sites. When a seasonal trend or gradual incline begins appearing, we are able to ramp PPC spend in anticipation of the search query inventory growth, and all things being constant should capitalize on the percentage of people converting into sales.