The Google Dance

Every month many website owners sit in front of their computers and press their browser's refresh button repeatedly as Google begins its monthly update. The update, now widely know as the "Google Dance," has become a monthly online festivity where webmasters closely watch as Google's new index gradually comes online and either rejoice or despair over their new rankings.

Why is it called the "Dance"?

The update period, which typically lasts a few days each month, is known as the "Dance," because the result pages at the main Google page and its two test domains (www2.google.com and www3.google.com) frequently fluctuate as the new rankings gradually come online. Until the dance begins, the results from the main domain and the test domains are mostly stable.

What's happening during the Dance

The Dance is when Google adds a new index with the results from the month's earlier "deep crawl" of the Web. I'll say a little more about the "deep crawl" a little later. During the Dance, Google uses the previously mentioned www2 and www3 domains to test the new index for any abnormalities. Once the new index has been tested, Google will steadily update each of their data centers with this new index. While this is happening, search results from the main Google domain can change on a minute-by-minute basis, since some searches will pull its results from an updated datacenter and others will pull from one that still has the old index.

Deep Crawl vs. Fresh Crawl

The Dance solely concerns itself with integrating the latest monthly deep crawl into the Google index. However, if you frequently search Google using the same keywords, you may notice that the search results fluctuate more than once a month. This is due to a relatively new feature of Google known as the "Fresh Crawl." The fresh crawl occurs almost continuously to spot frequently updated sites and to add the new content to the engine's index. This frequent fluctuation in the engine's search results has come to be known as "Everflux." Due to these slight variances, the accepted method to detect the beginning of the dance is to look for differences in the number of backlinks to major sites (such as Yahoo) between the main site and the test domains, and not by looking for changes in search results.

You can easily spot pages that have been fresh crawled by examining Google search results pages and looking for pages that have a date in the last line of their entry (between the page URL and the "cached" link). The fresh crawl uses a different spider, known as "freshbot," than the deep crawl, which uses "deepbot." These two spiders use two different blocks of IPs and can easily be differentiated, with deepbot using IPs that start with 216. and freshbot using IPs that start with 64.

With the emergence of freshbot, does the deep crawl still matter? Yes, it still matters quite a bit. The "fresh" results are not stable, with fresh pages popping in and dropping out of the results pages on a daily basis. The stable, persistent rankings are based solely on the deep crawl. Also, Google uses the deep crawl results to calculate PageRank, that magical numeric value representing a page's importance, which is displayed on the Google Toolbar via a little green bar.

Watching the Dance

As stated earlier, the surest way to know that the Dance has started is to check for differences in the number of backlinks between the main Google site and the test sites. You can do this by searching for "link:www.yahoo.com" at the main Google site and www2.google.com and www3.google.com and watching for any variance between the results. The Google Dance Tool streamlines this process by allowing you to search all three domains simultaneously, presenting all three results in a frameset. This tool also allows you to simultaneously search each of Google's datacenters.

During the Dance, you can use this tool to get a sneak peek at the new search rankings, watch for new page's inclusion into the index, and to check on the progress of the dance as the new results propagate to the separate datacenters. You can also get a sneak peek into a page's new PageRank by changing the IP to which the Toolbar points.

Further Reading:

• About the Google Dance
• Dance Alerts
  A service that sends an e-mail announcing the beginning of a dance.
• Google's explanation of PageRank
• Google Update History
  A chart by Brett Tabke of WebmasterWorld, showing the dates of the previous Google Dances.
  There are numerous WebmasterWorld posts concerning the Dance and all other things Google in the Google News forum.
  Some of the more notable posts include:
  • Googlebot: Deepbot and Freshbot FAQ and Information
  • Google Updates and Everflux, the Monthly Mid-Cycle Changes
  • FAQ: Everflux - mid-month Google spidering and minor updates