It seems that over the past few days many webmasters have noticed a significant proportion of their pages going into the supplementa index for no apparent reason.

Googleguy has acknowledged that there is a problem, and expects it to be cleaned up within a week or so.

It might be a good idea to check your pages with a site: search to see if this is happening to you.

<added>

While this may or may not be related (it is in a the same timeframe) SEOchat forums appear to have been dumped from the google index.

What I'm seeing is the majority of my pages (75%) go to a URL only partially indexed page.

Dave

I am seeing about 25% of one sites pages go supplemental - these are pages that have been up for years in some cases, used to rank OK and in most cases have been updated in the past few months.

The pages that went partial have been up and indexed for for almost 2 years as well. Some with #1 rankings that are now gone.

Dave

Lets hope that Googleguys "fix" includes this sort of stuff, as I can see no reason that any of those pages should go supplemental - except that there have been multiple copies of at least some of these pages which have been purloined and floating around on other sites.

Could it be that these pages are being hit as duplicate pages (even though they are original and the first published) due to copies on other sites???

Interesting thought. I believe that a filter for duplicate content is applied at intervals and not continuously. Perhaps this is time for that cycle and the results aren't quite what was expected.

Mel, are you aware of any duplicated content that has not slipped to supplemental whereas the original has?


Yes, lets hope they "fix" this soon. Given the blunders this past week by the CFO, one has to wonder if Google knows how Google is supposed to work.

Dave

I am going to carve out some time to compare the pages that have slipped vs. those that haven't in the next day or so.

One thing being seen from the BigDaddy changes is that many pages that were 301'd are ending up in the supplemental index en masse.

I just checked 50+ IP's/DC's and found that 19 of them are having "issues" when it comes to my site.

In those 19 IP's/DC's only 91-96 out of 643 pages are fully indexed and cached with the rest of the pages being URL only partially indexed.

These 19 IP's/DC's are...

64.233.189.104
216.239.53.99
216.239.57.99
216.239.59.99
66.102.11.99
66.102.9.99
66.102.7.99
216.239.59.104
216.239.53.104
66.102.11.104
66.102.9.104
66.102.7.104
216.239.57.98
216.239.57.105
216.239.59.105
66.102.7.105
66.102.7.147
216.239.57.147
216.239.57.98

On the surface, it appears if they're trying to rebuild their index from scratch.

Dave

I went through and checked several of the top ranked sites for KP's I follow on some of the IP's/DC's I listed above. All of the sites had scores of pages placed in the supplemental index.

<edit to add>

This appears to be a far more reaching issue than possible problems with duplicate content filtering or 301 pages. Far too many pages being placed in the supplemental index from what I see.

</edit>

Dave

This might be a newbie question, but what does it mean for the site to go into supplemental index?

The supplemental index is the very last place Google looks for pages to satisfy a query. First they go to the short barrels, if they find enough ranking pool candidates there (originally they were looking for 40,000 pages but it may well be more now) then they look no further and proceed to rank those pages for the query.

If they don't find enough there, then they go to the long barrels for ranking pool candidates.

Only if they don't find enough pages from the combination of the short and long barrels do they try the supplemental index.

In a nutshell the only thing any supplemental pages ever rank for are strage and usual queries.

OK I have had a bit of time to look into this problem on one site and find in this case that all the pages which are in the supplemental index are old pages that do not actually exist any longer (due to a site reorganization eight months ago) and which were 301'd to a new location with new content in most cases.

This confirms WilliamC comments about Big Daddy putting 301'd pages into the supplemental index.

I have verified that clicking on the supplemental listed link does indeed take you to the new page and that it does return a proper 301 response, so it looks like this is a case of Google not handling 301's correctly.

Post by Googleguy on Webmaster World:

Thats good news Timbo, but I guess the big question is why the hell did it happen in the first place.

Don't they test out these changes before they go live with them???

Probably the work experience guy trying to be proactive

This appears to only be part of the problem though. I'm seeing hundreds of pages never 301'd that are returning the supplemental or URL only results.

Could this be a case of the primary pages in the short barrel being dropped or lost and thus the supplemental pages are what are being used to fill in the results?

Dave

I personally think that this looks like the Big Daddy update ran into some indexing problems and they are in the process of rebuilding the index, but then why not just revert back to an old index temporarily?

I should clarify that on the site the I investigated the old pages which should have been dropped after having been 301'd for so long were instead put into the supplemental index, but the new pages are fine. This make my suspicions of indexing problems stronger.

Good question. Since the problem is only on select DC's, one would think they'd only display the results of the ones that are unaffected. It appears they may be trying to do this to a point.

Second, I suspect, for whatever reason, they don't have the old index to revert back to, or are unable to do so in the affected DC's and the only answer is to rebuild the indicies on those DC's from scratch.

Dave

Thanks for confirming the 301 thing Mel.

I will throw another supposition into the mix. Maybe BigDaddy needed more data from the pages than old indices stored. Or maybe they didn't store them in a way that can be fully used by the BD changes. That would require an actual full rebuild of all internal data if BD handles things using different/more data than before.

Yes but I think they did a total rebuild with a different addressing scheme (64bits) some time back and that went smooth as silk.

I think WilliamC might make a good point. Especially with trying to fix the canonical issues and maybe even pulling in more information about links and relavency of those links. who knows. let's just hope it gets fixed soon so we can all see where we are going to stand.

I have used the WebBug application to get server headers from the sites of mine which have been hit with a supp index and/or had their cache date set to Jul 2005.

WebBug returned this:

HTTP/1.0 200 OK
Age: 14901
Accept-Ranges: bytes
Date: Thu, 09 Mar 2006 19:18:27 GMT
Content-Length: 639
Content-Type: text/html
Server: Apache
Last-Modified: Tue, 26 Jul 2005 05:30:16 GMT

26th July is the cache date on the sites. The servers don't currently return the Last-Modified date but it seems that it is returning this Jul 05 date by default.

A post on Webmaster World seemed to indicate that it was an issue to do with the HTTP protocol retrurned. Webbug was using HTTP 1.0.

theres a interesting article by Aaron wall on the topic

http://www.seobook.com/archives/001545.shtml

From my little corner of the universe, it appears that Google is trying to keep the afflicted DC's out of the rotation as the default (.com) DC. If this is true, they're at least taking a positive step while they sort this out.

If you do a search for St. Roch Medals, my sites product page will show up as #1 on the DC's that are not affected and not at all on the ones that are.

<edit to add>
I'd be curious to know where the site shows up for the above term on your computer to see if indeed Google is keeping the "bad" DC's from going live.
</edit>

Dave

I see he completely missed the 301 issue

I had thought (hoped) that Google may have been trying to keep the affeced DC's out of the rotation from going live. This is not the case.

FTR, none of my pages have ever been 301'd and it is affecting new pages as well as old ones. The one difference in my case is that pages are not showing as supplemental but as partially indexed URL only.

I suspect that the reason for this is that that is what Google has in their supplemental index for my site. The URL only, therefore nothing to display as supplemental.

Any thought as to this affecting dynamic pages rather than static?

Dave

Unfortunately, this site has succumbed to the Supplemental syndrome. All of the pages, except the home page, are now Supplemental.

There are no 301s or 302s anywhere in the site, and, although some articles have been reproduced in may places, most of the site's pages could not fall foul of a duplication issue. The main site's pages could fall foul of a duplication issue, due to the Printable versions, but the forum's pages couldn't be seen as duplicates. So, if the problem has a single cause, then I can't see it being 301s, 302s, or duplication.

Hi Grinler - nice to see you here

I understand what you mean about pages already being Supplemental, but didn't show until there were very few non-supplemental pages. I've seen that happen before. But I don't think it's the case with this site, so it probably isn't what's happening.

I am seeing a lot of forums affected, so possibly another facet may be the way forums do a wee bit of cloaking, or the possibility for duped content via diff url structures.