Error message: The SharePoint item being crawled returned an error when attempting to download the item.

I was getting my crawl logs filled with error messages when crawling a particular section of one of my web applications that house a lot of Project collaboration sites. After investigating this issue what we found is that our standard template for projects used that Gantt chart extensively on some of the pages since the visual representation of tasks looks so darn good. The bad news was that the crawler was having problems interpreting these views. The solution was actually fairly easy once we knew the issue. For some reason it appears that the search server is presenting its crawl request as a very old version of Internet Explorer (IE 4!). The solution is to change the registry settings that control the gathering manager to present the User Agent to a higher version of IE on the search server.

  1. Open Regedit on your search server.
  2. Navigate to this registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\14.0\Search\Global\Gathering Manager
  3. Change the value of “UserAgent” from “MSIE 4.01” to “MSIE 8.0”
  4. Reboot the server and run a full crawl to index the items that were erroring out.

It does seem strange that in an environment that no longer supports IE 6 that Microsoft hasn’t updated this registry setting to a more modern broader/user agent, but this also explains why on some of my non-SharePoint sites that I index I was seeing a proliferation of older browser hits!