Category Archives: FAST Search

FAST Search | Index Server Runs Out Of Disk Space During Crawl

I’m not an expert with FAST, I just have to deal with it.  This is a fun little thing to have happened recently.  SharePoint adoption has been going really well.  More people are using it, more people are adding content, more content is being indexed, more space is being used.

The drive that we installed FASTsearch on is fairly small for drives these days, roughly 136GB of free disk space.  This particular company also has a policy of 80% utilization of drives before an alert goes off to go look at the server disk utilization and reduce it.  As I know from getting these alerts, when FAST is doing an index, there are times where the %FASTSEARCH%\tmp and the %FASTSEARCH%\data\data_index directories get pretty full.  Like an extra 60GB worth of full.  This is enough that along with the other items on the drive it tips past 80% utilization and I get the email alert.  This is because FAST Search Server keeps a read-only binary index file set to serve queries while building the next index file set. The worst-case disk space usage for index data is approximately 2.5 times the size of a single index file set.

This generally happens at night and by morning all the indexing is done and the drives have plenty of space in them.  It’s not really worth ordering another drive at this point as it is a temporary condition and doesn’t significantly affect performance, but would be nice to minimize the number of alerts I get by reconfiguring FAST to use a different drive for temporary files and even for storing index data.

As it turns out, FAST is not as easily configurable as say, Windows when you want to move the temp directory, or even SharePoint when you want to move the ULS and usage logs.  A quick Bing search did not turn up any useful articles about how to reconfigure how FAST utilizes directories, except for one example which suggested editing some of the XML – bad idea because you are in an unsupported configuration and possible possible upgrade issues in the future.  Turns out the resolution for this was actually relatively simple though thanks to the use of junction points.

In my case, we had just purchased a rather large HDD of 300GB in which to house the ULS and usage logs because Microsoft Best Practices in regards to SharePoint says keep the log files and the SharePoint binaries on separate drives if possible, and 300GB was the smallest standard the company supported when we ordered them.  This means I had a lot of free space out there on the new drive, if only I could get FAST to utilize it.

In my case, I was able to use KB2506015 to reconfigure the directories with junction points, stay in a supported mode, and utilize the extra space.

If additional storage can be added to the server, the entire %FASTSEARCH%\data directory can be moved to a new location with the same permissions ("Full Control", granted to the FASTSearchAdministrators local group), and connected back to the installation via a junction point. To do so, follow the steps below on each FAST index server:

  1. Stop the FAST Search for SharePoint service.
  2. Stop the FAST Search for SharePoint Monitoring service.
  3. Move %FASTSEARCH%\data to the larger storage you have added.
  4. Run the following in a command prompt: mklink /j %FASTSEARCH%\data %NEW_LOCATION%\data
  5. Start the FAST Search for SharePoint Service.

Please note that while other methods of relocating the index outside of the %FASTSEARCH% parent directory are not supported, the entire parent directory can also be moved to a new physical location without using a junction point (skipping step #4 above) if the drive letter, path, and permissions remain identical.

One I started back up the FAST Search for SharePoint Service everything came up perfect and I could perform a crawl without issues.

Fast Search – Failed to initialize session with document engine: Unable to resolve Contentdistributor

At a client site FAST suddenly started having problems with crawling data sources, they would continue to crawl indefinately and when trying to stop them they would sit in a “Stopping” status for hours and hours without ever actually completing a crawl or loading any documents into the index.  Reviewing the event logs turned up this error:

Failed to initialize session with document engine: Unable to resolve Contentdistributor

Having worked fine for about a year we could not find any significant changes that were made to the system that would cause this behavior… and that’s when the little lightbulb went off that it had been almost exactly a year since this server was set up…

During installation on development servers a self-signed certificate can be created for communication between FAST and SharePoint. It turns out that the self-signed certificate is only valid for one year and when it expires the above problems will occur. Unfortunately there is no mechanism making it obvious to the user that the certificate has expired, thank you MSCryptic… The fix is to generate and deploy new self-signed certificate and this can be achieved with the following steps:

  1. Make sure the FAST Search for SharePoint & FAST Search for SharePoint Monitoring windows services are stopped.
  2. Open the FAST PowerShell on the FAST server as an administrator.
  3. Navigate to the FAST directory (i.e. <FASTSearchFolder>\installer\scripts)
  4. Run the following command: .\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true
  5. This will generate a new certificate, valid for one year in the following folder: <FASTSearchFolder>\data\data_security\cert\FASTSearchCert.pfx
  6. This certificate will need to be copied to the SharePoint server (if running a multi-server environment).
  7. Start the FAST Search for SharePoint & FAST Search for SharePoint Monitoring windows services.
  8. Now the certificate needs to be loaded on the SharePoint server.
  9. Open the SharePoint PowerShell as an administrator on the SharePoint 2010 server.
  10. Navigate to the location of the SecureFASTSearchConnector.ps1 script (this script may need to be copied from the FAST server as mentioned in step 6).
  11. Run the following command (userName should reflect the details of the user running the SharePoint Server Search 14 (OSearch14) windows service):.\SecureFASTSearchConnector.ps1 –certPath “path of the certificate\certificatename.pfx” –ssaName “name of your content SSA” –username “domain\username”
  12. Don’t foreget that at this point you’ll be prompted for the cert password used when you initially set up FAST, so have it handy.

Assuming there were no errors when running the PowerShell scripts the SharePoint server certificate has been deployed and will be valid for another year. The content sources will begin working normally again.

FAST Search–100 Year Certificate Expiration

So one of the fun things about setting up FAST for development is that when you set up a development environment you have the option of creating a self-signed certificate if you want to set up HTTPS communication between FAST and SharePoint – always a good idea if you are doing the same in production as you want to make sure that the SDLC farms match production as closely as possible.  Unfortunately what this means is that you now have a cert that is valid for one year after installation.  As with most environments, it seems as if the DEV and QA environments don’t always get as closely monitored as production, and when things go sideways in these environments you don’t always have all hands on deck helping you figure out what happened.  Either that or you have to remember to up the cert every year… tedious for SDLC environments and often forgotten about when employees leave, or you have to have some sort of non-expiring certificate.  We don’t have perpetual certificates, So how to keep your DEV and QA environments in a stable and low maintenance state?  100 year expirations on your certs!

If you are running FAST on Windows Server 2008 R2 here’s an easy solution to generating 100 year certs for your environments:

1) Open up C:\FASTSearch\installer\scripts\include\certificatesetup.ps1 and scroll down to line number 246 which reads:

Add-Content -Path $infFile -Value "SuppressDefaults=true"

2) Append the following lines underneath it:

Add-Content -Path $infFile -Value "ValidityPeriod=Years"
Add-Content -Path $infFile -Value "ValidityPeriodUnits=100"

3) save the file.

4) Then recreate your certificate with replacedefaultcertificate.ps1 as explained at TechNet. Remember to import it on your SharePoint 2010 server as well.  If you apply this edit during installation of FAST for SharePoint you save yourself a step and now you have an environment that potentially will give you one less headache a year from now.

FAST Search Server for SharePoint–Number of Items Indexed

Having set up the FAST Search Server for SharePoint, I can tell you it is an interesting ordeal.  Because the product was purchased by Microsoft and productionalized for SharePoint, there are a few gotchas in the product like having to use FQDN domains for the User IDs and some fun with certificates.

However, once you’ve got the system up, running and integrated with SharePoint it is a very useful and powerful product.

One useful tool in your administration bag is the IndexerInfo.exe, which is useful when retrieving basic information about the number of items indexed or the status of the indexing process.

In order to use this tool, you have to log onto the FAST server and run the executable located here:

<FASTSearchFolder>\bin\indexerinfo

And that will give you some pretty useful basic information about your server’s index, such as total number of items indexed:

image

However, the tool has two drawbacks:

  1. You have to log into the server to use it.
  2. It doesn’t have a lot of granularity or visibility to SharePoint scopes and permissions.

In many cases, a great workaround for this is to use your SharePoint Search which is using the FAST engine and simply query for the ‘#’ character.

image

And this should give you a ball-park figure of how many items are in your index.  You’ll notice though that the numbers are slightly different.  If I were only querying against a SharePoint source and I was a Farm Admin, the numbers would be identical.  However, since I also index against websites and proprietary sources that I don’t have the ACLs for, these results are security trimmed to only show the total number of items that I have access to.  In this case, I can see from the two numbers that my account doesn’t have access to 1,528 documents (388252 – 386724 = 1528)

If I log in with my restricted test user account (and every SharePoint Admin should have a “regular” test account so they can see their site from the normal user experience) I can see I have access to even less documents:

image

This is again a good thing, as it allows me to see the number of documents that a particular user profile has access to.  If I perform the same ‘#’ search when using a contextual “This Site:” drop down scope, I can further refine the number of documents indexed to a particular subset based on scope.  For instance if I wanted to quickly know how many indexed items I had in my Knowledge Base I could use the contextual scope search and see that there were 751 indexed items in this particular sub-set of items that my account has access to (and since I am an Admin and this is running on SharePoint I know this is the total number for this site).

image

Note that when you want to know how many items are indexed in a particular scope, you don’t need to actually use the ‘#’ character, just reference the scope, and since you’ve got text in the search box it won’t pop up with the “Please enter one or more search words.” warning box.

image

Of course, if you want to get into the fine-grained status of your Indexer and the information that an admin might need to fine tune the engine itself, then it’s back to the command line to get your granular information:

image