Sometimes I miss the 90s. Search was so easy in ECM environments. Everyone used a bundled Verity and was happy.

Then things changed. People started to notice that if you actually used the system on an large scale, search performance degraded. There were many reasons for this. One was that vendors weren’t upgrading their bundled Verity engine. Another was that the engine was sitting on the same machine as the primary ECM server, so resources were being consumed at an increasing rate.

To solve this, EMC switched to using FAST with 5.3. On paper, this was a good idea. It was Java based and could be scaled. However, to tune it to work with Documentum effectively, and with minimal end-user interaction required, several things had to be locked in the configuration. It took a few service packs, but it now works fairly well when done correctly.

However, once you get over a certain threshold of objects and/or index size, you have to switch to a multi-node environment. This isn’t as easy as anyone would like.

So EMC looked at alternatives. As I’ve mentioned in the past, they had planned on allowing clients to use Lucene as their Search Engine with the upcoming 6.5. They were also going to upgrade the FAST engine to a more efficient version. Then 2008 hit.

Enter Autonomy

This wasn’t the item that hit the news wire in a big fashion yesterday, but it is the more revealing into the future of search in Documentum. EMC now has an OEM agreement with Autonomy. Looks to me that FAST is going to be out at EMC in the next year. I can’t see EMC paying 2 OEM licenses. Hopefully Lucene will stay in the picture, but you have to think FAST is out.

Now, some of you more astute readers may remember that Autonomy bought Verity back in the tale-end of 2005. So how is this a step forward? Has EMC completely thrown in the towel? Not necessarily. One of the biggest problems with the old Verity search was that it was OLD. The engine wasn’t upgraded. Now, I assume, we will get the latest engine. After the issues with FAST, EMC may even be willing to upgrade the core engine in the future, keeping pace with their partner’s innovations.

The question remains, why now?

Don’t Blame Microsoft

Making a much bigger splash yesterday in the blogshere was Microsoft putting in a bid for FAST. From what I can tell, the two are coincidental, if not fortuitous for EMC. The previous news release appeared to be driven by Autonomy. They also announced an OEM license agreement with Oracle yesterday. Now, they already had one in place, but that may have been more limited in scope than the new one. Also, I doubt EMC had any say into the timing of the Microsoft announcement.

Microsoft is paying a hefty premium for FAST. Most reports are saying in the 40-45% range. When you look at the end of 2007 price, before the drop the first few days this year, it isn’t quite as dramatic. Still it is a lot of money for a Java-based technology. They must really have thought that SharePoint needed some search help.

This deal isn’t going to close until Q2 according to Microsoft. It will be quite some time before they start killing the core product, if they do so. It is a strange acquisition as FAST is Java based and SharePoint, to put it bluntly, isn’t. FAST was probably the only option that was the right size for their acquisition strategy and had working tech.

The Net Result?

I think that from a Documentum perspective, it doesn’t matter. It looks like a few months ago EMC decided to part ways with FAST and started down that road. If anything, the two announcements hurt Microsoft the most as FAST is probably going to lose a revenue stream.

Life is good. EMC is working on making search better in the Documentum product, and they are doing it by turning to experts, or at least other vendors that we hope know more than EMC about search. I just hope that the effort to switch from FAST to Autonomy doesn’t push back the Lucene option.