EMC Search Potpourri

Sometimes I miss the 90s. Search was so easy in ECM environments. Everyone used a bundled Verity and was happy.

Then things changed. People started to notice that if you actually used the system on an large scale, search performance degraded. There were many reasons for this. One was that vendors weren’t upgrading their bundled Verity engine. Another was that the engine was sitting on the same machine as the primary ECM server, so resources were being consumed at an increasing rate.

To solve this, EMC switched to using FAST with 5.3. On paper, this was a good idea. It was Java based and could be scaled. However, to tune it to work with Documentum effectively, and with minimal end-user interaction required, several things had to be locked in the configuration. It took a few service packs, but it now works fairly well when done correctly.

However, once you get over a certain threshold of objects and/or index size, you have to switch to a multi-node environment. This isn’t as easy as anyone would like.

So EMC looked at alternatives. As I’ve mentioned in the past, they had planned on allowing clients to use Lucene as their Search Engine with the upcoming 6.5. They were also going to upgrade the FAST engine to a more efficient version. Then 2008 hit.

Enter Autonomy

This wasn’t the item that hit the news wire in a big fashion yesterday, but it is the more revealing into the future of search in Documentum. EMC now has an OEM agreement with Autonomy. Looks to me that FAST is going to be out at EMC in the next year. I can’t see EMC paying 2 OEM licenses. Hopefully Lucene will stay in the picture, but you have to think FAST is out.

Now, some of you more astute readers may remember that Autonomy bought Verity back in the tale-end of 2005. So how is this a step forward? Has EMC completely thrown in the towel? Not necessarily. One of the biggest problems with the old Verity search was that it was OLD. The engine wasn’t upgraded. Now, I assume, we will get the latest engine. After the issues with FAST, EMC may even be willing to upgrade the core engine in the future, keeping pace with their partner’s innovations.

The question remains, why now?

Don’t Blame Microsoft

Making a much bigger splash yesterday in the blogshere was Microsoft putting in a bid for FAST. From what I can tell, the two are coincidental, if not fortuitous for EMC. The previous news release appeared to be driven by Autonomy. They also announced an OEM license agreement with Oracle yesterday. Now, they already had one in place, but that may have been more limited in scope than the new one. Also, I doubt EMC had any say into the timing of the Microsoft announcement.

Microsoft is paying a hefty premium for FAST. Most reports are saying in the 40-45% range. When you look at the end of 2007 price, before the drop the first few days this year, it isn’t quite as dramatic. Still it is a lot of money for a Java-based technology. They must really have thought that SharePoint needed some search help.

This deal isn’t going to close until Q2 according to Microsoft. It will be quite some time before they start killing the core product, if they do so. It is a strange acquisition as FAST is Java based and SharePoint, to put it bluntly, isn’t. FAST was probably the only option that was the right size for their acquisition strategy and had working tech.

The Net Result?

I think that from a Documentum perspective, it doesn’t matter. It looks like a few months ago EMC decided to part ways with FAST and started down that road. If anything, the two announcements hurt Microsoft the most as FAST is probably going to lose a revenue stream.

Life is good. EMC is working on making search better in the Documentum product, and they are doing it by turning to experts, or at least other vendors that we hope know more than EMC about search. I just hope that the effort to switch from FAST to Autonomy doesn’t push back the Lucene option.

6 thoughts on “EMC Search Potpourri

  1. GC says:

    Despite DCTM previous integration with Autonomy, the OEM deal does not have an impact on Content Server’s search.


  2. Thanks for sharing GC. I wish you had shared more. I am sharing so that everyone else can have the same, confused, look on their face as they try and figure out what it does impact.


  3. I second that – GC – please explain why there is no impact. Has EMC stated why the agreement was needed and where in their stack they plan on using it? With the Legato line perhaps? I would hope that you are not basing this position on the premise that we can easily put any search engine we want in the architecture.


  4. Mark says:

    I thought I read or heard somewhere that EMC was considering moving to a plug-in connector based search model. Where the customer could choose from several search vendors? Is this not the case?



  5. First, check out my latest post. It seems that Autonomy is meant for RSA. The word I heard confirms GC’s statements.

    Mark, this direction would not change. I think they are still moving that way. However, I don’t anticipate them ever not bundling some sort of Search engine. You will just unplug it and insert your own.


  6. We have recently integrated FAST ESP with their Documentum Connector to Documentum D6 and have really positive experiences from that. The search GUI is a nice and fast complement to searching inside WebTop/DAM. Much easier to learn as well. Also we like that we have a GUI for the analytics log file (even though the log analyzer is a bit unstable currently). Since FAST already have around 3500 stand-alone installations I hope that the FAST ESP product will still exists as product. Having FAST inside MOSS 200X is definately not an argument to switch from Documentum to MOSS 🙂

    Being able to choose which internal search engine to use is of course good. However, I now believe it is way more important for EMC to create a much more capable search GUI in WebTop/DAM. Facetted (drill-down) navigator and search analytics seem like a good first step.


Comments are closed.