Taking a break from talking about the economics and future of Content Management to look back at the reality we are facing today.  For all of the need to start moving to the future, we all still have problems that have to be solved today.

One of my most challenging problems in Documentum is full-text search.  While fine for the average system, FAST is a beast.  It requires a large cage, is barely tamed, and takes a lot of work to train for larger tasks.  When you invest in it, life is good, but it is an investment of time and effort.  Like many, a chance to have a scalable, highly-available, reliable, and EASY full-text search as part of the system is something we’ve been missing in Documentum and most Content Management systems.

Well, that is changing.  At the end of October, the new Documentum xPlore search engine is being released.  Known during development as Documentum Search Server, xPlore promises to make life much easier for people.

Before I add any more details, the usual disclaimers apply.  Anything in this post talking about things not released, including dates, is subject to change.  If anything in this post fails to take reality, don’t yell at EMC, yell at me….and I’ll yell at EMC. 😉

Why Kill FAST?

Let’s cover the obvious.  FAST is a resource consumer and is not simple to scale beyond the simple single node.  It will do it, but you need to do a lot of manual planning and work.  It doesn’t work well on NAS (trust me) and is dangerous to run in production in VMWare.  It does work and performs better than its predecessor, but it is in no way ideal.

Then there are the business issues.  Microsoft bought FAST.  Microsoft has ended support for OEM versions of FAST.  EMC negotiated support through the end of 2011 for FAST, but it is clearly NOT even an option for the future.  This has led the FAST engine to be sitting in maintenance since D6.5 allowing effort to be focused on the next engine.

So a replacement is needed.  How about using X-Hive to store the index and Lucene to create it?  One is mature, proven, and owned by EMC.  The other is a solid open source engine that is also proven.  We’ve used it on some of our projects and it works great.

Sounds great.  So let’s look at what we are going to get in the next month.

Coming Soon

It’ll be time to start downloading Documentum xPlore 1.0 in November.  I’ve heard the end of October repeatedly, but a few extra days seems like a realistic timeline for planning.  If you want to use it, you’ll need to be on Documentum 6.5 sp2 or higher.

Most importantly, it is free to those with Content Server (which should be everyone).  This is a replacement for a core feature of every Content Management system since the 90s, so free is the right choice.  Starting with D6.7 (Q1 2011), xPlore will be the only option for search.  The version of xPlore to be packaged with D6.7 will be tagged xPlore 1.1.

Now for feature facts/notes from the call I was tuned to on Friday:

• Architecture details:
  • Runs on Windows or Linux, not Solaris or other UNIX flavors
    • Solaris not in roadmap
    • Feedback from customers say that VMWare and the Windows/Linux option has been suitable.  If you NEED UNIX, let them know.  They aren’t opposed, they just haven’t heard the absolute demand.
  • Supports all Content Server platforms
  • VMWare support
  • NAS Support (guidance/tips to come for disk configurations)
  • High Availability options
    • Active/Active (was the only option with FAST)
    • Active/Passive with Clusters
    • N+1 Sparing (JOY!!!!)
  • Sizing spreadsheet being refined. More efficient than FAST, so existing hardware can support current installations.
  • Less transient disk space needed.
  • Online backup (hot backup!)
• Language/Internationalization Support:
  • xPlore 1.0 supports English, German, and Chinese
  • xPlore 1.1 supports French, Italian, Spanish, Japanese, and Korean
  • Works with other languages.  Those languages are not “supported” as grammar nuances have not been tested.
  • Brazilian Portuguese support is planned.
• All DQL/FTDQL automatically translated into XQuery, so no need to change code
• Zone search (VQL) is being replaced with XQuery
• Integrated Documentum Security in search (This is big. No more taking results back into the database to validate against the security model.)
• Native Facet computation
  • Exposed within CenterStage 1.1
  • Requires 6.6 DFC to utilize
  • It can be exposed in Webtop, but it is not out-of-the-box
• Query and Ingestion Analytics (tune those queries!)
• There is an xPlore Administrator interface
• Federated Search Server 2 is supports xPlore

Some good stuff if it all comes to pass. It has been in controlled release since July.  With Ed Bueche is running the whole thing on top of XDB technology, I’m optimistic.

Talking ‘Bout my Migration!

Okay, let’s think about the migration issue.  Full-text indexes of large systems aren’t created overnight.  I’m also optimistic on this front.  Let me tell you why.

The migration to FAST wasn’t that bad.  FAST was an issue, but not the migration process.  Migrating to FAST was not that much more work than installing a new system and ingesting content.

This migration may be easier.  Aside from the same EMC engineering team planning this out, there is no change to the Indexing infrastructure within the Content Server.  That means you can technically run both FAST and xPlore at the same time.