There has been a lot of talk the last few months about the integrity, and completeness, of Gartner’s Magic Quadrant reports.  While the lawsuit against Gartner from ZL Technologies was dismissed, at least for now, there are a lot of questions being asked about the level of influence upon the market by Gartner, and upon Gartner by the market.

The questions can also be applied to Forrester and other analyst reports in the ECM industry, and other industries for that matter.  I’m confining my discussion today to Gartner’s Magic Quadrant and Forrester’s Wave for ECM as I know them best.

The Power of Suggestion

Putting how products are “scored” aside, let’s look at the impression that the actual score, or lack of one, has on the world.  There seem to be two approaches to evaluating vendors here:

1. Include vendors across the spectrum that meet a minimum “viability”
2. Include only vendors that score well enough to avoid the least favorably sounding category

Gartner appears to be more inclusive than Forrester in adding vendors to their report.  This is not to say that Forrester is using the second approach, but it appears that way based upon the vendors presented and their scores.

If I was a vendor, I would favor the Forrester report when talking to potential clients.  Included vendors are in an exclusive club and can boast as much, regardless of any score or classification.  Excluded vendors can pull out a long list of other excluded vendors to cite how it isn’t important that they were not included.  Both of those approaches are harder to take when explaining positioning, or lack thereof, in the Gartner MQ.

Given all that, potential buyers still look at the pretty picture and make decisions. This gives the reports more power in the market than any report should carry.

Opinion or Fact?

Both Gartner and Forrester clearly state that their report represents an opinion based upon the analysis made at a specified time. They acknowledge that things change.  Gartner makes a point of saying that a Niche Player may be a proper choice for a client. In fact they state,

Gartner advises organizations against simply selecting vendors that appear in the Leaders quadrant. All selections should be buyer-specific, and vendors from the Challengers, Niche Players or Visionaries quadrants may be better matches for your business goals and solution requirements.

So does it hurt a vendor to be classified as a Niche player? I would say no, in theory. The reason is that if you look at the criteria to even be included in the Gartner report, credibility is gained just by being on the report. The danger is that people look exclusively at the graphic and ignore everything else in the report, including the statement to consider all of the vendors.

Hovering over all of this is the charge of bias. There is bias, but it isn’t necessarily intentional. Analysts talk to vendors to learn about their direction and strategy. The vendor will take that opportunity to sell the analyst on that strategy. Given enough interactions and the right client referrals, that message will start to sink-in, at the very least, subconsciously.

Hell, I’m biased and I suffer with the pains of many ECM solution. Of course, the evil you know…

Do I Have a Point?

The reports are useful.  They provide a relatively consistent view into the state of each vendor.  If you look at the vendor write-ups, you will gain some insight into every company that you may not get from the vendor directly.  You will also see why a vendor may have been discounted in their scoring and determine if that factor is relevant to your selection process.

Are there issues?  Yes. They include:

• Vendors should not be evaluated as much as the complete, integrated solution platform. If a vendor says that it all works together, then they should be able to find several clients that have made them work together that can speak kindly upon that effort. Marko looked at this vendor versus platform issue and while I disagree with his scoring, I agree with his approach.
• The reports are too exclusive. I thought Gartner was bad (give us 3-4 open source options please) until I saw Forrester with a paltry eight vendors analyzed.  EIGHT! Last spring I had a bunch of vendors show their wares to a client and we had more than eight vendors come through the door.  You can use the report to compare vendors on a high-level, but if one vendor isn’t included, the report loses value without knowing exactly why a vendor wasn’t included.  Even then, it may be difficult to determine if it matters.
• Gartner’s Magic Quadrant is a victim of its own success.  The report has been around so long and has become so well known that people lean on it like a crutch.  It is just one tool that should be looked at with a critical eye.  The criteria, and their weight, may not match the goals of your organization.

If the solution works and the company isn’t about to go under, does it matter that it doesn’t have a strong Digital Asset Management component that you don’t need or that it won’t scale to support an organization three times the size of yours?

The Summary of the Summary

Like any tool, you need to study the reports fully to get any real value out of them and to understand the limitations.  Someone may be a Niche Player, but maybe you only need a Niche solution. Maybe you need something new and innovative that isn’t “mature” enough to be included in the reports.

Don’t rush to judgment from looking at a graphic or even reading a single report. Multiple sources are important and when all else fails, talk to people that have already tackled your problem.

Your combination of needs may be unique, but the odds are incredibly good that they have all been solved before by multiple organizations, probably in your city. Find them. Talk to them. Learn from them.