Like you, I think much of the criticism of analysts has been undeserved. Firms like Gartner and Forrester are providing useful data in a concise form.

But I do think the reports suffer from a systemic bias toward the vendors’ point of view, rather than the customers’ perspective. This is the root cause of the reports evaluating products rather than full-fledged solutions.

The bias is probably unintentional. Vendors have a vested interest in wooing analysts, so it’s far easier for these market research firms to do a vendor roundup than it is to do original research into the success or failure of specific company initiatives.

As you point out, it’s how these tools operate in real-world conditions at actual customer sites that matter, not whether a particular software company has a bigger market share or more visionary whitepapers.