Rigging the game — Vendor case studies and Competitor analysis

In most walks of life, when you want to know whether a product you are about to purchase is any good, you are unlikely to go to the manufacturer for advice.

Much more to the point, we now live in a world where independent reviews are the backbone of how we measure whether a product is good or bad, which is most commonly done through independent user reviews.

You are far more likely to go to Yelp or Google if you want to know if the restaurant you have booked is suitable for the occasion. Or Amazon reviews if you want an independent unbiased (haha) review.

So why, when it comes to the purchase of software, is there still an antiquated behaviour of expecting that vendors provide case studies and analysis of how they measure up against their competition?

Let's talk about case studies...

If a vendor is going to write a case study (and by the way it is the VENDOR that writes them, not the client) who are they going to base them on?

Well, I can tell you it will be with the customers that they have the best relationship with or that they are willing to participate…

So, case studies are only ever a rose-tinted, cherry-picked version viewed through an Instagram filter, it doesn’t represent reality and is not a fair way of assessing the viability of whether the product is suitable for “your” organisation.

“Obsessing about the competition is not cool”

- Jeff Bezos

Then, if we take competitor analysis…. 

Now, in my industry there may well be 50 companies you would class as competition…. So, consider the following:

• How much time would it take to assess all the competition?
• How frequently do software vendors release new features and update their platform? (spoiler alert — it’s a lot) This renders any assessment redundant and inaccurate straight away.
• Would we be better off spending more time developing and improving our product?

This mentality is out of date, especially given that with most products you can go to the website, get a free trial (sometimes without having to speak to a salesperson), and then go through an onboarding session and evaluate the product yourself.

Neither case studies nor competitive analysis provides any benefit to partners or clients as they are biased and subjective assessments where the game is rigged so far in the vendor's favour that, if this was the sole basis of a purchasing decision, you would never lose.

What is even worse, is that this isn’t a trade secret — everyone knows this and is fully aware that they carry limited or no value.

So, why do people ask for case studies?

My personal view on this is that it is just an echo from a time where it was actually quite tricky to evaluate software products independently. A time before Reddit and even Amazon where vendor case studies were probably your best bet of getting an insight into a product before you would use it.

So, there is a predisposition to ask for them as it’s “what we have always done”.

But times have changed.

Surely there are more effective ways of reviewing a product...

If you want to know whether a product is any good, I suggest the following:

• Get a free trial and try it out for yourself, remember you are looking for the right product for your business, not what is right for someone else.
• Look online for independent reviews
• Post an article on Reddit and ask the question to the community. You will always get someone to come back or a debate happening which can help steer your understanding and what to do next

Either way, there are BETTER WAYS to evaluate products these days which are independent, up to date and not massively swung in the vendor's favour.