product evaluation process

As I casually browse online radio websites and consider the current product offerings, a constant theme I'm encountering is "bare-minimum checklist completion".

One product, which shall remain nameless but will serve as the focus of this article, combines a digitally-tuned shortwave radio with a detachable MP3 player. The detachable MP3 player has been criticized for having a complicated user interface, particularly for managing recordings made from the radio with which it can connect. The MP3 player has inadequate memory capacity, considering what else is available in the standalone MP3 player market. I make no claims that I've used this product, but without owning one, I can make observations about its shortcomings. It seems to be a case of the whole being barely a sum of its parts.

How did the product end up this way? Perhaps a goal of combining a shortwave receiver with an MP3 player came about. It would be cheap to include a small amount of memory, so choose some small storage size and go with that. Ignore how quickly the flash-player market will change over the next two years; this product is in a different category (even though it may likely end up in the same home as a flash-based MP3 player). Find some way to allow the MP3 player to detach from the radio, no matter how ugly the resulting product ends up. Put it in a box and sell it. This is a pretty cynical view of things, but how far is it from the truth?

What is the manufacturer doing to resolve the criticism of this product? Probably nothing. They may decide to focus on implementing other features, or they may decide that there's not enough interest in the product, so it will get scrapped. Or it will remain on the market, unchanged, to languish (from the manufacturer's perspective) and potentially damage the manufacturer's reputation (from the consumer's perspective).

There are two distinct ways to evaluate a product. They differ in the method and, I would argue, in the quality of the purchasing decision. One way is to compare a product against a checklist of required or desired features: the product has this, the product doesn't have that. The other way is to individually rate each of the features. "This radio has a below-average speaker (2 out of 5) but above-average sensitivity on shortwave bands (5 out of 5)," for example. I prefer to use numeric ratings for product features, such that I can come up with a formula to express my overall rating of a product.

This ratings-based feature list evaluation is the method I used for selecting my current car from among about 10 possibilities. Obviously, it would've been far too time-consuming to test-drive all of the candidate products, so the initial ratings were based on aspects that I could rate without using the product. This included details like exterior appearance, safety rating, line-in support, fuel economy, price, and so on. When a few clear winners emerged, I test-drove those models and made my selection. And the ultimate winner wasn't the one I expected, but I made my purchase and had full confidence in it.

The fact that two individuals may rate the same product completely differently is due to something that we call taste. This is perfectly normal, and both people can be right no matter how far apart their individual ratings are. But when it comes to technology products, the average consumer doesn't pay very much attention to the important factors for enjoying a product after purchase. It's easier to make a checklist of required features and find a product to fulfill that, than to come up with a scoring system to determine what the best choice would be. And sometimes, people base much or all of their decision on price. In some cases, if they really thought about it, price would not turn out to be the only important factor in the decision.