Customer satisfaction survey asks more questions than it answers
OEMs go to great lengths to demonstrate the reliability of their bikes, but this is chillingly over the top.
A 2002 Victory V92 TC was frozen four years ago and put on display at the Ice Hotel Sweden. It was recently thawed and transferred to Swedish shop Stonetown Custom, where technicians re-started it after only an hour of tinkering. Amazingly, all it needed was an oil change, fresh fuel and a new battery — the spark plugs and electrical system are still those that were frozen in 2011. (The takeaway from the Ice Bike: forget winterization this fall and dunk your Victory in the lake.)
The bike had already had a hard life before being frozen. “It was originally a ‘shaker’ at the Polaris R&D facility in the U.S.A.,” says Niklas Frisk of Victory Motorcycles. “It did the equivalent of 300,000 km.” After that it was brought to Sweden as a guinea pig for European mechanics to complete technical training. “The engine was probably disassembled a couple of hundred times,” Frisk says. The company used the Ice Bike’s resurrection as an opportunity to tout the results of a recent Consumer Reports survey that ranked Victory as best brand in owners’ overall satisfaction.
Manufacturers are fond of trotting out such survey results (when those results are positive) and many are taken at face value — who has the time or desire to look beyond the final findings? We do, and decided to dig into this one.
Consumer Reports (CR) is a U.S.-based, independent, nonprofit research organization formed in 1936 to serve consumers “through unbiased product testing and ratings, research, journalism, public education, and advocacy.” The organization is “unconstrained by advertising or other commercial influences,” which lends some credibility to its findings; OEMs do not fund or have any influence over CR’s surveys or testing practices.
CR is funded through paid subscriptions from the general public. For this motorcycle satisfaction survey and brand rating, 11,000 respondents from the CR subscription pool were surveyed to rate their motorcycles (12,300 in total, representing 10 brands) on a scale of “better to worse” through a number of categories (fun, styling, acceleration, handling, cost of maintenance, and comfort).
CR admits — in the fine print — that, “These findings may not be representative of the U.S. population.” This survey, as with any survey, is only representative of those that respond, and, generally speaking, respondents are either very satisfied or mightily displeased.
Consider the ratings structure and that final results are gathered under the umbrella of each brand. Victory and Triumph owners, for instance, both scored their motorcycles as “better” in terms of handling. Those owners may have very different ideas about what constitutes “better” handling — if a Victory Magnum owner is happy with how his bike handles, how can someone else disagree? And what type of Triumph are we talking about? In terms of pure performance, a Street Triple would obliterate a Speedmaster in a head-to-head on a twisty road — but both owners may emerge at the other end entirely satisfied with their bike’s handling, each getting a “better” rating. And “better” than what? The owner’s expectations? In comparison to the clapped-out Commando they used to have? Individual interpretations of the ratings structure could vary greatly across the survey base.
And then there’s the categories themselves — for example, the contentious category of “comfort.” In the CR survey, Victory is the only brand to score a top rating for comfort. But this doesn’t mean Victory makes the most comfortable motorcycles; it means that in this CR survey, more Victory owners were “better” satisfied with the comfort of their machines. And comfort doesn’t necessarily mean physical comfort; a shorter rider may take psychological solace in the fact that a low-slung cruiser puts the ground within reach.
This CR survey oversimplifies owner satisfaction by asking respondents whether they would buy the same motorcycle again, of which Victory owners topped the rankings with 80 percent saying “yes” (Harley-Davidson was second with 72 percent and Honda third with 70). But is this really indicative of overall brand satisfaction? The question is whether owners would buy the same motorcycle again — what about people who would buy a different motorcycle, but from the same manufacturer? Someone may want a supersport and buy a GSX-R, then decide later they’d be more satisfied with a V-Strom.
Finally, there’s the fact that this is one survey among many. Around the same time results of this CR survey were released, Pied Piper — another compiler of company rankings based on “mystery shopper” experiences and sales success figures — published its Prospect Satisfaction Index U.S. Motorcycle Industry Benchmarking Study (clearly, they’re more concerned with facts and figures than catchy survey names). According to the Pied Piper rankings, Ducati is the top brand in the U.S. (followed by Harley and Victory).
The point is simple: survey permutations are endless, their results often confusing and sometimes conflicting. The findings of all should be taken with a grain of salt — or, better yet, with a cube of ice.
