In part one, we talked about gun reviews versus car reviews.
I’ll probably never buy a Lamborghini, and even my chances of owning a Mustang are beginning to fade as my kids grow up and bills increase. What matters to me more right now is whether my wife’s Nissan Pathfinder will be as reliable as my old Frontier pickup or what are the long-term maintenance costs of a Civic Hybrid.
The fact of the matter is, most cars are reliable for the first 75,000 miles or so and, with a few exceptions, most guns are reliable for the first 1000-ish rounds (we’ll leave the “break-in” discussion for another day). Because I don’t buy a gun a month, I need to know that what I buy will last, and the 2000 Round Challenge is good way to mimic the life of an ordinary (non-competition) gun and how a typical plinker/CCW gun owner will treat their gun.
So how do car magazines review cars?
- Define the genre. Fortunately for the car magazines, this has already been done for them. We know (for the most part) what a Grand Touring car is supposed to be like. We know what a minivan is. We know what a family car is supposed to do. Yes, there are cars out there that bend (or break) the rules, but they are dealt with as the rule-breakers they are.
- Define what makes one product within the genre better than others. 0-60 speeds don’t mean as much to a minivan test as interior space does, and the amount of trunk space in a sports car doesn’t mean squat compared to it’s time in the quarter-mile. We know and can test for 0-60 and interior volume and slalom
- Test within each genre. There is no subsitute for competition, and the value of any given object is ALWAYS tied to similar products. Testing a Dodge Caravan against a Vidper is fun and all that, but testing a Caravan against other minivans is wear the rubber (literally) meets the road.
- Reduce the human element. Just like Hamster’s love of all things Porsche, I’m going to come down in favour of CZ’s. Top Gear (and other car-related media) eliminate that variable by spreading out the test across a number of testers, allowing for results that are more than one person’s opinion.
Also, be honest about your subjectivity. If you like 1911’s, make sure your audience knows this. I’m not saying lead with it in every review, but be objective about your subjectivity.
- Treat first impressions as first impressions, and leave the real test for later. Yes, you’ll see “FIRST LOOK AT THE NEW Z-75 SUPERSPEEDSTER” on the front cover of a car mag, but you’ll also see a follow-up article a few months later when they test it against its peers. The first article is wonderful for people who like new cars, the second is wonderful for people who actually buy them.
- Go for the long-term. In any given car magazine, after you’ve waded through the editorials and the cover stories and the endless advertisements for all-weather floor mats, you’ll find the long-term road tests. This is what car owners need: We don’t really care about cars we’ll never own, we care about what we’ll need to do to keep the car we have running. And if it turns out the long-term tests show there might be major repairs in our future, THEN we’ll start to look at all those tests and reviews in the front of the magazine.
Ok, how does this relate to guns?
- Define the genre. We know what a compact defensive pistol is supposed to do. We know what a pocket pistol/backup gun is supposed to do. We know why some shotguns are better for trap shooting versus duck hunting. Yes, there are guns that break the genre (the Taurus Judge, for one), but for the most part, we know what task a given gun is supposed to perform.
- Define what makes one product within the genre better than others. What are the criteria for a good dailycarry gun or home defensive pistol or self-defense shotgun, and more importantly, how can those criteria be measured so that can make meaningful comparisons within a given genre? Accuracy at X yards is just one of those criteria: What are the other criteria from which to measure success, do they vary from genre to genre, and how can they be measured?
This is one of the biggest failures I see right now with firearms-related media. I know that if a given car does 0-60 in 6.2 seconds but does 0.5G on skidpad, it’ll be a blast to race off a stoplight but uttterly suck in a corner. Similarly, seating capacity and cupoholders are important in judging minivans: In a sports car? Not so much.
There are precious few comparisons of similiar performance-based data points in the firearms review business, and further discussion needs to happen about this issue. Larry Potterfield and Michael Bane are on the right track, but more thinking on this line needs to happen in order to create metrics that are portable across all gun genres.
One of the advantages of a metric-driven firearms review process is that it will show what the level of dedication is to getting it right. Not everyone can set up a skidpad or a slalom course and not every owns a radar gun, so I (and others) treat a Car And Driver review with more gravitas than other source. Taking the time to invest in such things and using them properly shows a commitment to the cause that others will respect.
- Test within each genre. Can Gen 4 Glocks hang with M&P’s? Is the premium price of a premium brand like Sig or H+K worth it? What is a good, inexpensive gun for trap? Once we’ve defined the criteria, we can test to them and come up with a winner based on the data, not on one person’s opinion.
- Reduce the human element. Test across multiple people. What works for me may not work for someone else, but eventually, some form of consensus will be reached. If you’re a blogger, link to other bloggers or sites who’ve reviewed the gun you’re reviewing. If you’re gun writer for a magazine, give someone else some time behind the trigger.
- Treat first impressions as first impressions, and leave the real test for later. I agree 100% with Andrew:
You do a disservice to your readers/viewers (if any) and you take up space in rankings that should be occupied by people like Caleb at RomeoTangoBravo. When he comes across something new, he calls it “(Product) Hits the Shelves” or “(Product) First Impressions.” See? Honesty.
If I wanted a press release, I’d download from the manufacturer.
- Go for the long-term. Aye, there’s the rub. This will involve acutally owning the guns we review, and not just borrowing from the factory for “Test and Evaluation”. How do we get around that? Links. Post in the 2000 Round Challenge thread. Maybe setup a list much like Jay’s Dead Goblin Count, showing what guns make the 2000 Round Challenge and which do not. Let’s tap into the wealth of gun owners out there that can help the rest of us figure out what is a reliable gun and what is not and use it to all our advantage.