IGN’s review score for Prey has caused some controversy. Dan Stapleton wrote the review, and gave Prey a 4/10. The main problem Stapleton had with the game is that it corrupted his save file. Despite receiving assistance from the developer (Arkane Studios), he could not finish the game due to the save corruption bug. Later, a patch was released updating the game and fixing the problem. Stapleton then updated the review and changed the score from a 4/10 to an 8/10.
Many people on reddit, Twitter, and in blogs across the internet complained that the game’s score on MetaCritic will be forever tainted, since MetaCritic only records a publication’s first score of a game. In an extended tweet, TotalBiscuit said IGN should have held off on scoring the game until they knew how widespread the game-breaking bug was because the game worked for his wife Genna, and “plenty of other outlets.” Erik Kain, writing as a Forbes Contributor, agreed: “Holding off issuing a score for a game for any number of reasons is perfectly fair so long as you explain to readers why…If the game’s issues aren’t resolved in a timely fashion, and especially if they’re pervasive (The Master Chief Collection comes to mind) then by all means, have at it. A score should be impacted by missing or broken features, it just doesn’t hurt to see if a quick patch can fix those issues.”
TotalBiscuit and Erik Kain misunderstand MetaCritic and commercial reviews. A game’s Metascore is a snapshot of its critical reception at launch. It’s an aggregation of product reviews and consumer advice given out at time of launch. Product reviews and consumer advice are distinct from serious appraisals of a game’s artistic value, which will be done by critics years down the road. Before people go out to buy a video game on launch day, they check MetaCritic to see what the general consensus on the game is, and if there are any red flags. As such, it is perfectly acceptable for a review to note a game-breaking problem he experienced hours before the product’s launch. If Dan Stapleton was supposed to do a serious artistic analysis of Prey instead of a product review, it would be right for him to wait for the product to be fixed. However, that is not what a product review is. Reviews from outlets like IGN are consumer caution and advice for early buyers. Serious bugs are more than fair game.
TotalBiscuit might acknowledge that MetaCritic is about consumer advice, not serious artistic analysis. He might insist, though, that it doesn’t matter. People look to a game’s Metascore to gauge its artistic value anyway, so we should try to get the score as close to gauging artistic value as possible. We should do that because good games deserve to be seen as good by the public. I think this view overrates how attached people are to Metascores. Gamers will obsess over it at first. It can confirm the wisdom of their purchase, their taste, their opinion about a franchise or company. However, most of all that fades away after launch.. Look at Bioshock Infinite. Its Metascore is 94, and people at launch were talking about how it was a leap forward for the medium, an instant classic, the most mature and artistic video game ever made, and so on. Today, the majority of people do not see it that way. Criticism made years later has changed the public’s view of the game. The reverse has happened for Fallout: New Vegas, which is now beloved as one of the greatest games ever by many. Its artistic stature has grown despite its relatively low Metascore. TB and Kain shouldn’t worry about protecting Prey‘s Metascore so much, because Metascore is not destiny.
The demands from TotalBiscuit and Erik Kain would defeat the purpose of reviews if implemented. A gamer considering buying Prey on day one checking MetaCritic would see a higher score than is deserved because IGN held off on bringing the score down. If IGN did what TotalBiscuit and Erik Kain wanted, they would have failed to warn customers of the save game corruption problem (which is not as rare as some people would have you believe). Why should customers suffer because TB and Kain don’t want an arbitrarily sectioned-off part of Prey to not factor into the score?
Not only would TB’s and Erik Kain’s demands defeat the purpose of reviews as consumer warning, but also as a punishment mechanism for publishers and developers. Publishers actually care about review scores because scores affect the bottom-line. Demanding that reviewers give publishers the opportunity to fix their game after it has been released and many customers have already bought it means the scores don’t affect the publisher’s bottom-line as much. This is the reason Bethesda does not send out any more early review copies. They have a pattern of releasing buggy games, and delaying low review scores due to those bugs ensures they get sales anyway. If reviewers give developers time post-launch to fix a game before giving it a low score, they fail to punish those who release broken games.
Critics of Dan Stapleton acknowledge that MetaScores function as a punishment for those who release buggy games, but they see that as a justification for the guidelines from TB and Kain. They propose that reviewers should delay their scores (or reduce a score later if bugs aren’t fixed) because their scores can hurt the bottom-line of those who released the buggy game. This demand makes reviews ethically compromised. If reviewers accept “this will hurt the bottom-line of the company whose product I am reviewing” as a justification for changing or delaying their score, then reviewing is basically dead anyway. At that point, every review might as well be written by Nintendo Power or Official Xbox Magazine. It almost seems too obvious to say, but reviews should be honest and independent. The last thing in a reviewer’s mind should be the bottom-line of the developer.
Let’s say reviewers decide to follow their advice, anyway. Erik Kain says he will accept a reviewer marking down a game if the bugs are “pervasive” and are not resolved in a “timely fashion.” The game has released, and the reviewer cannot finish it due to a game-breaking bug despite receiving help from the developer. How long should that reviewer wait before he is allowed to honestly score the game? Hours after release? Days? Weeks? How many hundreds of thousands of customers have to buy a game before the reviewer is allowed to score it? How many patches does a developer get to push out? How prevalent does a bug have to be to be marked down? How many thousands of people affected by a bug of what magnitude constitutes a point deduction? Which methods of data-collecting should a reviewer use? Can he just do a Twitter poll, or should he search the Steam Forums for bug reports? Does he need to hire an outside firm to contact customers of the game and ask them about their experiences?
These questions all point to the fact that Erik Kain is demanding something absurd from reviewers. Reviewers cannot know for sure how prevalent a bug is, and there is no way to figure out how much to mark a game down based on estimates of how common a bug is. Even if reviewers could reliably do so, crowdsourcing information about how buggy a game is defeats the purpose of a review. The only reasonable alternative is for a reviewer to review a game based on his experience with it, which is what Dan Stapleton did.
If reviews were done as TotalBiscuit and Erik Kain wanted them to be done, they would be pointless. They would artificially inflate a game’s MetaCritic score by overlooking its flaws when scoring. They would fail to warn customers who check MetaCritic for low scores. They would fail to punish publishers for releasing broken games. They would be ethically compromised, too. TotalBiscuit, long-lauded as a consumer-first critic and a bastion of ethical judgment, misses these facts. The same goes for Erik Kain, whose popularity rose when he sided with GamerGate, a movement supposedly about consumer advocacy and ethics in journalism. If these are the best consumer-advocates we have, then we may be in trouble.
I have a few other thoughts on the issue that aren’t of much consequence:
- Stapleton’s review is already somewhat unfairly charitable toward the game because he received personal support from the developers. I think it’s safe to assume reviewers get better customer service that ordinary gamers.
- TotalBiscuit suggests that bugs should be given lee-way in scores because PC builds vary so wildly compared to consoles. I’m no PC gaming expert, but I don’t think save corruption is really a category of bug affected by different builds. I understand the developer can’t control performance in the same way that they can on console, but is save corruption really affected in the same way? I would believe it if TB were talking about some random schmuck. But I have to assume Dan Stapleton is competent enough not to cause save corruption issues on his own.
- In internet terms, this controversy is ancient. It occurred around two weeks ago. However, there was a surprising lack of push-back to TotalBiscuit and Erik Kain. I hope this post is at least helpful for posterity and future reference, even if I didn’t strike while the iron was hot.
- TB seems to assume that people will come away from a review like Stapleton’s and assume that the game is broken for everyone. Not only is this not true because reviewers regularly say things like “I don’t know how widespread this big will be come release day,” but also because MetaCritic serves as a sample of reviewers. As of now, it shows that Stapleton was the only mainstream reviewer to have been unable to finish the game due to the bug.
- When in doubt, the simplest thing to expect from reviewers is a honest assessment of the product they are reviewing. This is the first principle of game reviews.