Can we ever 'trust' reviews?

[Rogue Trooper]

Been posting quotes and review scores from various magazines here myself, plus sending stacks of stuff for Laird to put up, plus questioned what we read regarding how NES games were scored in the UK, so time to take things on a notch.

Can reviews ever be 'trusted'? (i use that word as seen the line 'The only reviews you can trust' used in several publications over the years).For those of us who in past or present have relied on review scores to decide what to purchase, sure many of us have come away feeling a game deserved far better score or was unfairly rated, but does it go deeper than just differing tastes or reviewer expecting something different to game he or she was given?.

Off top of my head there have been clear examples of reviewers reviewing very unfinished code, from Zzap 64's Operation Thunderbolt on C64 to C+VG and DCM's review of Half Life on Dreamcast, but i put the question:

Just how far through a game will a reviewer play before submitting their 500+ review? what changes will be made by those higher up the chain (Zzap 64 reviewers admit there was pressure from higher up's to rate magazine publishers studios games or current Ed might be on off day and say his score is what was going in).Jurno's are under pressure off deadlines, of advertisers etc, covers sell magazines, so at preview at least your going to see some startling claims of what a game will be delivering, but at crunch time, review wise, do we really see the full story?.


Few examples for food for thought:

1)Diablo II PC various US jurno's were invited over to Blizzard to see game and play unfinished code, which they did and then went away and wrote full reviews based on said beta code.

2)C64 Bobby Bearing-Zzap 64 gave the tape version a very good score for sound-there was'nt any in finished version, Zzap just went on assurance sound in tape version would be same as disk version.

3)Sega Europe PR had a disaster on their hands, disc 3 just would'nt work properly as it was from a different batch, would'nt pick up data from DC VMU, so they had 2 options, send 1st 2 discs now and then supply 3rd when they had a working copy, or send 3 discs, but 3rd would have Jap code on it, so they went for 2nd option, several reviewers admitted they only played through 1st 2 discs before reviewing, 2 others admitted they struggled on with disc 3.


4)ex-staff writer for Edge went public claiming in return for exclusive cover art, Rockstar wanted minimum score of 9 for likes of GTA series, Red Dead
Redempt. hence why review text often feels at odds with score at the bottom of page.

Examples 1+3 supplied by industry freelancer, example 2 Zzap admitted themselves.


Any thoughts people?.

[TrekMD]

There are so many factors that can go into how someone rates a game.  For folks who write them professionally there are logistical reasons, politics, and finances to consider as well.  Imagine rating a game for a system that you hate for whatever reason (an employee of the company that made the console pissed you off, for example) and then you are on a deadline to get it done.  How likely is it that you'll do a bad review of any given game? 

Then there's personal taste.  Reviews on a site like Video Game Critic can usually have their score predicted based on game difficulty.  If a game is difficult, they get low scores. 

So, can reviews be trusted completely? I don't think so.  They do give you an idea of what the game is about, though, and you have to decide for yourself whether or not you like the game.

[zapiy]

Can reviews be trusted completely? I can honestly say a huge fat NO..

Reviews are guides and thats it, why would anyone put 100% faith in something someone else says?

I never have and never will..

[AmigaJay]

ESP not from official publications!
I never go by review score as such, but read the plus and negative points, but a good game on average will be rated highly everywhere.

[TL]

Yes and no.

As a reviewer myself I like to think that people read my reviews and it helps them make an informed purchase. However I do realise that not everyone will agree with my reviews, but I also know there are plenty of people out there with the same kind of taste as me. I remember back in the day looking at who reviewed a game before I took the opinion seriously, and this is my big point. We will all find reviewers who opinions we trust because we generally agree with them and this can be very helpful.

I think there is a place for scores as long as people understand what they mean and still read the accompanying text. There is this horrible thing now where people seem to think anything under a 7 is bad, when in fact a 5 is average and 4 and below is bad. And it doesn't help when you have pressure/money from publishers to award higher scores.

When it actually comes down to scores I am definitely a fan of the 10 point system, it's easily the most concise and simple to understand. The 100 point percentage system that we were used to back in the day was deeply flawed because how can you differentiate between 86% and 87%? What tiny thing means 1%?

[Rogue Trooper]

Yes and no.

As a reviewer myself I like to think that people read my reviews and it helps them make an informed purchase. However I do realise that not everyone will agree with my reviews, but I also know there are plenty of people out there with the same kind of taste as me. I remember back in the day looking at who reviewed a game before I took the opinion seriously, and this is my big point. We will all find reviewers who opinions we trust because we generally agree with them and this can be very helpful.

I think there is a place for scores as long as people understand what they mean and still read the accompanying text. There is this horrible thing now where people seem to think anything under a 7 is bad, when in fact a 5 is average and 4 and below is bad. And it doesn't help when you have pressure/money from publishers to award higher scores.

When it actually comes down to scores I am definitely a fan of the 10 point system, it's easily the most concise and simple to understand. The 100 point percentage system that we were used to back in the day was deeply flawed because how can you differentiate between 86% and 87%? What tiny thing means 1%?

Some very good points there Laird, going to deal with reviewers you could trust 1st:Could we really trust any of them though? ALL magazinesunder same pressures with manufacturers, publishers etc threatening to pull adverts, stop giving prizes for compo's, cover art etc.Magazines needed the advertising revenue to survive, so i'm also keen to learn just what corners were cut and when by reviewers, walking the fine line between being honest and not pissing off the hand that feed them, too much, heard before of advertising managers in publications asking reviewers if they could be a little more considerate.

Gary Penn once said things always fluctuated at Newsfield between publishers wanting to suck them off and cut them off.

Scores wise? i just feel the industry has become far too focused on scores, Metacritic to blame for a lot of sins, staff bonuses are sometimes paid based on them, MS refuse to publish a game on their games on demand service unless it's scored 80+ on Metacritic etc

[Rogue Trooper]

ESP not from official publications!
I never go by review score as such, but read the plus and negative points, but a good game on average will be rated highly everywhere.

The officials are a bloody nightmare, bad enough you had O.PS2 mag giving Tomb Raider A.O.Darkness a 9/10, then months later doing a mini-piece saying did we really give....and tearing their own review to shred.This generation i've seen some sort of bizzare score war going on between Off.360 and Off.PS3 magazine, you'll get what's essentially the same game on PS3/360 bar minor cosmetic differences and yet what 1 scores high, the other will score lower and vice versa.


Found Off.DC magazine would often seem to score games quite harshly, just as to not appear to be biased.


In fairness to Off PS3 mag, review of Skyrim was very honest, pointed out just who bugged game was and said score was going to reflect that and so it scored 7 i think...at a time other publications glossing over the issues with PS3 version.

[AmigaJay]

Yeah remember off.DC mag being honest which was strange at the time for a n off. Mag. Îmo off. Nintendo mag are the worst culprits for keeping scores high.

[Rogue Trooper]

Few Edge changed their mind on in retrospect along with good few they stand by, cannot see any anti-Sega bias:

Gunstar Heroes MD 6/10-They admit they were too harsh, original score based on fact game was quite short and had little in way of hidden extras.

Myst Mac 6/10-Edge say they over-scored as it amounted to 'little more than a glorified Grattan's catalogue of sci-fi landscapes'.

Doom PC 7/10-Admit they were wrong, but still say it would have been great to form alliances (err hello? Edge? with whom exactly? your now undead fellow marines, or the hellspawn?).

Super Puyo Puyo SNES 7/10-too hars, reviewer failed to see games depth, dismissed it as blatant Tetris rip-off.

Virtua Racing MD 8/10 Ouch, piles into saying difficult to understand the impact this made on original release as game is dire, the conversion dreary.Bloody hell Edge, only so much you could do with a MD+SVP chip, shessh.

Tie Fighter PC 7/10 Too harsh at time of review, but reviewer scored it such due to moral reservations (i.e your fighting for the Empire)-seriousily? who does'nt like to play as the bad guy? gaming all about escapism, they scored it 7 based on moral grounds?.

Daytona USA Saturn 8/10 scored too highly-'A rare lapse of concentration for Edge'.

Wave Race 64 N64 9/10 Over-scoredsharp difficulty curve, too few courses prevent it from really being a 9.

Goldeneye N64 9/10-say a 10 was considered, but rejected, they stand by the 9.

Panzer Dragoon Saga Saturn 9/10.No changes.

Turok 2 N64 9/10 Scored too high, games pacing and level design lets it down.Why did it score a 9? probably due to levels of 'giddy anticipation and cover art' say Edge-Here is a very good reason i never go by Edge reviews, if score based on these 2 factors, god help me.

Sonic Adventure DC 8/10.Stand by score, yes camera was poor, but  games intricacy and beauty made them forgive the flaw.

Powerstone Dc 9/10-Tad over-scored.

Silent Hill PS1 8/10 Score stands.

Soul Caliber DC 9/10 Score stands

System Shock 2 PC 7/10.Score stands (do not agree, game is an 8 at v.least)

Crazy Taxi DC 7/10-Edge mentions the hate mail it got from 'Sega Fanboys' saying game should have been an 8, 7 given as game too short.

Excitebike 64 N64  8/10, would have got a 9, but N64 hardware limitations prevented it.Edge hoped for G.C version for smoother, more stable frame rate.

F355 Challenge DC 7/10 (which did not go down well), blames high difficulty level+wild understeer.

Severange:Blades Of Darkness PC 2/10 (Again Edge took hatemail).

Shen Mue DC 8/10.No change, you either love it or hate it.

The Bouncer PS2 3/10-'The general gaming press, however, desperate to evangelise Sony's new machine, still gave the game excessive scores'-Again, here they raise a valid point why reviews cannot be trusted so often.

Tempest 3000 Nuon 9/10.No change

Phantasy Star Online  DC 9/10 No change.

Nights Saturn 8/10 No change.

Radiant Silvergun Saturn 8/10.No change.

JSR DC 8/10. No change.

[dougtitchmarsh]

Never trust just one review. One persons opinion is just that, one persons opinion, and it has been said already, it can be biased for any number of reasons. If you can read several reviews and take an average of them then all the better.

[Arethius_RGC]

Nice subject guys. I'd like to add the point of view of a little frenchy  ?? And they corrected the game to make it easier... Now I do not know If the game sold well but ... I guess not... To bad it had its flows but could be a nice franchise...

So game developeprs use the reviews to change the game but do they have a choice

Plus I'd like to say that I usually do not agree with them I foind them always extreme detailled, like they were expecting too much from the games....
I was dispointted by games in the past but never really for the same reason that them

PS: if someone follwed w<hat I whrote please tell me I'm not sure I'm understanding myself

[DreamcastRIP]

'Can reviews be trusted?'

I'd say one first needs to establish what it is we're referring to by the word 'trusted'.

For instance, I'd like to think reputable game reviews are not only the honest opinion of the reviewer but that the reviewer himself/herself is writing in an objective manner. By that I mean the reviewer should be professional enough to not let their own personal prejudices and/or bias colour the review. A reviewer should always bear in mind that the purpose of a game review is to inform the audience in an objective manner of a game's quality - it is not an opportunity to let the world know what you think because, reality check, few if any people actually give a damn what you think.

If a reviewer is incapable of acting in such a professional manner then it's incumbent on the editor to carefully select which games said reviewer is allowed to review, e.g. it would be an irresponsible editor who let a known 'hater' of a particular game series review a game from that series. The same rule applies to not letting known fanboys review the latest game from the company and/or game series that they blindly worship - reading endless gushing hyperbole from a fanboy where they describe everything as 'amazing' while consciously glossing over any negative points does the publication/website no favours whatsoever.

There's also the issue of whether one can truly trust in the integrity of a review. If I see a publication or website that regularly features paid for adverts from a certain company but then also notice that the products of that company are given mostly +90% review scores and that said company is given prominent and consistently positive coverage then it's only reasonable to question the independence and integrity of the reviews.

It is for reasons such as these that I tend to,

* check a game's rating on metacritic as it's the average of all review scores so in theory should provide a more reasoned rating which strips out the overly positive and overly critical scores to some extent

* examine a selection of reviews that are linked on metacritic - ignoring the most gushing and most critical reviews works to some extent

* examine a selction of reviews looking to identify common positives and negatives highlighted so as to try determining if such issues are genuine (or just the subjective likes and dislikes of one person)

So, in summary, I don't consider any individual review to be truly trustworthy. A review, even a well written one that strives to be objective, can only ever be the opinion of the reviewer.

Just as I believe it would be foolish to wholeheartedly trust in the,

* comment of any individual newspaper, TV news channel, etc, for an accurate and non-biased telling of the news
* personal recommendation of one so-called expert when deciding on which car to buy, mortgage to get, etc

... I also think it best to do some proper research before committing to purchase.

[Rogue Trooper]

I've seen Bioware and few other developers say Edge's reviews are 'very much respected' in North America (by development community i'd assume), so maybe it's joined likes of RG as seeing the USA as the market that matters?.

Gamestm dropped a few clangers when it comes to reviews and they are the 1st to admit it:

Be it issue 10 review of KOTOR on Xbox where the plot twist screen was used as a screenshot, never actually reviewing Half-Life 2 as they assumed that by time they recived a review copy, everyone would already know everything about it, proclaiming PC Boiling Point:Road To Hell as being as good as PC Deus Ex and that it was more of an experience than a mere game, Bomberman act Zero (360) getting 3/10 for being ' An ugly, joyless shell of a game' in 1 issue, before getting a 7/10 for the online play, the month after.

[Rogue Trooper]

Bit more 'food for thought'

A PC mag of old, would ask reviewers to produce a 'box-out' section if a game scored over 75% to explain to the reader why game was so good, since this would mean more work for the freelance reviewers, they made sure they never scored games above 75%.

An average monthly games mag is planned, written and sent off to printers on average in 19 working days, so if your faced with reviewing a stack of games and some of them take 10+ hours, just to really get going, how much of the game will your average reviewer actually see, if they are under pressure to submit articles?.

Good scores in exchange for front page art and early reviews:Driv3r:2 UK mags awarded it 9/10 and had it's art adorning the front cover just to steal early reviews over rival publications, games was horrendous (and unfinished).

[Rogue Trooper]

Official Magazines have always been 'dodgy ground' this generation-worst example i saw Official PS3 magazine's:

Exclusive Vedict:COD:Black ops.

Notice use of word 'verdict', not review...inside a 9-page feature with 'verdicts' (all scored on a 5-Star rating) with Campain getting 4, Online getting 5, the tech a 5 feature closes with a verdict by guy who spent 3 days in L.A with developers.

Very, very poor piece of jurnolism.