Reviewing A Bad Review

Amidst the current tide of really great reviews and praise that has been coming in for AI War (and which I am very grateful for), comes the “on trial” review of the demo in the Indie Gaming Mag, which gave it a cumulative 3.66 out of 10. Don’t bother trying to read the review online, it’s something you have to buy a subscription to their magazine to read (Update: Now they’ve put it online. Hooray.).

Disclaimer: I am pretty pissed off, which is hard state to get me to. However, I’ll try not to let that color this post too much, because I really do think that IGM is doing some things very, very wrong with their “on trial” system.

The “On Trial” Review System: “N/A” For Uninstallable?
All games in the “On Trial” section are looked at by four reviewers at IGM. The four for AI War scored the game 7, 1, 3, and N/A respectively. I’ve noticed a surprising number of “N/A” reviews in IGM from the two issues I’ve looked at. Essentially, anytime a reviewer can’t get a game to run or install on the first try, for whatever reason, they give up and post an N/A. In the “on trial feature” in the current issue #6, there is an N/A score for two out of the ten games, for instance (one out of the four reviews of each of the two games getting an N/A, that is). It seems like there were more than that in issue #5, but I may be misremembering; I can’t find my copy of that issue, and the download link has expired.

I found it odd when it was other people’s games they were reviewing, and figured those games must be very poor indeed if a reviewer could not get them to install, but now I see how little effort they actually put into it. It seems almost downright irresponsible for a reviewer to do something like that. If reviewers gave up every time a AAA title did not install or run correctly on the first try, there would be a lot of N/A reviews in most magazines. This is often not the fault of the game (though sometimes it is), but rather is something that just sucks about PCs in general. I love the PC, but — news flash — Windows is not always the most stable, sensible, or reliable.

Also? Because Windows is so unreliable, it takes experts on installing software in order to write software to install other software (say that 10 times fast). Almost no game developers code their own installers (I am tempted to say that no game developers do, but I can’t be certain there isn’t one out there who did for some reason). So, what you are really reviewing when you review the installer is some other third party (in our case Advanced Installer). There are just so many things wrong with this I don’t know where to begin, but I will say that for a magazine supposedly devoted to indie games, it’s funny to complain about something so outside the developer’s control — did I mention that installer software is fricking expensive? Advanced Installer was over $500, and it was one of the cheaper alternatives.

Bottom line to players is that they want games to install and run on the first try. I get that, and I agree. I think that any reasonable developer tries to make that happen. But this is Windows, folks. I think we all understand that things will not go perfectly every time. Still, with a dozen or so reviews and an undisclosed-but-bigger-than-a-breadbox number of customers, I’m not fighting with installer issues on an ongoing basis with AI War. People seem to be working it out pretty straightforwardly in most cases, aside from a few odd edge cases where either some other sort of software is interfering, a corrupt copy of the installer is present, or their copy of Windows is just plain effed up.

Music and Sound
The music was “fairly torturous” to one IGM reviewer and “annoying” to another. Fricking ouch. Some other reviewers outside of IGM have also been nonplussed by the music, or didn’t mention it at all, but one said “the music is beautiful, engaging and compliments the game wonderfully” (Steve Blanch, Bytten). Others have also commented on the fact that it was good, and it has also been heavily complemented by a number of actual players (one calling it “some of the best music I’ve heard in any Western game.”).

On the flip side, some reviewers have very rightly pointed out that the sound effects are like “extremely small fireworks” (James Allen, Out of Eight Reviews). These reviewers have a great point, and this was justly deserved criticism of the game. Sound effects are something we are working on for future free DLC, to make those more imposing and interesting.

I don’t understand the polarization in opinions on the music, perhaps it has to do with occasional drops in sample quality — Pablo, like me, has to work with the tools he has for now. Or maybe some just don’t agree with the style, or maybe it’s just taste. I feel like Pablo is being mistreated with the caustic IGM reviews, whatever the case.

More Installer Complaints
The list of prerequisite downloads, and the fact that our EULA is in the installer, really pissed off two of the reviewers. They wanted something that you can just click Next a bunch on and be done with it. While I understand that view, this would make the download around 400MB instead of 80MB. Also, SlimDX does not support that sort of silent install, which pretty much kills that right there.

Also also, for anyone who is already up to date on the .NET Framework and DirectX 9, there’s just one little install of SlimDX and the game itself and no reboots. The .NET Framework and DirectX are not exactly mysterious, unusual requirements for software. The only reboot required is from the .NET Framework 3.5, which many people should already have on their system, anyway. I don’t feel that AI War should be penalized because a reviewer hasn’t been keeping up to date with their Windows Updates.

Bad Review vs. Bad Reviewer
It’s not that I think that AI War doesn’t have any faults — it, like all games, does. And tastes vary, too. I get that. Even amongst good reviewers, there have certainly been criticisms of the game that I don’t agree with. Those I have to accept and move on. This is not what these IGM reviews are.

Some other reviewers criticize the game for not meeting up with what their ideas of what the game should be (such as: lacking story or lore, or not providing some specific play option). In those cases, as a game designer it is my job to listen to those criticisms carefully, but keep the game true to what its core experience is actually supposed to be. Some criticisms result in changes to the game (I treat critical opinions like I would suggestions from any other player suggestion). Still others I just have to chalk up to differences in taste/opinion, which happens. Every game is not for everyone.

IGM is one of the first times I’ve run up against what I felt were irresponsible reviewers. Maybe they just had too much else to do with other games that were higher on their priority list, but if that was the case they should not have tried to review AI War at all. If you are going to review a game, you should try, you know, actually playing it a bit. I noticed that, among the IGM reviewers who played AI War and talked about it, the ones who played it longer liked it the best. Is that because they liked it better, or did they like it better because they actually saw more of the actual game? I can’t begin to say, but I think the latter is a factor to some degree.

How To Review A Game You Don’t Fully Love
When I look at the recent WiiWare game My Life As A Darklord, the first hour or two of that game is absolutely boring in my opinion — too little is going on, everything is overly simplistic or mysteriously complicated in the wrong places (semi-bad tutorials), etc. And if you lose, you are treated to a lot of repetition. However, once you get past that, the game starts to make sense and is actually a load of fun. This is causing the game to get some mixed reviews, and those who recommend it still tend to caution players about the slow start — but you’ll notice that the reviewers actually played enough of the game to get an idea of what is going on.

Furthermore, if you’re just not into a genre it’s okay to say so. There’s no way I would ever try to review a sports game or a fighting game — I just don’t play those, and I couldn’t tell you whether they are good or not. In one sense they all seem pretty uninteresting and mediocre to me, even if they do have pretty graphics — and the interface of fighting games is unintelligible because I am just not practiced in the art of button combos. Does that make those genres worthless and bad? Of course not. It just means I am not their target audience. I actually have a lot of respect for what both genres do, and they certainly are well-loved by many players (including a number of my friends). I’m not insulted that those other genres exist, and I’m not interested in them simplifying to try to bring me into the fold. We can just be friends, and that’s okay.

Other Chicanery
The tutorials in AI War have largely been praised and have seemed to be pretty effective in getting players up to speed. But it seems clear to me that the IGM reviewers, some of them at least, were interested in getting into the game and back out as fast as possible. I got the same impression with Mike and Caspian’s reviews of Dark Souls, another indie game skewered in IGM issue 6. Having not played Dark Souls I can’t comment on its quality, but from Sam’s and Kayla’s reviews in IGM I am definitely intrigued. Next to them, the comments from Mike and Caspian seem downright flippant, though. Given the game’s review on Game Tunnel, I think I will definitely have to try it out.

Light of Altair is another indie title that I have not played but which seems to have been doing well enough for itself in terms of reviews, and which seems to be doing even better with sales. The verdict in IGM is a 5.75, which is several steps lower than other reviews have been. This in itself is not terribly insidious — perhaps they are just harsh graders. But I find this telling of the fact that they averaged their scores from non-genre fans and genre fans alike. Their scores for the game were 5, 3, 7, and 8. I’m not sure that averaging those into one number gives any real meaningful information to me about Light of Altair.

Light of Altair is a game that looks to not be my cup of tea for a few reasons, but I respect the game and from what I have seen it deserves more than a 5.75, which is a pretty miserable score (and yet so much better than AI War’s 3.66 score, which is what I would give to the guttertrash Wii cash-in games that are sold for $10 — thanks very much, IGM).

See how easy it is to say “this game’s not for me, but if you’re into it’s style of play you might like it?”

“Style”
A big thing in a lot of the IGM reviews seems to be the “style” of the game. Does the game have lots of “style” and is it consistent throughout? If yes, hooray. If not, as claimed with AI War, then what were the developers thinking?

Answer: there is more to games than just visual style, and we had a $0 budget at start. If you limit your interest to games with a level of polish like the AAA titles, you will only get indie games from previous AAA developers. Oh, but wait, even the art in Light of Altair was criticized off and on by IGM.

We are sure spending a lot of time talking about art in reviews of indie games, which I thought were supposed to be about more. I know that the art in AI War is not the best. We are actively working on that now that the game is bringing in some money to pay for its own upgrades, but even with all of these upgrades it is not going to look like Homeworld or Prince of Persia or something.

That is, uh, a big part of the difference between most AAA titles and most indie titles. Some indie games look absolutely stellar, but my complaint with them tends to be that they are then too short. Budgets are limited at indie developers, this is an almost universal truth, so you are either going to get a lot of content and lesser production values, or AAA level production values and less content. I’m probably always going to be in the former group, primarily because I value function over form, but also because that’s what I’m good at making (I don’t have a AAA gaming development pedigree, my background is in business software).

In all fairness, IGM did give a pretty favorable review to Blueberry Garden (7.75 is not a stellar review in my book), and a very good review to Plants versus Zombies. Blueberry has what I would call fairly lackluster graphics, but it has the mysterious “style.” As does Plants vs. Zombies. I can’t actually disagree with that, both games are quite stylish despite the other detractions, although I do disagree that some of the other games in their list — AI War included — don’t have a style of their own. I thought Light of Altair was one of the more stylish-looking ones, but it was criticized for certain lapses.

Are all of the above prettier than AI War at the moment? Yes, in most respects. We’re working on that, but still I don’t feel that AI War is unattractive. Does the game lack visual personality? Yes. So does Chess. Maybe that’s part of what they were talking about with “style,” which they never bother to explain.

Celebrity Obsession
In the end, looking at the games that were scored relatively well and which received the most attention in IGM, I see that it is mostly the same list that has been getting a lot of attention in the mainstream gaming press. Those games are covered by the mainstream press for a reason — they are excellent — but isn’t there supposed to be more out there? Shouldn’t we actually give more time to games we’ve never heard of?

Put In Some Time, Please, Or Don’t Bother At All
Instead, IGM reviewer Mike complains about AI War: “Honestly, I was so fed up with
this tactical game that I didn’t give the gameplay much of a chance.” His prior complaints being the installer, the pixelart, the “annoying music,” and the fact that there were “three extensive tutorials.” Incisive journalism there. I similarly enjoyed his other reviews of the bottom-five rated games in IGM #6.

IGM reviewer Caspian notes that the tutorial is “particularly dull,” as contrasted with positive reviews elsewhere such as “It is rare to see a tutorial that not only lets the player jump right into the action, but also entertains while instructing the player” (Chris Beck, The Wargamer). Caspian also criticizes the interface as being “pretty bad an unintuitive,” compared to others calling it “a great interface” (Beck) or “the interface in AI War is almost completely fantastic” (Allen).

Caspian also asks “why break convention?” This is an excellent question, and one that Tom Chick does a much better job of answering than I could: “because it’s big, different, entirely unprecedented and an exciting way to play an RTS.” Caspian further snarks that “It’s as if absolutely no external testing managed to provide feedback to the developers.” Which is just hilarious, given our growing reputation as one of the developers most open to player feedback. The interface isn’t graphical pretty at the moment (another thing on our list to upgrade), but it’s extremely functional and has been ever further refined with the help of dozens of players.

One telling fact about how little time Caspian put into the game is the fact that he claims “The tutorial stages can’t even be skipped.” This is patently untrue. The game shows the tutorials menu when you first load the game, with a big button for “Main Menu” if you prefer not to do them first (although, you’ll be pretty lost without them, because this game is so different from most RTS titles). The second time you open the game, it takes you right to the main menu — so this reviewer never opened it more than once.

Finally, Caspian notes that the game is “essentially… the same fare as pretty much any other RTS.” Well done, sir. Thanks for that. Not only did none of the IGM reviewers actually look at or comment on the effectiveness of the AI itself (which is admittedly hard to review unless you sink in a lot of time), they also never mentioned it. It seems that having AI that is “some of the best I’ve seen” (Blanch) doesn’t warrant any discussion or investigation at all, or count as a huge differentiating factor. Similarly, with no actual investigation into more than the most basic parts of the interface and mechanics, there’s no time to discover that “there’s a lot of 2009 to go, but I’ll be surprised if anyone else twists the RTS formula this dramatically and this effectively” (Chick).

In Conclusion
There’s a hell of a lot of snark in IGM, and consequently I’ve been inclined to respond with more snark than I normally would. Normally I try to respect the opinions of others, and allow for differences of opinion. I’ve weathered quite a few beatings and a fair bit of commentary that I don’t agree with, without ever resorting to snark. But this manhandling by IGM is just too ridiculous.

First, I highly recommend if you are an indie developer that you give these guys a miss. Either you are already critically popular and you don’t need these guys to reconfirm it, or you are not yet critically popular and you don’t need a trashing from them. I’m not sure what the goals of this magazine are — it’s certainly not supporting or nurturing upcoming indie developers, since they are pretty brutal when there is the slightest thing they don’t like. I guess they are going for sort of a Simon-from-American-Idol vibe? That’s entertaining to a lot of people (not me), so maybe that’s what it is.

In fact, a lot of their reviews — from the way they talk about their favorites to the way they trash those they don’t like — are reminiscent of Idol to me. If I’d known, I’d have never volunteered to be a part of it. My resolution in general is to cast a wide net and let no potential opportunity pass by (this is the only way to make it as an indie developer), but these guys seem to be trying to drive indie developers out of making games. The game Enlightenus “stomps out any flashes of inspiration” in its own design, according to IGM; Hollywood Tycoon is “great in its ability to admit the existing of a sci-fi romantic comedy, but little else;” and even Blueberry Garden “would have been much better as a free Flash game.”

Way to support the indie community, guys. I don’t know what else to say than that.