A new approach to MMO reviews
- February 01, 2010 10:26 AM PST
Reviewing an MMO is difficult for a variety of reasons, not least of which is the enormous chunk of time that must be invested to fully appreciate it. In this article, we look at some of the challenges, and propose some new ways to approach criticism of online games.
Editor's Note: We have been discussing the challenges of reviewing online games for some time here at GamePro. As the experiences continue to evolve, the requirements of our process must evolve too. Already this year there are a number of games that demand a different approach from our reviewers due to the reliance on online play and community, with Sony's MAG and White Knight Chronicles leading the charge, followed by this week's release of Atari's epic sci-fi MMO, Star Trek Online. To help us craft a new approach, particularly with regard to MMOs, we have enlisted the help of Kyle Horner, a regular contributor to the MMO blog Massively, and author of its weekly Digital Continuum column to form a manifesto of sorts. Below, he outlines the challenges of "reviewing" a game experience that is in a state of constant flux, and proposes a solution for reviewing MMOs. Kyle will be putting his money where his mouth is in the coming weeks too. He will be employing the approach discussed here in his first review for us as he tackles Star Trek Online.
- John Davison
The current model for MMO reviews is overwhelmingly old fashioned. It takes a single snapshot of a quintessentially massive game and expects that to serve the reader. If you were to play EVE Online today based on a review from launch, you'd discover drastic changes to almost every aspect of the game. EVE of 2010 is quite unlike the version that existed back in May of 2003, and it's not alone in this regard.
The very nature of the massively multiplayer genre causes its games to grow through content updates of varying sizes and substantial community involvement. These games evolve while their reviews are frozen in perpetual sameness. That needs to change, and I've got a proposition on how to do it.
Reviews ultimately do two things: They serve as a "buy or don't buy" guide for the interested reader and evaluate the years of investment made (and yet to be made) by developers. Even now, the way in which an MMO grows over time is evolving in new directions. World of Warcraft and Age of Conan are adding expansions that revamp those games from the starting zones to endgame instead of simply raising the level cap and tossing in a few new features. How do you even begin to review something like that?
Star Trek Online will be the first game we review using the new approach to MMO reviews discussed in this article. For that reason, our full review of the game will be delayed until the reviewer is able to play the game over an extended period of time and invite your questions and commentary.
The fact of the matter is that both players and developers deserve a review that better reflects the investment required to make and grow these games past release. This reality asks for another kind of investment from the reviewer as well. Most games come with several classes that need to be played, more than one starting zone, both player versus environment (PvE) and player versus player (PvP) elements and even a crafting system that can be explored.
"Time is everything." has never been a truer statement than when it's in the context of reviewing MMOs. Most, if not all of these games are about time management. In order to provide a fair assessment of these games, most of them require varying amounts of a player's time and (consequently) money. Some are satisfying right from the start, while others have a slow burn. Some start off ablaze before sizzling out soon thereafter. It's for all these reasons and more that MMO reviews absolutely need to evolve along with the games they're critiquing.
The Proposition
I'm going to say it up front: coming up with a perfect solution won't be easy, but we need to tackle this now so we can begin evolving the overall process. We need to be looking at a "photo album" of the game as it's experienced over a longer period of time, instead of just taking a "snapshot" of it, and this takes considerably more time and effort. Total play time could be extraordinarily long before our reviewer is sufficiently experienced in a given MMO before being able to effectively inform others.
In a perfect world, our author could hold off on writing much of anything about "X of Y Online" until that truly authoritative voice is acquired. However, this doesn't serve the interest of readers left wanting to learn about a title's quality on launch day. That's where the idea of multifaceted reviews comes into play.
Long-term, flexible thinking should drive the new MMO review. The idea is this: Throughout the beta phase, our reviewer plays their assigned game with a critical eye towards what's working now and what isn't. On launch day, said author writes a report on their beta experience up to the execution of launch. The report won't, however, be all encompassing like a true "review." At this point, the goal isn't a "final word," instead, the objective should be to inform the person who's teetering on a day one or early purchase with a general assessment of the early aspects of the game. For everyone else, there's the second stage of the MMO review.
With a launch report delivered, the author gives readers a date when they can come back for a more comprehensive review. They experience the live game for a week or two and return with a greater understanding of the early-to-mid portions of the game. This approach is designed to help grant insight into a game's less immediate qualities like community, the pace of leveling, developer support and (possibly) early end-game features. Elements like these are significantly under-represented in many of today's reviews, and GamePro itself has previously dodged reviewing MMOs at all. Considering that many people who purchase the MMO will continue their experience for months, they could benefit tremendously by a second look that digs deeper and discusses how it holds up.
Getting it Right
Beyond these steps is where your input becomes essential. After all, these reviews are for you. Lots of questions crop up but, "How many times and at what intervals should we take another look?" is a big one. Some would say a month, others would say three and yet some might say even longer.
There's also the matter of what long-term elements are the most interesting to read about; community, leveling curve, content updates, end-game, bug fixes, etc. All these pieces are meaningful to readers, but it's important to know to what degree each one matters. And for all we know there could be other areas of importance we haven't thought of focusing on.
Perhaps you've got a specific issue with MMO reviews but don't quite know how to approach the problem. That's also something worth hearing for a reviewer, especially if they think of a potential way to solve the issue.
This proposition isn't perfect. Mistakes will be made, and that's part of the process, too. That's why your input will be hugely important; communication between the community and our reviewer is key. Without that, there's no way to know how to begin improving an MMO review, let alone gauge the improvement post-review.
To get the kind of review process that delivers useful information to a wide variety of readers (and fair feedback to a developer) GamePro will be asking its writers for something above and beyond what we've provided before. We have an idea of the problem and how to fix some of it, but your thoughts are vital to the success of this new experiment. The goal is to create and meet a higher MMO review standard, and we can't do that without you.
Like this story? You may also enjoy these articles.
Comments [19]
-
- 1
- 2
-
- Feb 01 2010 at 06:52:43:PM PST
-
I like this new approach. I was always a little curious as to how reviewing an online game the day it hit seemed like a valid review when a lot of the complaints in the first review would be obsolete at the first patch. However, the doesn't give developers an excuse to ship a broken game mind you, but now, in the ever changing world of online games, a serious complaint today is a memory tomorrow.
I mean, I know that for the magazine end, you guys have only a certain amount of space to give to all your games, and revisiting a game isn't an easy thing to do, because of that limited space, but I do believe that games that have an online only content, whether they be MMORPGs, or shooters like M.A.G., can't be fully experienced in the first week, month, or sometimes even the first year of playing them. They're constantly being fixed, upgraded, and changed, so they're hard as hell to stay current on.
In the very least, I give props to GamePro to finally realizing that there was a problem with the review setup of some games, and that first review won't be representative of the product in a few months from now.
- Vote:
- Down
- Up
- +1
- report user
wow some PC love
+1 Gamepro
I'm hoping for a STALKER: Call of Pripyat review
And smart idea about delaying your MMO reviews.
MMO games don't really start until you hit end game. Plus launches have a lot of problems
- Vote:
- Down
- Up
- +1
- report user
S4NDM4N_v2 wrote:
wow some PC love+1 Gamepro
I'm hoping for a STALKER: Call of Pripyat review
And smart idea about delaying your MMO reviews.
MMO games don't really start until you hit end game. Plus launches have a lot of problems
It's interesting that you bring up the fact that MMOs don't start until the endgame, as it's something we've been discussing internally with regard to assigning a score to a review. At what point is it appropriate? My hunch is that it's going to be different for every game, as they are designed with different of play in mind (STO seems to be very much a "sandbox" MMO) but do you think that the player must have reached the "endgame" (if the game has one) before assigning a score?
- Vote:
- Down
- Up
- +1
- report user
The thing is, MMOs are all different and so hard to review. I actually feel bad for reviewers haha
Endgame is different for all MMO games and all MMO games are set up differently.
For example, EVE Online's end game is just pure PvP. The beginning is just a long ass tutorial for the complicated game, and then you join a corporation after say 2-5 months and things go all PvP from there. No PvE at all.
Another problem with MMOs are that they specialize in different things. EVE Online is all PvP. Lord of the Rings Online is all PvE etc..
THe best possible way to review these games, is to constantly keep your review or quick look up to date. Not a huge article, but just something quick describing the pros and cons. You could comment on how the beginning of the game is, mid game, PvP, Endgame, PvE, RvR etc... It is so tedious, but MMO reviews need to be very in-depth and kept up to date unfortunately. Not like an update every week for two years. I'm thinking of something like this:
You put a preview of the beta of Star Trek.
When Star Trek comes out, you post your thoughts so far on the game, the pros and cons and keep on doing this until you reach endgame. Talk about PvP and the community as well.
Awhile after reaching endgame, you would write a review, and maybe update it every 5 or 6 months. Like a short update, say what has changed, improved or gotten worse, talk about the community. Community is important!
MMOs evolve constantly. Age of Conan needs new reviews because of how different the game is after one year.
I wish I could explain it better, but I'm in a bit of a rush
- Vote:
- Down
- Up
- +1
- report user
The current model for MMO reviews is overwhelmingly old fashioned. It takes a single snapshot of a quintessentially massive game and expects that to serve the reader. If you were to play EVE Online today based on a review from launch, you'd discover drastic changes to almost every aspect of the game. EVE of 2010 is quite unlike the version that existed back in May of 2003, and it's not alone in this regard.
*Claps*
- Vote:
- Down
- Up
- +0
- report user
Thanks @S4NDM4N_v2, any and all feedback like this is EXACTLY what we want to see.
- Vote:
- Down
- Up
- +0
- report user
John, first off let me say that I trust you enough that I bought the last issue of Gamepro. I thought the redesign was nice but I didn't really connect to the content. Regardless let me just say that I've spent way too much time working on the following wall of text, I hope it helps.
Now onto the topic of MMO reviews. I think the approach outlined above is a step in the right direction. However it seems to me that regardless of how it's structured a MMO review is going to take a lot of time. Most MMO's are designed to consume time and provide the player with an apparently endless cycle of micro upgrades to keep them paying its monthly fee. The common analogy for that process is a carrot and stick.
At their core MMORPGs are about progress and social achievement, such as filling a role in group or assisting a friend. So how do you as a reviewer give one a score? How do you rate the potential social experiences you have in the game? If either of those questions had easy answers I bet I wouldn't have made this post.
It seems to me that you can describe the facts of a MMO, you can tell people about the patch notes and the class mechanics, but that's not what I would call a 'review'. When I write reviews I try to relate the experience I had with the game and how it's mechanics shaped my time with it. In the case of MMORPGs the people you play with are as much a part of the experience as the tangible mechanics. Soloing your way though one can inform you on the content but might not allow you access everything the game may have to offer.
So how do you relate over forty hours of potentially distinctive experiences in two sets of 1500-ish words? My answer for that is 'hell if I know'. The update idea is good, but perhaps a twitter like travel journal micro blog could augment it. A post a day with stuff like highlights, frustrations and events. Might not be great for every game, but it could be good for a few like Star Trek Online.
Karl Rosner
http:// data-cube.org
- Vote:
- Down
- Up
- +2
- report user
Yeah, didn't an older mag have a feature about FFXI, or something called my life in Vana'diel, or something to that effect where he'd write about his experiences every month? I think it was GMR or something, but I'm not sure. I never read the feature, because I had no interest in the game, but I remember it being a monthly thing. Perhaps you guys could do a quick journal log every month for those games. Something like that where it doesn't eat up a huge amount of page space, but is informative, and interesting. I'll actually have to go hunt down an old GMR to check and see what exactly it was about.
Anyways, those games are so expansive, that you could seriously make a monthly magazine dedicated entirely to them. I do believe that Dan Amrich (formely of GP and Dan Elektro, and then of OXM) actually works on a magazine dedicated to WoW.
- Vote:
- Down
- Up
- +0
- report user
I think end game content is HUGE when looking at a MMO from a value perspective. For instance how fast did it take for someone to reach the level cap in the WoW expansions? Two or three days of hardcore play. HARDCORE play, its the hardcore that keep MMOs alive. I know this may be an obscure reach of an example, but take Final Fantasy for instance. You can play through the elaborate lengthy campaign of a Final Fantasy game or a JRPG which ranges from 25 - 40 hours, after that your done with it right? or are you? Those games offer additional boss kills and items for those that put more effort and skill into their games, it keeps them playing.
The leveling experience also plays its part, but the key part I think in a good leveling experience is its accessibility. Is there a good tutorial? Are there a lot of options for the player to take when creating their character and playing it in the world? I remember when WoW first launched, it was the second MMO I ever played (first being Anarchy Online) it had an amazing grasp on me but it did take me a while to get a handle on the games stats system and how it effected my class so it did lose me a couple times. After years of refinement, I have a hard time staying away from the game. I feel like if there's an MMO that could beat WoW it would be one that could easily teach the player how their system works.
So I guess in the end my answer sounds a bit straightforward, but its the details that can help. I think if your MMO reaches the heart of the hardcore it will help gather in the casual, I mean even the hardcore have a tolerance for a bad leveling up experience.
So... I suppose my advice would be is to keep tabs on the developer. DLC can be a bit of a polar issue with gamers, but when it comes to MMOs, the question for everyone will always be, "When will there be more?" Like I said, end game content is huge and it's something that should be available in MMOs as soon as possible.
- Vote:
- Down
- Up
- +1
- report user
Great idea and direction for MMO reviews. Looking forward to read more stuff like this.
- Vote:
- Down
- Up
- +0
- report user
-
- 1
- 2
