I want to get more opinions and arguments for and against for a blog to grade shows using some kind of quantified grading system.
Basically, if someone were to review an episode of a series, by writing a review and ending it with a letter grade (or a 1-10 or 1-5 or 1-4 or whatever), how would you do it? Would you even do it at all?
The question is pretty hard to answer because I think the better one to ask is why are we using a grading system at all? Confession: the only grade I read is from Psgels’s, mainly for the sake of checking out his general impression on an episode without reading the actual review, to avoid being spoiled. And to me that’s the real strength in a grade system like Metacritic or RottenTomatoes–you get an idea of value without looking into it too much. Of course, it is not really a precise measurement on a personal level nor is it a particularly accurate one. Or I should say, it comes with a margin for error and often times reading one reviewer you trust thoroughly can often give a higher quality impression than seeing a score based on hundreds of thumbs up or down.
Actually, I’m more curious as to why you would give a grade? On the one hand, some people enjoy grades for reasons I mentioned above, and more. Some people only care about grades, actually. And sometimes it is another mean to express what you’ve failed to express in a TL;DR post. I mean I’ve read my fair share of episodic blogs and far majority of them don’t have anything to say that you can’t pick up from the screen caps they’ve also posted, so having a grade system actually adds to what these blogs offer.
On the other hand, for every show you note down this way you run the risk of making some kind of over- or underestimate. This is particularly the case in terms of how people judge shows by cover while ignoring that margin of error we all work with when doing so. This is particularly true when anime tend to be serial and each episode is merely building up some kind of bigger picture where the picture is greater than the sum of its parts. It’s kind of a meaningless thing to do. To that extent Psgels uses his impression-based measurement which makes a lot more sense to me, so maybe that objection can be avoided by using a smarter metric.
Some people take this seriously. I’m not sure I do, but I can see why you would want to. I don’t know, any good ideas? I am not really going to implement a grade system, but someone else might, and I want to get some ideas as to why it is a good idea, what is a good way about it, and how do you manage people’s expectations with them.
March 20th, 2011 at 11:10 pm
ANN has the best grading. It sets rather sensible criteria for each grade that are easy to follow, which in turn permits cross-account stats. If people were giving grades arbitrarily, ANN’s histogram would never work.
March 20th, 2011 at 11:21 pm
Are you referring to their review grading? To me that makes no sense, and from what I can discern it is arbitrary to the reviewer.
As for their user-submitted stuff, it’s not quite what I’m looking for.
March 20th, 2011 at 11:34 pm
I personally don’t believe that overall grades are indicative of anything. What ends up happening is that most grades that you give out will be rather subjective. It might be better to just essentially give out grades by category (i.e. Music, Plot, Pacing, etc) and average them all together, but that’s still pretty messy and not really indicative. I also feel that when you begin breaking things down like that, you begin to deviate away from a more “subjective” blog and into the realm of objective review, which might not be your intent.
March 21st, 2011 at 5:42 am
A less-detailed description is equal or less indicative than a highly-detailed description? Seems to make sense to me. But that’s accepted.
March 21st, 2011 at 6:44 am
Re Zaitcev: I believe he’s talking about the graph you see for each specific entry in the ANN encyclopedia, where you see how the show works for a variety of people.
March 21st, 2011 at 8:39 am
Right, the user-input stuff. Which is not what I am interested in.
March 21st, 2011 at 4:49 pm
I think you should keep it to small numbers. What’s the difference between a 7/10 and an 8/10? Probably a very personal difference. But the difference between a 4/5 and a 5/5 is clearer because there are probably some obvious reasons as to why that difference exists. It’s also something more people can agree on. I think there’s a pretty good sense in the difference when you rate out of 5 stars.
1 star – Absolute horses of shit
2 stars – It wasn’t terrible but that’s about all you can say.
3 stars – Average, you might like it.
4 stars – A good solid watch, probably won’t disappoint.
5 stars – Why haven’t you watched this yet!?
I’m more in favor of the twitter conclusion – write an impression in 140 character that replaces the rating system.
Side note, the rating I give to my MAL list: Cowboy Bebop is a perfect 8/10, everything else is based off of that.
March 21st, 2011 at 4:59 pm
Twitter is tl;dr for the stuff I’m thinking of, which is, well, Metacritic-style aggregation.
I mean, I can understand the ANN style of reviews, but that doesn’t factor in user bias in the sense that only a few shows garner the largest audience, and generally speaking people only rate things they watch, and people only watch things they like, so…
It’s a different sort of statistic sampling, so, again the ANN or MAL style of rating is not what I’m looking for.
March 21st, 2011 at 11:36 pm
What about something simpler:
* watch
* if you have time
* don’t bother
and a “watch now!” for very select titles. I mean, its not like there is that much difference between a 4 and a 5, or a 7 and an 8.
March 22nd, 2011 at 5:18 am
Perhaps the main problem with the grading systems people typically use is the lack of distinction between appreciation and enjoyment, so that an average grade can mean not just “meh” but also “I had fun watching this but honestly, it’s crap”. For number crunching purposes, I guess averaging “appreciation” grades can make sense, whereas some sort of deviation metric is more suitable to an “enjoyment” scale (to test if the show has narrow or wide appeal, subjectively).
I often use my own MAL grades to suggest shows for anime club showings, but I couldn’t pass them over to anyone else in the club for that purpose (or rely on someone else’s grades about shows I don’t know) because of that compounding-appreciation-and-enjoyment problem. So maybe having two scales would make such a system easier to read.
March 22nd, 2011 at 12:32 pm
thanks for the input guys.
mt-i: that is basically what psgel uses in his episodic things. a 2-dimensional grade system makes sense to me too. Well, something to think about.