Pointless Debate #5: Subjective Ranking Systems

#392: Eiken will not be bothering any of my top lists

It’s a pretty common thing for people to have a top 10 list of favorite things, or worst 5 shows ever, or any other completely subjective thing ever. As a result of something commonly known as boredom, I figured I would spend some time putting together a complete ranking of every anime I had ever completed. In the process, I made a few discoveries about myself and the shows that I watch.

#387: Green Green might be worth resurrecting my Worst Things I've Ever Watched feature.

The process began by figuring out a way to copy over my completed list from MAL into a spreadsheet. As annoying as having to erase meta tags and data was, I eventually got it working.

Then came the process of actually ranking them. MAL’s 1-10 scale would work if I had watched 10 or less anime, obviously not for 392 items, but I kept my default rankings in tact. I then went through and gave each item a score of between 1-100 just on gut feeling since I thought this would be the most fair way. I also did not compare them to anything else I had already ranked.

Already two things stood out as I was ranking items. I had not ranked anything as a 10 or a 1 when I had originally scored them on MAL. Remembering why I had done this wasn’t much of a mystery; I thought there was some fundamental flaw that couldn’t justify a 10 in everything. Stupid logic I know, but I’ve never gone back and revised them. The other thing was the difference in scores between my rough 100 point scale and the original 1-10. Scores of 4 suddenly turned into 68, 9s turned into mid-70s scores. This made me realized that a lot of ranking comes down to how I felt at the moment I put a score in. A score from late 2007 didn’t carry over to today.

When that was done, I went through the ties and for the first time began to compare series against each other. While there was surprisingly an uncontested number 1, suddenly I was weighing one thing against the other, but I limited myself to 2-3 seconds to think about each one. Eventually, I got to a ranking of all of them without ties.

So in the end, I finally have a complete ranking. With an average score in my 100-point scale of 72.59, I think I may have been a bit lenient with my scoring today. Now comes the question of how do I use this?

Is one person’s internal measure of quality particularly useful? If combined into a mass of opinions, it suddenly becomes a useful measure of opinion, but does one person’s opinion really matter. Does anything in positions 51-383 matter at all? Has anyone else gone through this particular exercise? and if so, why?

20 thoughts on “Pointless Debate #5: Subjective Ranking Systems”

  1. Scores through 100 are messy to deal with, in my opinion. Would not attempt something exactly like this myself.

    Anyway, I think the 51st-383rd ranked anime matter as much as 1-50. After all, if you’ve seen that much anime, even good stuff is likely to be pushed out of the top 50. And there may be a reason you have chosen to place something further down on the list, but the show may still be good in general, regardless.

    1. The 100-point scale was more a way to maximize my gut feeling over making direct comparisons. I know others would do it entirely on direct comparisons and I have no problem with that either, but I think I would just confuse myself.

      The 51-383 comment was more an observation about lists in general. Eventually you reach a point where something is merely average or mediocre. I thought it would be at after the top 50, and the bottom 10 is usually so bad it’s like watching a car accident.

      1. I suppose it depends on the approach. I forced myself to spend as little time as possible just gauging how I felt about each one individually. If I were to try to do everything on its own merits compared to everything else, then it would be a grand project for me.

  2. http://aloedream.animeblogger.net/archives/148
    http://aloedream.animeblogger.net/archives/234
    http://aloedream.animeblogger.net/archives/264
    http://aloedream.animeblogger.net/archives/497

    http://blog.melative.com/archives/46

    Just a note, an interface for this relative-rating system was implemented in 2007 (here’s a guide I made for a class back then http://akameta.com/style/instructions.html ), where organizing series was top-to-bottom in a directly comparative manner. Rather than force this kind of rating mechanism which tends to confuse some, I’ve shifted towards what I call an “open” rating system, in which any representation with any values and tiers is allowed by the user. This allows the conception that rating is same/different than other sites, meanwhile computation on what the ratings mean relative user-to-user is strictly based on where an item appears in a rating list AND where every other item in the list appears.

    I think this is what you were getting at.

    1. This might not be related entirely, but I was thinking that it could be possible to take ratings from a base of users and run a large-scale pairwise comparison on each series. With thousands of series though, this might be technologically unfeasible since each one would add a further layer to the calculations

      1. The original equation weight tiers based on number of items and number of tiers, with degrading downwards; items near the bottom have trivial values compared to other items near the bottom, regardless of rating. I abandoned the degrade, because of various exploits, but the idea of infinite margins can help build interesting comparison operators. These things vary, but rating values are only relevant inside one user’s list… that’s important.

        There is also the tendencies, say we take the top 10% of the populations rating, and then compute which series appear (or are likely to appear) in the top 10%. This sort of calculation is independent of the numerical user-rating, 10%’s vary, but it’s empirical… the true algorithm would be able to calculation the X% [tendency] of any given series, based on all rating lists… that’s the idea (currently the calculator is off). And in a real system, the size of percentages is neither a consistent range or value.

      2. What I was thinking was a way to take all lists and say anime A has a better score than anime B on 65 out of 100 lists, so it would rank ahead of that. Then, comparing anime A to anime C, etc. until one is this is all finished and there is 1 on top. Then redoing it for all the other ones until there is a definitive number 2. Incredibly time consuming, and probably impossible to do.

  3. I and my roommate were recently asked why we didn’t have any ratings or rankings of the anime we watched, since between us we’ve seen more anime than we care to admit. Our answer was that reducing it to a couple numbers would take too long, and only add to the mass of numbers you’d already find online.

    Over the years we’ve simply collected notes on what was memorable about an anime, and whether we felt it added any value to our overall anime experience (and why). It makes it easier to match what a person might want to watch given their mood and tastes, and takes a lot less effort. Lazy? You bet.

    But it’s helped us recommend anime to our newer club-members more than a 1-10 scale ever could, and it also removes the 51-383 ranking problem, because it tells you why you might want to (re)watch that show instead of reducing it to another faceless entry. It also makes it easier to ignore the actual samey stuff that we feel deserves to be lost in the crowd.

    It’s also useful in the sense that we can still easily figure out what “we’d rather watch” (since that’s a common note), and if others want to rely on our flimsy opinions for their next candidate show then they can still do so.

    1. I think you’ve pretty much highlighted one of the major issues with ranking in general. There’s a tendency to turn everything into a mere number out of a larger number. Your way would make the middle numbers stand out more depending on how vivid the notes or ability to describe to someone what makes it stand out. Numerical ranking systems largely take out a human element even if they are made by a human.

  4. I just rate everything on my gut feeling, mostly because some series really stand out for me and I’m too lazy to do the rest. Good on you for this, though, that’s pretty epic.

    1. I really don’t see this as really epic, or at least it felt like it should have taken more effort to qualify as such. This was all gut feeling, though.

  5. When I ranked all the anime I’ve seen, I just did it out of curiosity, to see, when it came down to it, which anime I preferred to others when distilling them to the most basic level. I never really use my list to point people in the direction of series they need to see; really, it’s mostly an act of self-indulgence, haha.

    1. I probably did it for most of the same reason, just to see how it turned out. I don’t know how much people would care that I had the second season of Hayate the Combat Butler at 248, and I doubt anything good will come out of having revealed that.

  6. I find this interesting, and while I can’t say that there is a point to it, I am obsessed with it. My entire life, I have listed everything I could, and there is no list that I spend more time micromanaging than my favorite anime list? Why? Because it is a representation of myself. Anime is what I care about most, so the positioning of anime in comparison to one another is like a defining of my personality.

    I spent a LOT of time perfecting a ranking system that could compare every single anime that I’ve ever seen and organize them into a strictly defined list. I’m very proud of it.

    1. This is something else I was interested in. The act of putting things in a ranking also reflects on the person creating said list. I am still in the process of trying to find the utility in what I did, maybe it was something as simple as trying to figure out my own taste with as little bias as possible.

Comments are closed.