I am a big fan of marketing metrics. And even though the validity of its scores are somewhat suspect, I am still addicted to checking my score on Klout. Maybe it is my love of trend data that brings me back every day or just the orange colors. Either way, I have to admit that I get excited when my number goes up and upset when it goes down, even if I don't know exactly what causes either outcome.
For the most part, the Klout score is more entertainment than useful. Not knowing how it is derived, it is impossible to tell if it really has much value. The validity question increases when you see some of the topics about which Klout believes you are influential. They can range from absurd to hilariously absurd. For instance, right now Klout believes I am influential about "Big Mac," "Forums," and "Relationships." Where this came from, I have no idea.
For Klout, a perfect world would mean its scores would represent THE metric for online influence. Marketing companies would use their score to woo new business and to target online outlets for their products. In a perfect world, it would be the social media world's version of a credit score. This is not a perfect world and Klout is most definitely an imperfect tool.
In the last couple of weeks, Klout started rolling out a new piece to its score puzzle: member voting. Now, each day you can vote on who is most influential to you and who you believe should be more influential. It is an interesting feature clearly designed to increase site engagement and daily visits. However, the end result is creating more questions about the overall validity and credibility of the Klout scoring system.
Before I detail my concerns with their method of data collection, let me preface it by saying I have no clue how they are using this data other than the small blurb where it says it will be incorporated into the Klout scores. Maybe it only affects which Klout members are featured on your Klout pages. I don't know. However, assuming that this data will be incorporated into Klout scores, I have my concerns.
When you first open the voting panel, you are asked to vote on ten comparisons. As you can see in the picture above, you can either vote for one of the two people or vote for "I don't know these influencers." I have a few of problems with this setup.
How influential are they?
First, by asking people to pick one or the other, we have no idea about the actual relationship between the people involved. Is this person more influential because the other person is not influential at all? Are they both big influences, but one is only slightly more influential than the other? The format of the question is such that the magnitude of the difference is not captured, thus the usefulness of the information is limited. I understand that Klout is most likely using an algorithm that accounts for this and by regularly including a person in votes, over time, they can compensate for this and more precisely estimate the person's influence over you. However, the actual magnitude of the comparison will never be known. So even if I know that person A is picked as more influential 60 percent of the time and person B is selected 50 percent of the time, I still don't know any real actionable information about either person.
What does "more influential" mean?
Even were we to accept that the algorithm being used by Klout accounts for this lack of magnitude and can correct estimates over time, there is still an issue with interpretation of the question. Klout asks, "Who is more influential to you?" Pretty straightforward, or is it? There are several ways to interpret this question.
Take for example the two accounts listed in the photo above. One is a marketing research organization and the other is a friend of mine. Now, I am more aware of Seth's posts. And, he more frequently posts something I find interesting. However, the value of the posts by LiveInsights is greater since it applies directly to my career. Here, we have an issue of quantity versus quality. On general topics, I would say Seth influences me more. But on certain topics, LiveInsights is much more influential. So in that case, how do I decide? Klout has left the interpretation of the meaning of "more influential" up to the survey taker. If we assume that the respondent consistently uses the same definition throughout the poll, we still have no way of knowing if his or her definition was the same definition everyone else used (here's a hint, it probably wasn't). It limits the validity and reliability of our questions and hurts our ability to compare results and draw conclusions. In short, it adds error to all our results.
How do I answer neither?
Still, there is another problem with this particular question. What do I do if I believe that both people are equally influential? The only option for not voting is to click "I don't know these influencers." I assume that would be akin to saying, "I don't know these people, and they are not at all influential to me." But that is not what I am trying to say. I am trying to say both of these people are equal. But, as I said before, I have no way to show equality or an incremental difference between the two comparisons. I have no way to indicate that while one is influential, the other is also still influential. There is something to be said for forcing a choice and not allowing for a neutral option. However, in my experience, neutral choices increase accuracy and reduce dropouts. The goal is to capture as much variation as possible. This simply captures artificial or incomplete preferences.
The second part of the Klout poll is ten questions about who should have a higher Klout score. Again, this question suffers from many of the same problems. There is no way to indicate the magnitude of the difference between the two people. There is no option to select that indicates they should be equal. And the definition of the question is frustratingly vague. In fact, this one is probably even more open to interpretation. Let's take a closer look.
"Who should have a higher Klout score?"
Again, this question seems extremely straightforward, especially if you work for Klout. However, each person is going to answer this question with different criteria. Klout scores take into account a lot of factors: number of people in your network, influence of people in your network, number of times your posts are commented on, retweeted, reposted, liked, etc. When I look at this question, I have no way of knowing that information. I am more inclined to answer from a totally different perspective. For instance, if I see Justin Bieber vs. Senator Harry Reid, I immediately think that a U.S. Senator should have a higher score than a pop singer. Of course, Klout score isn't based on importance in society. Rather, it is based on activity in social media. In that case, maybe Bieber is more active with more followers. Am I supposed to be making an educated guess about their social media presences or is this my chance to provide personal commentary on the values of society?
We have a huge problem with the way people are going to interpret the question. When you don't define the terms you use or attempt to ensure everyone interprets your question the same way, the value of your results drop precipitously.
Let's clear things up
So how do we correct this? First, I would not be asking people who they think is more influential. This is a vague, abstract concept that is difficult for the average person to evaluate with much certainty. Instead, reduce the question down to factors that indicate influence. For instance, "In the last month, whose posts did you read more often?", "In the last month, whose posts did you retweet more often?" By anchoring this in time and tying it to a more tangible concept, people can provide a more accurate assessment. This question serves as a proxy measurement of overall influence and reach, both being things that people cannot accurately evaluate.
Let's frame the question
Similarly, the second question really needs to be defined better. Something simple like, "Based on their social media interaction," can frame the question in a way that clarifies the criteria for the question. Now when I read the question, I know that I am supposed to be evaluating these people on a defined set of criteria and not an abstract perception of the importance of that person.
Let's talk about topics
As we mentioned before, someone who is influential about one thing may not be influential about another thing. Also, we pointed out that Klout is not really very good at figuring out the topics about which we are knowledgeable. If topics are important in Klout, why aren't we being asked about people based on those categories? Wouldn't it make more sense to ask, "Who is more influential about politics: Harry Reid or Justin Bieber?" compared to "who is more influential about music: Harry Reid or Justin Bieber?" Here, we have narrowed down the meaning of influential by restricting it to a specific topic. Obviously, this particular example would create extreme results. But, say you have have two friends, both of whom have "politics" attached to their Klout profile. Now, if Klout asks which one is more influential about politics, we are giving useful data for refining Klout profiles. Klout has plenty of access to actual behavioral data behind social media activity. That information is always going to be more reliable than self-report information. Where they lack is the more nuanced qualitative data. This is where these surveys should be used to fill in the gaps.
I applaud Klout for trying to find ways to improve its scoring system. And, self-report surveys of its users is a good way to individualize Klout scores and develop more relevant information for users. However, the current execution of this lacks refinement and creates more error and less reliability--something an already discredited rating system doesn't need. It will be interesting to see how Klout scores will change in the future because of these surveys. However, unless this survey methodology improves, I only see Klout becoming less relevant, not more.