Online Community Moderator Metrics
When it comes to managing a team of moderators at a high level, there is an inescapable human element to it. Moderators aren’t people you select and let loose without any guidance or support. Communication should be a constant.
I want to know what’s going on. If they handled a delicate situation well, I want to praise them. If a member is unhappy with a moderator, I want to take that burden off of them. If a mistake is made, I work to correct it.
Being truly in tune with your staff means that you are constantly talking. You can’t cheat that with data.
However, I was recently asked to think about moderator metrics. Specifically, metrics designed to get a picture of moderator activity and effectiveness for the purposes of identifying moderators who may have gone inactive. If I was building a dashboard for that reason, what would I want to see on it?
Activity in the Community
I like for my moderators to be present in the community, as contributors. They set a wonderful example for the community to follow – in how they treat others – and it allows members to get to know them, to see them as real people, rather than just an authority figure.
Simple things work here. How often are they visiting? How often are they contributing? Post count is, in general, a terrible metric for gauging quality, so the idea here isn’t to see a high post count, but the existence of one, so that you know they are present.
Time Spent
Moderators need to read. They are evaluating content not only as a contributor, but more importantly, as a moderator. It takes time to do that. How much time is the moderator spending on the community?
Responsiveness to Private Messages
If you have a “contact the moderator” option, you can measure how quickly those messages are responded to and by whom. Who is responding? Who isn’t? If the messages go to individual moderators, you can see exactly how responsive those moderators are, on average.
Note that a “response” could mean that the moderator responds directly to the member or it could mean that they escalate the issue to you or whoever is above them in the chain. A moderator might not be capable of answering every question they receive, but they can direct the message to the proper party.
If you have a general private message box where a moderator might receive messages of a personal nature and a moderation-related nature, you probably can’t draw meaningful data from that.
Responsiveness to Post Reports
You could do a similar thing with your post reporting system. When members file reports for questionable content, who is acknowledging those reports?
Some people might just be faster or there at the right time. This isn’t a game of who can clear the most reports – you want them to correctly handle the report. However, if you have one moderator that has cleared 214 reports this month and another that has cleared 2, there might be an issue.
Are They Using the Moderator Tools?
Moderators are given tools, and those tools will need to be used. It doesn’t matter how wonderful your community is, there will be things posted in the wrong section, there will be spam, and there will be issues that require some attention.
Again, this isn’t a quantity game. I don’t care who has moved the most conversations. I only care that all moderators are making use of the tools and feel confident in doing so.
Mistakes Made is Not a Good Metric
I thought about the idea of keeping track of how often a moderator made a decision that had to, in some way, be corrected. But the more I considered it, the more I didn’t like it. Mistakes aren’t that big of a deal. They represent teachable moments. When it becomes an issue is when you fail to recognize mistakes and make excuses for them. Persistent, sloppy mistakes are a big deal. But in that case, a dashboard can’t really help you.
It’s very misleading and would need to be counted manually. Your dashboard would need to be told a mistake was made. Of course, not all “mistakes” are equal.
If someone is making bad mistakes, that is something that the leader of the team should be aware of because they would be the one correcting those mistakes. They should be documented, but I don’t like the idea of my moderators knowing that somewhere, in a database, is an overall tally of all mistakes they have ever made. There isn’t really value in that number. Not for me.
You Don’t Want Your Moderators Thinking About These Metrics
I’m not saying that, if I used these metrics, I would keep them a secret. If asked, I wouldn’t act like they didn’t exist. But I also wouldn’t advertise them. It is my responsibility to empower, support and guide my moderators, helping them to be as effective as they can be. I do not want them thinking there is a leaderboard that they must win at. That would be a disaster and totally goes against the idea of good moderation.
The data, in this sense, would be a tool for me (or the team leader) to be more effective. It is not something for moderators to worry about. If the data unearthed a concern for me, when I contacted them, I wouldn’t reference the dashboard or any hard data. I would simply address the concern and see how I could help.
Applying the Metrics
What I am really looking for here is to ensure that people are active, that they are responding to members and that they are using the tools. This would come in handy if someone has a really, really large community with a lot of different sections and many moderators. It’s easy to see who is active, but harder to identify who has not been around (without manually checking each person). You can then follow up with them. That is a gap that this data will bridge.
Another area where it will help is in identifying people who may not be comfortable yet and who might need some extra training or a boost. For example, a newer moderator who is contributing a lot, who spends a lot of time on the site, but isn’t really touching the tools. If things are being taken care of by yourself or other moderators, this can sometimes get lost in the shuffle. I have definitely had some late bloomers over the years – people who just took more time to get comfortable than others.
Similarly, the data could point out areas where some moderators may need a nudge. An active moderator who never looks at post reports, for example.
These metrics would be useful for both paid and volunteer moderators. Obviously, with volunteers, you can’t treat them like employees, give them shifts, etc. But there is still a general, minimum expectation of activity.
A balanced approach is best. Moderators come in different shapes and sizes and, naturally, some will do more (or appear to do more) than others. You don’t want them all to be the same. You don’t want to encourage them all to remove the same number of posts. There is no quota to fill. One moderator might post a lot, be quick to deal with reports, but not use the tools as much as others. Another might post less, read more and use the tools a lot. It takes all kinds.
As long as they are all working toward a common goal, that is what you want.