The virtues of moderation, part II

edited by SciSal

[This is an excerpt from James Grimmelmann’s paper “The virtues of moderation” published in the Yale Journal of Law and Technology (2015). The relevance to our webzine should be obvious. The excerpt presented below was pieced together by SciSal editor Dan Tippens and is reproduced here with permission from the author. Part I was published a couple of days earlier.]

Now that we have seen the problems that moderation faces, we can discuss how it solves them. We have already met the basic definitions: a community of members who use shared infrastructure to exchange content. The members have roles: as owners of infrastructure, as authors and readers of content, and as moderators. These are the nouns in the grammar of moderation.

Techniques (Verbs)

The real study of moderation begins with the verbs of moderation — the basic actions that moderators can take to affect the dynamics of a community. There are four: excluding, pricing, organizing, and norm-setting [in this post, we will only focus on excluding, organizing, and norm setting].

Excluding

Exclusion is fundamental in property theory because of its simplicity. Rather than attempt to calibrate specific uses, one simply excludes outsiders from all uses. In an online community, exclusion deprives the community of the contributions that those who are excluded could have made. But that loss can be justified when exclusion inhibits strategic behavior. It can be used against any form of strategic behavior by targeting those users who engage in that behavior — for example, to reduce congestion by excluding known spammers. The processes used to decide who will be excluded can fall anywhere along the spectrum from highly precise to absurdly crude. Mark Lemley individually vets each subscriber to the CyberProfs mailing list; for a time, Facebook was available only to users with a .edu email address. At any level of precision, a particularly important decision is whether the default is inclusion or exclusion. A default of inclusion gives everyone, well intentioned or not, at least one bite at the apple. Exclusion can also be applied independently to different roles. It is common, for example, to let anyone read a discussion board but to allow only registered users to post.

Organizing

Organization shapes the flow of content from authors to readers. It is the verb of moderation that most takes advantage of the informational capabilities of computers. Categorizing messages on a bulletin board by topic is organization. So is searching them by keyword, counting the number of messages, or deleting off-topic messages. These are all ways of remixing authors’ contributions to give readers a more satisfying experience. It is helpful to think of organizing techniques as being built up from several basic operations:

• Deletion is the removal of content. A bulletin board administrator who excises off-topic and profanity-laden posts is engaged in deletion.

• Editing is the alteration of content. It ranges from correcting typos to changing the very essence of a post. At the limit, editing is deletion plus authorship: the moderator rejects an author’s reality and substitutes her own.

• Annotation is the addition of information about content. eBay’s feedback system annotates buyers and sellers; Facebook’s Likes annotate posts and comments; Amazon’s user-written reviews and lists are annotations that have crossed the line and become content in their own right.

• Synthesis is the transformative combination of pieces of content. Wikipedia is the ultimate example of synthesis. There, small and heterogeneous changes by individual users are synthesized into entire encyclopedia entries. On a smaller scale, an online poll synthesizes individual votes into totals.

• Filtering is deletion’s non-destructive cousin: the content is still there, but readers see a specialized subset of it. A search engine filters; so does a blog’s list of the ten most recent comments. At the limit, filtering asymptotically approaches deletion: the ten-thousandth search result might as well not exist.

• Formatting is the styling of content for presentation to readers. Good typography improves readability; sensible ordering and grouping of images makes it possible to scan through them quickly. Like the other verbs, organization is itself costly but can reduce strategic behavior.

Organization directly attacks cacophony by helping readers see only the content they want. At the same time, organization indirectly reduces cacophony by reducing the incentives for authors to create low-value content that readers don’t want and will never see. Only deletion directly attacks congestion, but all forms of organization have the same indirect effect of reducing the incentive to spam. On the other hand, organization can be a tool for manipulation in the hands of self-interested moderators. Think, for example, of a Judean People’s Front sympathizer deleting every mention of the People’s Front of Judea on a Roman forum [2]. Finally, depending on how it is used, organization can either greatly amplify or greatly inhibit abuse: compare a gossip site that deletes crude sexual comments with one that invites them.

Norm-Setting

Moderation’s biggest challenge and most important mission is to create strong shared norms among participants. Norms can target every form of strategic behavior. For example, if every author refrains from personal attacks, there is no further personal-attack problem to be solved. Beneficial norms, however, cannot simply be set by fiat. By definition, they are an emergent property of social interactions. Moderators have limited power over group norms. Most of the levers they can pull will only nudge norms in one direction or another, possibly unpredictably. Good norm-setting is a classic example of knowhow. There are heuristics, but knowing whether to chastise an uncivil user publicly or privately is not a decision that can be made in the abstract. Blogger Jason Kottke summed up the challenges of norm-setting with characteristic verve:

“Punishing the offenders and erasing the graffiti is the easy part . . . [F]ostering ‘a culture that encourages both personal expression and constructive conversation’ is much more difficult. Really fucking hard, in fact . . . it requires near constant vigilance. If I opened up comments on everything on kottke.org, I could easily employ someone for 8-10 hours per week to keep things clean, facilitate constructive conversation, coaxing troublemakers into becoming productive members of the community, etc. Both MetaFilter and Flickr have dedicated staff to perform such duties . . . I imagine other community sites do as well. If you’ve been ignoring all of the uncivility on your site for the past 2 years, it’s going to be difficult to clean it up. The social patterns of your community’s participants, once set down, are difficult to modify in a significant way.”

Some communities depend on shared norms. Discussion groups, for example, are acutely sensitive to group norms. It only takes a few determined spammers or trolls to bring a discussion to a screeching halt. But other communities can prosper even when some norms are widely flouted. Spammers and trolls still abound on the Internet, but they have not yet managed to ruin it for everyone. Google may not be able to make spammers clean up their act, but it can hide their antics. The difference illustrates the two roles that the other verbs of moderation can play. Sometimes, they keep order directly, in the face of bad behavior; at other times, they keep order indirectly, by encouraging good behavior. That is, the other three verbs are both substitutes for and sources of norms, and communities vary in the balance they strike between these two roles.

Moderators can influence norms directly by articulating them. They can do this either in general, with codes of conduct and other broad statements of rules, or in specific cases by praising good behavior and criticizing bad. The difference is the difference between “Don’t post images containing nudity” and “This post has been deleted because it contained nudity.” Note, however, that stating a norm does not automatically promote it. There is empirical evidence that, in some circumstances, expressing a norm about user behavior can induce exactly the opposite response. Moderators can also influence norms indirectly, through the other verbs. A list of “new and noteworthy posts” doesn’t just help users find good posts through organization, it also educates them in what makes a post good in the first place. Put another way, moderators can use the other three verbs not just to regulate but also to nudge.

The flip side of this point, though, is that any time a moderator uses one of the other verbs, she nudges participants’ norms, whether she intends to or not. For example, excluding a well-known commenter can reduce participants’ sense of trust in a moderator, even if the exclusion is justified. Experienced moderators evaluate every design decision in terms of its effects on community norms. A few particularly important ways to promote good norms reflect the accumulated wisdom of community managers.

By far the most significant is fostering a sense of shared identity that reinforces participants’ sense of belonging and their commitment to the good of the community. Another is the initiation of new participants, who must be taught the community’s expectations at the same time as they are made to feel welcome. Highlighting good behavior and hiding bad behavior reinforce participants’ sense that good behavior is prevalent while also teaching them what to do. As a result, designers frequently worry about how to balance competitive and cooperative impulses. Competition can spur users to individual effort at the cost of social cohesion, and different communities strike the balance differently.

Distinctions (Adverbs)

Transparently / Secretly 

Every moderation decision has some observable consequences, but some are more observable than others. Transparent moderation makes explicit and public what the moderators have done and why, revealing what the overall moderation policies are and how they apply in each specific case. Secret moderation hides the details. This distinction is really a spectrum: moderation could be transparent about the what but not the why, or transparent only some of the time. Generally speaking, transparency takes additional work to implement, just as having judges give reasoned explanations of their decisions increases the judicial workload. It is easier to be secretive about some kinds of moderation than others. Someone who is excluded from a community will generally be able to tell that they are being denied access, although it is sometimes possible to disguise the fact that it is deliberate.

Organization has the most room for secrecy. Search users don’t know what pages Google hides from them; Facebook users may not realize that the News Feed is only a partial list of posts from friends. Conversely, secret norms are close to an oxymoron: norms must be known to be effective. The choice between transparency and secrecy in exclusion, pricing, and organization can have indirect effects on norms. On the one hand, transparency enhances legitimacy, providing community support for moderation; on the other hand, secrecy raises fears of censorship and oppression.

Centrally / Distributedly 

Moderation decisions can be made either centrally by a single moderator whose decision affects the entire community, or by multiple distributed moderators whose individual decisions affect only part of the community. For the most part, the consequences are as one would expect, and track the usual legal debates about hierarchy and federalism. Centralized moderation provides consistency. Distributed moderation promotes diversity.

Centralized moderation offers the ability to stop unwanted content and participants by creating a single checkpoint through which all must pass: a spammer kicked off of Facebook will not bother anyone else on Facebook. But chokepoints are also single points of failure: a spammer who gets through on Facebook can bother a lot of people. In comparison, distributed moderation offers more robustness and defense in depth.

Centralized moderation offers a clear focal point for policy-making. If you don’t like my post, you know where to complain. Distributed moderation permits those with ideological differences to agree to disagree: if you don’t want to read my weblog, no one is putting it in front of you. In a sense, the choice between centralized and distributed exclusion is the choice between a single community and many. But norms, by their nature, cannot be fully centralized. The power to adopt, shape, or reject them is always in the hands of members. The larger a community, the more competing voices and normative focal points it is likely to have.

Identity

The final community characteristic is the distinction between identity and anonymity. At one extreme, participants in an online community could be completely identified, bringing with them a complete biography of their online and offline lives. At the other, they could be completely anonymous. Compare Google+, which launched with a strict, stringently enforced, and much-criticized “real names” policy, with 4chan, where “most posts . . . are disconnected from any identity.”

There are many gradations in between. Participants could have identities that mostly match their offline lives, but in which the details are potentially questionable, as in an online dating service where participants sometimes lie about their height. They could have rich and persistent but avowedly fictitious identities, as in virtual worlds where they play the same avatar thirty hours a week for years. They could have stable but thin identities, as on a discussion board that uses pseudonyms and keeps users’ real names and email addresses secret. They could have thin identities purely as a matter of convention, as in some blogs’ comment sections, where a commenter can pick a fresh display name with each comment.

Participants could even have one level of identifiability at the infrastructure level (supply a valid email address to sign up) but a different level at the content layer (that email address is hidden from other participants). Whatever its nature, the most important role of identity is creating stable reputations through time so that others can link past behavior to a present identity. All four verbs of moderation can tap into identity. Exclusion absolutely depends on it; without identity, the distinction between “outsiders” and “insiders” collapses. You can identify the unwanted outsiders and blacklist them or identify the wanted insiders and whitelist them, but both versions require some notion of identity. As anyone who has moderated a blog’s comments can testify, this is a huge problem for communities that are open to new and unknown members from the Internet at large. There is often no way to tell that a “new” commenter is actually an old and well-known miscreant, back from a ban for another round of malice.

It is well known that identifiability plays a significant role in setting social norms. Persistent reputations make it possible for participants to build credibility as respected elders within the community. They make it possible to hold participants accountable for their actions, enabling the effective monitoring and graduated sanctions beloved by commons scholars. By contrast, anonymity enables consequence-free norm violation and can undermine the appearance of reciprocity among real human beings. But stronger identity is not always better. Sometimes it creates a badge for misbehavior: a leaderboard is an invitation to fame-seeking cheaters.

Making participants more anonymous (for example, by resetting a server) can drive trolls away because it deprives them of the opportunity to make a (bad) name for themselves. Paradoxically, both identity and its opposite — anonymity — can be expensive to establish. Externally produced identity requires participants to prove facts about themselves, which can cost both time and money. It also requires owners and moderators to be prepared to check these assertions, which too is costly. Internally produced reputation systems require participants to take the time to learn about, comment on, and rate each other.

An important question for online communities is who controls these socially constructed identities: users themselves, the community, or infrastructure owners. Anonymity might seem cheaper, but genuinely effacing participants’ identities requires some significant effort — deleting log files, stripping out snooping software, and taking action against participants who “out” one another’s offline identities.

Finally, identity can be the enemy of privacy, for good and for bad. Divulging information about oneself is itself a cost. Privacy is virtually a precondition for some kinds of speech. Some conversations simply cannot take place in public. This phenomenon can be good: think of therapeutic conversations on a discussion board for adult victims of childhood abuse. It can also be bad: think of virulently misogynistic and racist conversations on a law student board. Two forms of abuse are characteristically tied to the misuse of identity. Impersonation — the hijacking of another’s identity — requires that participants have recognizable identities to hijack. And sock puppetry — creating fake personas to create the false appearance of support for a position — requires that the community recognize personas as distinct participants in the first place. Both become possible when a community accepts claims of identity that it is not capable of properly validating.

_____

[2] Monty Python: What have the romans ever done for us.