Who Wants Standards?

Most of us live in worlds of conversation, like books or blogs or chats, where we tend to give many others the benefit of the doubt that they are mostly talking “in good faith.” We don’t just talk to show off or to support allies and knock rivals – we hold our selves to higher standards. But let me explain why that may often be wishful thinking.

I’ve previously suggested that coalition politics infuses a lot of human behavior. That is, we tend to use all available means to try to help “us “and hurt “them”, even if on average these games hurt us all. Coalition politics is a dirt that regularly accumulates in most any corner that is not vigorously and regularly cleaned.

This view predicts that coalition politics also infuses a lot of how writings (and speeches, etc.) are evaluated. That is, when we evaluate the writings of others, we attend to how such evaluations may help our coalitions and hurt rival coalitions. Especially for writings on subjects that have little direct relevance for how we live our lives. Like most topics in most blogs, magazines, journals, books, speeches, etc.

However, while we may find such cynicism plausible as a theory of rivals, we are reluctant to consciously embrace it as theory of ourselves. We instead want to say that we mostly evaluate the writings of others using different criteria. And when we are part of a group that evaluates writings similarly, we want to say this is because our group shares key evaluation criteria beyond “us good, them bad.”

Now some groups can offer concrete evidence for their claims to be relatively clean of coalition politics. These are groups who declare specific “objective” standards to judge writing. That is, they use standards that are relatively easy for outsiders to check. For example, outsiders can relatively easily check groups who evaluate writings based on word count, or on correctness of spelling and grammar. Yes, a commitment to such standards may favor some groups over others, such as good spellers over bad spellers. But it can’t be adjusted very easily to shifting coalitions. Which makes it a poor tool for supporting coalition politics.

Some groups say they judge writings based on their popularity in some audience. And yes, it can be pretty easy to evaluate the popularity of writings. However, it could easily be the audience that is using coalition politics to decide what is popular. Thus using popularity to evaluate writings doesn’t at all ensure that coalition politics doesn’t dominate evaluations.

Some groups claim to evaluate written “maps” based on how well they match intended “territories”. And when it is easy for many clearly-neutral outsiders to visit a territory, it can be easy for outsiders to check that territory-matching is actually how this group evaluates maps. But the harder it is for outsiders to see territories, or to read their supposedly matching maps, and the more easily that outside critics can be credibly accused of political bias, the more easily a group could pretend to evaluate maps based on territory matches, but actually evaluate them via coalition politics. For example, anthropologists watching the private lives of the very rich might write descriptions of those lives that pander to academic presumptions about the very rich, since few academics ever see those lives directly, and the few who do can be accused of biased by association.

Some groups use objective criteria for evaluations, but don’t give those criteria enough weight to stop coalition politics from dominating evaluations. For example, economic theory journals can claim that they only publish articles containing proofs without obvious errors. And the ability of readers to seek errors may ensure that this criteria is usually satisfied. But such journals may still reject most submissions that meet this criteria, allowing coalition politics to dominate which articles are accepted. Winning coalitions may be constrained to include only members capable of constructing proofs without obvious errors, but this need not be very constraining to them.

Another approach is to only use objective evaluation criteria, but to use many such criteria and to be unclear about their relative weights. The more such criteria, the greater the chance of finding criteria to reach whatever evaluation one wants. For example, in many legal areas there is wide agreement on the relevant factors, and on which directions each factor points to in a final decision. Nevertheless, given enough relevant factors, courts may usually have enough discretion to favor either side.

For any one group and their declared criteria of evaluation, it can be hard for outsiders to judge just how much leeway that group has left for coalition politics to influence evaluations. We tend to give the benefit of the doubt to our own groups, but not to rivalrous groups. For example, pro-science anti-religion folks may presume that peer review in scientific journals is mainly used to enforce good evidence norms, but that religious leaders mainly use their discretion in interpreting scriptures to favor their allies.

If they were honest, each group would either declare objective evaluation criteria that leave little room for coalition politics, or accept that outsiders can reasonably presume that coalition politics probably dominates their evaluations. And everyone should expect that even if their group now seems an exception where other criteria dominate, it will probably not remain so for long. Because these are in fact reasonable assumptions in a world where collation politics is a dirt that regularly and rapidly accumulates in any corner not vigorously and regularly cleaned.

Hey there reader, I really am talking about you and the worlds of writing where you live. Do you presume that your worlds are mostly dominated by politics, where different coalitions vie to support allies and knock rivals? Or do you see the groups you hang with as holding themselves to higher standards? If higher standards, are they standards that outsiders can easily check on? Or do you in practice mostly have to trust a small group of insiders to judge if standards are met? And if you have to trust insiders, how sure can you be their choices aren’t mostly driven by coalition politics?

Years ago I struggled with this issue, and wondered what evaluation criteria a group could adopt to robustly induce their writings to roughly track truth on a wide range of topics, and resist the corrupting pressures of coalition politics to say what key audiences want or expect to hear. I was delighted to find that for a wide range of topics open prediction markets offer such robust criteria. Each trade can be an “edit” of the highly-evaluated “writing” that is the current market odds on each topic. Such edits are rewarded or punished via cash for moving the consensus toward or away from the truth.

I had hoped that many groups would be anxious to avoid the appearance that coalition politics may dirty their evaluations, and thus be eager to adopt new standards that can avoid such an appearance. So I hoped that many groups would want to adopt prediction markets, once they were clearly shown to be feasible and practical. Alas, that seems to not be so.

Today’s winning coalitions seem to prefer to let coalition politics continue to determine who wins in each group. This seems like how police departments would like to appear free from corruption, but not enough to actually make their internal affairs departments report to someone other than the chief of police. We are fond of tarring rival groups with the accusation that coalition politics dominates their evaluations, and we are fond of pretending that we are different. But not enough to visibly block that politics.