Ensuring quality vs. ensuring visibility (or success)

In the CAS group Jochen Fromm, the group founder, raised the issue of how to ensure quality in Web 2.0 systems. I replied as follows.

Your question about how to increase the quality of user-generated content is a very good one. I was one of the original doubters that Wikipedia would be at all useful. I'm very pleased with what it has become. I'm a bit less enthusiastic about it than the general buzz, though. My experience is that many of the technical articles are not useful unless you already understand the subject. I tend to use it for two things: (a) to get a general sense of an area that I'm not sure about and (b) as a user-aided search site. (I never use del.icio.us, but I often use Wikipedia as a search aid.)

In both cases, I feel reasonably confident that the people who work on a page are committed enough to the subject matter to ensure (a) that its not way off the mark and (b) that it has a reasonably good list of current references. Of course that's not guaranteed, but it's generally the case.

This seems to be self-enforcing in that anyone who cares enough about some subject area to work on a Wikipedia page will want the page to be reasonably accurate and up-to-date. This doesn't guarantee that there won't be vandalism. The Wikipedia structure seems to handle that reasonably well. It also doesn't guarantee that the person creating the page is knowledgeable enough to do a good job. But at least it comes close to guaranteeing good intentions, which is a lot. (This is probably the case about most web sites.)

Of course there are always cases in which people want pages to make something look good as in the examples of corporations and politicians spinning Wikipedia pages. But that's a somewhat different kind of quality question. Commercial ads, for example, tend to be high quality in terms of production value as well as a certain kind of information content--even if they can also be misleading in that they may leave out important information. So that's like listening to an advocate rather than an objective observer. We have to learn how to deal with that.

You mentioned mechanisms to ensure quality. Many of them do more to identify quality than to ensure it. Reputation systems, for example, point to people who have produced quality. But they don't ensure that others won't produce low quality work. That seems to be what Google and BBs that rate posters rely on. The people with better reputations become more visible. So this is a mechanism for finding quality, which may be enough. In some sense that's what market mechanisms are supposed to do. Better products become successful and more visible. Of course there are lots of ways of gaming that as we all know. But I think that's the basic idea: build mechanisms so that quality will become more visible.

As you said, this is really quite different from Wikipedia, which wants to ensure that all pages have a certain level of quality and not just that the good ones are more visible. That seems like a much harder job.

What now strikes me as interesting is the difference between mechanisms (like evolution and market mechanisms) that encourage the establishment and visibility of successful variants and mechanisms that attempt to ensure an overall level of quality. The latter seems much harder. Evolution works because elements that do well in an environment are (by definition) more successful at establishing themselves in that environment.

An implication of this is that recommendation systems (and Google) work only because the recommenders actually favor quality. If recommenders favored cranks, that's what Google would serve us. So recommendation-based systems only reflect the tastes of the recommenders. I guess that's obvious. But it's worth noting how dependent we have all become on the fact that apparently the Internet favors quality overall. One can also point to this as a Wisdom of Crowds effect.

As I said above, recommender systems are very different from systems that attempt to ensure overall quality. The latter seems to be much more difficult. It strikes me that the latter in some ways reflects the dreams of centrally controlled economies. The fact that they don't work shows how difficult it is to ensure quality overall. Economies that are more market driven can allow poor quality because better quality products and services (in general) become more successful.