Collective Quality. How to design collective standards of knowledge?

Submitted draft. Do not quote without permission.

La barre de platine-iridium utilisée comme prototype du mètre de 1889 à 1960

Knowledge is a common good. A tiny part of our knowledge of the world is generated by our own personal experience. Relying on others is one of the most fundamental ways to acquire knowledge, not only about the external world, but also about who we are, (for instance about when and where we were born). To use Mary Douglas striking metaphor: “Our colonisation of each others’ minds is the price we pay for thought”.[1]

The collective dimension of knowledge is acknowledged in almost every field of thought these days, from the optimistic forecasts on the power of collective intelligence made by James Surowiecki[2], to the debate on the social dimension of knowledge within recent sociology of science and social epistemology[3]. Everybody seems to accept the blatant truth that without the import of other people’s beliefs our cognitive life wouldn’t be much different than that of animals. Yet, what is surprising in this debate is that the collective dimension of knowledge has been put forward to argue in favor of very different conceptions of the objectivity and the standards of quality of knowledge. On the one hand, within the so-called Big-Science debate, the collective dimension of scientific work is considered the ingredient that guarantees the objectivity of that form of high-quality beliefs we name science. On the other hand, the same social dimension has been used to argue against the high-quality standards of scientific method, for a more realistic view of common knowledge[4] empowered by the wisdom of the many that can overthrown the authority of the experts.

Generations of scientists have been raised in the dogma of the impersonality and collectivity of the scientific work, against a classical view inherited from the Scientific Revolution of the scientist as an isolated genius. To mention one of the most influential defenses of the collective view of science, in his famous essay on Little Science, Big Science, which laid the foundations of the contemporary scientometrics, Derek de Solla Price writes that the social nature of collaborative work in the Big Science is the only guarantee of objectivity: scientists do not base their results on their personal qualities, like artists do: scientists are interchangeable because what they do is to apply a collectively shared method of investigation of nature that has nothing to do with their own personal identity. As the zoologist J. R. Baker put it: “If Mozart had not composed that immortal work of genius, the ouverture to Le nozze di Figaro, no one else would have done so; but if Kekulé had not lived, structural formulae and the benzene ring would not have remained forever hidden: someone else would eventually have dreamed the same dreams”.[5]

Thus, according to this view, science is objective because is collective, it is a collective game of peers who scrutinize each other impersonally by applying a shared scientific method that is the Norm of Quality of our knowledge.

But, as I said, this view contrasts with a more recent view of the collective construction of knowledge, in which the standards and norms of scientific method are replaced by the rules of aggregation of lay judgments[6].

Both approaches insist on the equation: collective = objective: to achieve an objective result, that is not too biased by personal points of views, we must be many, no matter if laymen or experts. Knowledge is objective insofar as it is impersonal, disembodied, unvarnished from any singularity and subjective wisdom.

Take for example what Clay Shirky says in his last book on the power of social networks: “We are so natively good at group effort that we often factor groups out of our thinking about the world. Many jobs that we regard as the province of a single mind actually require a crowd. Michelangelo had assistants paint part of the Sistine chapel ceiling. Thomas Edison, who had over a thousand patents in his name, managed a staff of two dozens. Even writing a book, a famously solitary pursuit, involves the work of editors, publishers, and designers. Even if we exclude groups that are just labels for shared characteristics (tall people, redheads), almost everyone belongs to multiple groups based on family, friends, work, religious affiliation, on and on. The centrality of group effort to human life means that anything that changes the way groups function will have profound ramifications for everything”[7].

Thus collectivity is everything today, and knowledge seems to be a product of collective effort. Yes, but if it is so, then where do the standards of our knowledge come from? When a group is able to work out a right answer or an accurate prediction, on the basis of what do we judge that the answer or the prediction is the right one? Either we knew already that it was the right one, or it is just a posteriori verification that can guarantee the truth and the objectivity of the conclusion. In the case of science, even if it is now a truism to acknowledge the collective aspect of the scientific enterprise, the objectivity of the results doesn’t come from the collective dimension, but from the reliability of the method. A hypothetic-deductive method for inferring the theorems from the axioms of a theory, an experimental, statistical method, are the fruit of a long filtering of ideas, collective or singular, that have distilled through centuries the “right” way measure reality and make predictive models of its future possible states. Science is collective because our trust in scientific method is shared almost universally: that is why the same experiment can be replicated at the antipodes of the world and the results compared. But method is not intrinsically collective.

When we come to the more debatable case of knowledge out of aggregation rules of lay judgments, the question of objectivity becomes even harder. How do we judge reliable and true a result that comes from a collective aggregation of individual opinions? How do we know that our Google search for a certain keyword will end up displaying the “best” information available on that keyword? We know it out of personal experience: after many trials with Google searches, we have come out with the conclusion that the information Google is able to provide at the top of its results for a certain search is good enough to be believed. But we do not have independent means of granting this knowledge on the fact that it has been collectively produced.

In his provocative article on the end of scientific method, Chris Anderson simply states that we can live in a groundless world of good matches of statistic data without caring too much about method: “Google's founding philosophy is that we don't know why this page is better than that one: If the statistics of incoming links say it is, that's good enough. No semantic or causal analysis is required. That's why Google can translate languages without actually "knowing" them (given equal corpus data, Google can translate Klingon into Farsi as easily as it can translate French into German). And why it can match ads to content without any knowledge or assumptions about the ads or the content. Speaking at the O'Reilly Emerging Technology Conference this past March, Peter Norvig, Google's research director, offered an update to George Box's maxim: "All models are wrong, and increasingly you can succeed without them."”[8]

So, collectivity in this second sense, of simple aggregation of data or lay opinions is replacing the collective enterprise of science, based on the centrality and robustness of method. But the problem remains: where do the collective standards of quality come from? When I check the grammaticality of an expression by inserting it into Google, I trust the answer that has the largest number of results: for example, I have checked the English spelling for the word “acknowledgment” while I was hesitating between two spellings: acknowledgment and aknowledgment: given that the first for gave me 11 300 000 results while the second one only 34 300, I have opted for the first one. Of course, I was right this time, but why? Is it just a matter of “epistemic luck” or do I have any ground for believing this result? The only ground the people have is obviously previous experience: you have used Google many times, you know that it is reliable as a spelling checker because you have independent ways of controlling its reliability, like the spelling checker or your own word processor, or other written authoritative sources (like a dictionary). But is it enough to ground our knowledge? And when your independent control of the results you obtained on Google should stop? Is the “good enough” epistemic strategy good enough?

In the rest of this chapter, I would like to argue that in a collective world of knowledge the problem of the standards of quality remains and is even harder than within the classical image of science. What is the “right” quality standard for an item? What is quality, and how to filter a common standard of quality if we aggregate in a decentralized way the opinions, tastes and biases of very different people? That is a classical philosophical question that concerned philosophers such as David Hume, who writes in his famous essay Of the Standard of Taste:

“The great variety of Taste, as well as of opinion, which prevails in the world, is too obvious not to have fallen under every one’s observation. Men of the most confined knowledge are able to remark a difference of taste in the narrow circle of their acquaintance, even where the persons have been educated under the same government, and have early imbibed the same prejudices. But those, who can enlarge their view to contemplate distant nations and remote ages, are still more surprised at the great inconsistence and contrariety. We are apt to call barbarous whatever departs widely from our own taste and apprehension: But soon find the epithet of reproach retorted on us”.[9]

Standards of quality thus change, and each human being can cultivate his or her own idea of what is good and what is bad without harmonize it with the others. In his essay, Hume’s target was aesthetic taste and its subjective dimension and how common standards can rise and stabilize: his solution was to appeal to the experts, the connoisseurs, those whose expertise can be a guide for the others:
“ It is natural for us to seek a Standard of Taste, a rule by which the various sentiments of men may be reconciled; at least, a decision, afforded, confirming one sentiment, and condemning another.” In order to achieve this, human beings have to appeal to connoisseurs, men with special qualities:
“Strong sense, united to delicate sentiment, improved by practice, perfected by comparison, and cleared of all prejudice, can alone entitle critics to this valuable character; and the joint verdict of such, wherever they are to be found, is the true standard of taste and beauty” […] Though men of delicate taste be rare, they are easily to be distinguished in society, by the soundness of their understanding and the superiority of their faculties above the rest of mankind. The ascendant, which they acquire, gives a prevalence to that lively approbation, with which they receive any productions of genius, and renders it generally predominant. Many men, when left to themselves, have but a faint and dubious perception of beauty, who yet are capable of relishing any fine stroke, which is pointed out to them.”[10]

But the appeal to the authority of wise men outside the aesthetic domain doesn’t seem to fit the rhetoric of the Modern Age and the Scientific Revolution according to which the quest of knowledge has to be based on collectively controllable experimental method and not on the authority of the elder masters. Indeed, there are many domains outside of aesthetics in which standards of quality matter and we don’t want them to be produced by the discretionary power of an authority. Quality is not just a matter of taste when we look for standards of epistemic quality, that is, the quality of knowledge we may acquire, or food quality, that is, not only the good or bad taste of food that we ingest, but the quality of its standards of eatability. Also, industrial production quality control procedures cannot be the result of the appeal to an authority, as well as life parameters, like the minimum wage, should be based on collectively agreed standards.

The need of an objective notion of quality raises many questions that I will try to tackle in the rest of this chapter:

· Is it possible to get rid of a normative notion of quality and rely only on mechanisms of aggregation of lay judgments?

· How is a collective standard of quality constructed and maintained in a culture?

· Are there “better” and “worse” systems of quality assessment?

My point is that quality is an intrinsic normative notion that doesn’t make it less “objective”. It is a normative notion based on the historical records of an item, i.e. its reputation in a community. “Quality” as a term has always been employed with reference to a scale of value. In philosophy the “quality” of an item is an attribute of the item that makes it fit into a certain category. The activities of categorizing items and that of ranking are thus intrinsically dependent one on the other. Cultures produce rankings of quality standards, ratings of items because this is the way of making sense of the world outside us, of sorting things in order to make them fit into a certain category. I will claim that quality is a normative notion insofar as it is a standard that is constructed within a particular tradition. What is a tradition? Traditions are evaluated taxonomies and rankings that are selected and stabilized in a culture by many different “forces”:

· Institutions: public structures whose aim is to assure the coordination and maintenance of a collectivity.

· “Sacred values”: those values in a culture that are deeply related to its identity and are hard to question or change.

· Functionality: those aspects of traditions that are socially functional and help to accomplish socially coordinated tasks.

· Problem-solving: traditional cultural artifacts are ways of solving practical problems of information sharing and productivity.

· Biases: tendencies of a culture to reinforce in a particular direction a value or a position in a ranking. For example, a culture may give a special weight to literacy because of the intrinsic value that this represents in its development.

Thus, standards of quality come from the collectively evaluated corpus of knowledge and practices that we call tradition. We trust a tradition because it imposes on our way of seeing reality a ranking, a system of evaluation that orients us in our acquisition of information.

Let me introduce an example of a cultural artifact that is maintained and sustained as a fundamental part of our tradition by many of the various forces I have mentioned above: writing. Writing is a cultural technique that introduced at the end of the 4th millennium BCE in Mesopotamia as a device for external memory that makes possible the reorganization of intellectual life and the structuring of thoughts, neither of which are possible in oral cultures. With the introduction of writing, one part of our cognition “leaves” the brain to be distributed among external supports. The visual representation of a society’s knowledge makes it possible to both reorganize the knowledge in a more useful, more ‘logical’, way by using, for example, lists, tables, or genealogical trees, and to solidify it from one generation to the next. What’s more, the birth of “managerial” casts who oversee cultural memory, such as scribes, astrologists, and librarians, makes possible the organization of meta-memory, that is, the set of processes for accessing and recovering cultural memory. Printing, introduced to our culture at the end of the 15th century, redistributes cultural memory, changing the configuration of the “informational pyramid” in the diffusion of knowledge. Writing, among other functions, helps us to categorize our past history. But why in the modern era of printing and the contemporary era of computers and Internet hand-writing is still so reinforced in school programs? That is because it is stabilized by many forces: schools, “sacred values” against illiteracy, and functionality. Even if hand-writing is a very complex graphical technique that is no more “functional” to acquiring writing skills (typing is enough in many contexts) other forces such as “sacred values” maintain handwriting in our school programs. Our illiterate past is still too close to give up to the sacred value of writing, even if its functional role is reducing thanks to new technologies.

Here I would like to make a more general point about the role of past evaluations and preferences in filtering information. I’ll start with a parallel with some famous remarks Edmund Burke wrote about the importance of traditions. Burke was suspicious of revolutions because they risked to wipe out centuries of tradition, that is, of patiently collected and selected values, judgements and preferences refined throughout the ages. And this process of refinement is for Burke the essence of civilisation, of this thick cultural lore of judgements, values and opinions that penetrates into our minds through education and socialization and constitutes the necessary background of any form of wise thought. If we do not take into account the lore of traditions, we are condemned for Burke to reinvent the wheel at each generation. Our capacity of thinking the world and the institutions around us is much more limited without the contribution of the preferences already aggregated in the past by others. As he says:

« We are afraid to put men to live and trade each on his own private stock of reason; because we suspect that this stock in each man is small, and that the individuals would do better to avail themselves of the general bank and capital of nations, and of ages »[11]

Burke was politically wrong but, in some sense, epistemically right: there is something true in his reactionary remarks, even if their application to the analysis of the French Revolution is wrong for many reasons. One reason why his claims on revolutions are unacceptable today is that obviously not all traditions are worth preserving: the institutional biases and the social pressures that make a political tradition survive may be so wrong and oriented to defend the privileges of just one social class, that it is sometimes wiser to entrust a new generation to rethink the whole institutional design of a society from scratch. But, from an epistemic point of view, he captures the intuition that it is almost impossible to think from scratch, to know from scratch, without taking into account the lore of others’ preferences and values as it is filtered by a culture. This is an important epistemological point that evokes a similar, epistemological idea expressed by W.V.O. Quine in a famous article on his mentor, Rudolph Carnap: “The lore of our fathers is a fabric of sentences. In our hands it develops and changes, through more or less arbitrary and deliberate revisions and additions of our own […] It is a pale gray lore, black with fact and white with convention. But I have found no substantial reasons for concluding that there are any quite black threads in it, or any white ones.”[12] That is, the lore of a tradition, even of a scientific tradition, doesn’t transmit just a bunch of facts from a generation to another, but a sophisticated ensemble of judgements and conventions that shape the way facts will be extracted and classified in a given culture at a given time.

Preferences, conventions and values that others have expressed thus play a critical role in the making of collective wisdom: they shape the reputational landscape that we use to organize our own heuristics to extract information and provide a - sometimes reliable and sometimes too biased - shortcut to what is worth keeping, remembering and preserving as knowledge. The epistemological enquiry to collective wisdom I am advocating here implies that reputation and rating systems are an essential ingredient of collective processes of knowledge: their cognitive role in extracting information doesn’t depend on the intrusion of social factors than are external to the epistemological process, as many have argued. Reputation is a rational criterion of information extraction, a fundamental shortcut for cumulating knowledge in processes of collective wisdom and an ineludible filter to access facts. In my view, in an information-dense environment, where sources are in constant competition to get attention and the option of the direct verification of the information is simply not available at reasonable costs, evaluation and rankings are epistemic tools and cognitive practices that provide an inevitable shortcut to information. This is especially striking in contemporary informationally-overloaded societies, but I think it is a permanent feature of any extraction of information from a corpus of knowledge. There is no ideal knowledge that we can adjudicate without the access to previous evaluations and adjudications of others. No Robinson Crusoe’s minds that investigate and manipulate the world in a perfect solitude.

Thus, the standards of quality of collective knowledge are produced by a weighted incorporation in the production of our singular judgments of values filtered through time. That is what gives authority to a collectively produced piece of knowledge: we trust the wisdom not only of our contemporary crowd, but also of the past crowds who contributed to the crystallization of a tradition. This doesn’t mean that we are passive receivers of the authority of a tradition: traditions are indicators of value, they point to the proxies [13]that allow us to orient ourselves in a space of knowledge we do not yet master. When we enter a new domain of knowledge or a new cultural corpus we acquire the “taste” of the authorities in the domain in order to orient ourselves (the “you have to like this” effect). Who are the “good” and who are the “bad”? This is the way in which a canon is constructed. Then, the more we become autonomous thinkers, we challenge these traditions, participate to transform them and create new canons. It’s a salient feature of our contemporary knowledge world, so saturated of information, that different canons bloom, they rise and collapse in a much shorter period than it used to be before the advent of the decentralized society of information. That is, quality commons are structured in received traditions that are learned and amended from one generation to another.

Collective knowledge is presented today as a form of empowerment that frees us from the deference to experts and authorities. Nevertheless, as I have tried to argue here, experts and authorities have never been so present and influent in producing knowledge as a common achievement. Even is the impersonal game of science, as Steven Shapin has recently argued: “people and their virtues have always been pertinent to the making, maintenance, transmission and authority of knowledge”[14]. And even more in the aggregation of lay judgments, we must not forget that these lay judgments are based on received views and trust in authorities and traditions that do not come out of the blue. The power of collective knowledge is thus to articulate in a new way our trust in the transmitted authoritative views with the possibility of instantaneously sharing these values with others, thus amending these traditions and making them evolve more rapidly.

---

[1] M. Douglas (1975) Implicit Meanings, London: Routledge and Kegan Paul.

[2] J. Surowiecki (2004) The Wisdom of Crowds, New York: Random House.

[3] S. Shapin (2008) The Scientific Life, Chicago : Chicago University Press; A. Goldman (1999) Knowledge in a Social World, New York : Oxford University Press.

[4] R. Hardin (2009) How Do You Know? Princeton: Princeton University Press.

[5] Cf. J. Baker (1943) The Scientific Life, New York: Mac Millan, pp. 36-37, quoted in Shapin (2008) cit. p. 9.

[6] See for example C. Anderson (2009) “The End of Methods” Wired.

[7] Cf. C. Shirky (2008) Here Comes Everybody, New York, Penguin, p. 16.

[8] Cf. C. Anderson, cit.

[9] Cf. D. Hume (1757) “Of the Standard of Taste”, originally published in his Four Dissertations.

[10] Cf. D. Hume, cit. § 6; 27.

[11] Cf. E. Burke (1790) Reflections on the Revolution in France, in E. Burke, Works, London, 1867.

[12] Cf. W. V. O. Quine (1954) « Carnap and Logical Truth » reprinted in W. V. O. Quine (1961) The Ways of Paradox and Other Essays, Harvard UP, Cambridge, MA.

[13] For the notions of indicator and proxies see K. Davis, B. Kingsbury, S. Engle Marry (2010) “Indicators as a Technology of Global Governance”, IILJ Working Paper 2010/2, New York University School of Law; G. Origgi (2008) “Un certain regard. Pour une épistémologie de la reputation”, Rome. Workshop on Reputation. April 19-22.

[14] Cf. S. Shapin (2008) The Scientific Life, Chicago University Press, p.4.