Trahison des Clercs

veryard projects - innovation for demanding change

Trahison des Clercs

Notions, Patterns and AntiPatterns of Scholarship

veryard projects > knowledge management > trahison des clercs

standards and betrayal

background

on this page

links

The French term trahison des clercs refers to the betrayal of intellectual standards

The material on this page is part of a more general project to identify negative patterns in various domains.

Having served many times on programme committees, reviewing papers submitted to conferences and journal special issues, I have started to recognize some common patterns of poor scholarship.

In some cases, these patterns reflect deliberate abuses of the system, perpetrated by academics under pressure to publish widely and prematurely. In other cases, they reflect sheer ignorance or laziness on the part of the academic or his/her supervisor. In both cases, I believe, it is the duty of the reviewer or editor to detect such patterns and prevent their propagation.

Comments please Do you recognize any of these patterns? Can you add to the list?

Design pitfalls as negative patterns

Fallacies

Trahison des Clercs - Notions of Scholarship

veryard projects > knowledge management > trahison des clercs > notions

Scholarship

The process by which butterflies are transmuted into caterpillars. [Jerry Fodor]

Trundle, Trundling

Churning through books, documents and other secondary sources - and losing the ability to think for oneself.

"The scholar expends his entire strength in affirmation and denial, in criticizing what has already been thought - he himself no
longer thinks." [Nietzsche, Ecce Homo]

Trahison des Clercs - AntiPatterns of Research

veryard projects > knowledge management > trahison des clercs > research patterns

Vague objectives

Fearful of the possibility that a research project might fail to meet its objectives, risk-averse researchers and funding bodies conspire to set bland, vague objectives, whose achievement can be guaranteed in advance.

Sadly, there is sometimes little incentive for researchers to over-achieve on these minimal objectives. Any additional ideas or findings that appear original or interesting can be hoarded for the next funding application.

Unqualified questionnaire

I have sometimes reviewed papers whose main source of data is an opinion survey of some kind. For example, asking IT managers what they thought were the important technical issues in a given domain. Although such surveys could be useful for marketing purposes (enabling a vendor to align products and services to the perceptions of the customers), they rarely add anything to our technical knowledge or understanding of the domain.

I have also seen the same research process from the other side, having often received questionnaires from students asking my opinion on a range of topics. There seems to be no clear basis for selecting me as a member of some panel, no clear process (such as Delphi) for allowing the panel to converge on a collective expert opinion, and scant evidence that the student would be able to produce a meaningful result from the questionnaire returns.

I have even received such questionnaires broadcast via Internet email lists. On a recent occasion, the sender indicated that this was part of his doctoral research!

(In discussion with colleagues at the time, we wondered whether the guy’s supervisor really has no notion of what a ‘true sample’ might look like, or whether he was cynically promoting intellectually sloppy work in order to milk some funding institution that knows no better.)

Failure to connect theory with practice

For those [few] whose writings do engage with theoretical ideas, there is a continuing danger of falling between two stools – failing to add to the theory base but also failing to provide practical guidance.

Richard Heeks, What Did Giddens and Latour Ever Do For Us? (IFIP 9.4 Newsletter, Volume 11, No. 1, April 2001)

Trahison des Clercs - AntiPatterns of Published Papers

veryard projects > knowledge management > trahison des clercs > paper patterns

Unsupported "conclusions"

The conclusions often represent the hunches, beliefs and assumptions of the author, and are not supported by any theoretical analysis or empirical data. Sometimes they bear little or no relationship to the rest of the paper.

(It might be better to call them "preclusions".)

Projections

Premature attempts to publish preliminary work. Methods that have never been used, architectures and designs that have never been implemented, would-be standards and protocols that haven't been subject to any negotiation.

Academics are often put under pressure to publish something within the first year of a three-year research project, when they haven't yet got any results. Sometimes the ideas are good and interesting, and worthy of wider discussion, but they need careful critical analysis.

Inadequate references (except to oneself)

Some writers include more references to their own material than to anything else. This often seems to be a sign of desperation -- "nobody else will ever reference my papers, so I'll have to do it myself".

"In terms of communication between researchers, IS and development writings show a surprising and disappointing lack of inter-referencing. That is, theory-engaged writers rarely reference other IS and development writers, except self-references to their own previous work."

Richard Heeks, What Did Giddens and Latour Ever Do For Us? (IFIP 9.4 Newsletter, Volume 11, No. 1, April 2001)

Spurious, random and unselective use of references

There is a widespread but regrettable practice of citing references without any clear purpose. References should serve a clear function in a paper:

allowing theoretical concepts and frameworks to be used without lengthy definition or explanation;
citing empirical evidence;
or acknowledging complications and counter-arguments.

In many cases, however, a reference is thrown in without any evidence that the authors have actually read the book or paper being cited. Poor use of references sometimes gives the impression that the books and papers being cited are the only ones that the authors are aware of.

When reviewing papers, I sometimes helpfully provide references to relevant work. Some authors seem to think this gives them licence to add these references to the paper without even bothering to look at the work itself.

Salami tactics

A limited amount of material is sliced very thinly, so that it can be submitted to lots of different places. This can usually be detected because the authors cannot resist referencing all the other slices.

Pass the parcel

The same material is wrapped up in various ways. An academic typically sees a call for papers, and repackages an existing paper to squeeze it into the defined scope. This can usually be easily detected; often the key buzzwords can only be found in the introduction and conclusions, and nowhere else in the paper.

Dumplings

Indigestible lumps of material, copied whole from some internal report or specification document, with marginal relevance to the argument presented in the paper. Unprocessed data.

Bland and unconnected statements

For example, "This lack of progression may or may not be significant." Such a tautology can be used rhetorically, to lead into a discussion of possible significance. Without such a discussion, this tautology comes across merely as ponderous and pointless.

Trahison des Clercs - AntiPatterns of Research Conferences

veryard projects > knowledge management > trahison des clercs > conference patterns

Lack of focus

In the hope of catching as many good papers as possible, the conference call for papers is given as broad a scope as possible. The conference itself is given a bland and ambiguous title. Each submitted paper interprets the theme of the conference differently. Many reviewers seem to feel that it would be unfair to reject a paper simply because it has marginal relevance to the theme of the conference. The programme is determined by a large programme committee, often without meeting face-to-face, and there is no mechanism to maintain or impose any focus or coherence.

Vanity publishing

Some conferences accept far too many papers merely in the hope of increasing the number of attendees. Most of the papers are presented in multiple parallel sessions, with insufficient time for critical debate.

Out-of-keynote speakers

The theory of the keynote speech is that it sets the theme for the conference as a whole. Perhaps the first time you went to one of these conferences, you imagined that people at the conference would develop some of the ideas or themes of the keynote speaker. It certainly never happened at any of the conferences I have attended.

In practice, the keynote speech has become a form of patronage. Just as worthy charities have illustrious patrons on their letterhead, so international conferences need gurus, emeritus professors, media stars, articulate businessmen or politicians as keynote speakers. The hope is that people will recognize the name of the keynote speaker, and will be persuaded to attend the conference to hear him/her.

Everybody crowds into the main hall, laughs at the scripted jokes, and politely claps at the end. Then they disappear into their parallel streams and forget everything the keynote speaker said. Often the keynote speaker rushes to the airport to give the same performance somewhere else, and hasn't got the time to stay for the whole conference.

And in some conferences, there are several keynote speakers, offering rival keynotes. I attended one conference recently, where the third keynote speaker made jokes at the expense of the first (no longer present) keynote speaker. Cheap laughter, but not very enlightening.

"Proceedings" fixed in advance - don't reflect what happens at conference

Papers are submitted, revised and resubmitted months in advance of the conference. Delegates are given a bound copy of these papers on arrival. And that's it. There is no public record of the actual discussion.

Trahison des Clercs - AntiPatterns of Serial Research

veryard projects > knowledge management > trahison des clercs > series patterns

Lack of progression

Where an institution or working group organizes a series of conferences, there might be a reasonable expectation that each conference would build on the previous conferences in the series.

This rarely happens. Instead, each conference contains renewed (and inconclusive) attempts to invent conceptual frameworks, new terminology.

Lack of connection

When an academic submits a paper to a conference that belongs to a series, there might be a reasonable expectation that he or she has taken the trouble to look at the proceedings of previous conferences in the same series.

Sometimes the previous proceedings are barely mentioned. It is as if each conference is the first in the series.

Trahison des Clercs - AntiPatterns of Peer Review

veryard projects > knowledge management > trahison des clercs > review patterns

Defects in the reviewing process can be attributed to poor editing practice, poor reviewing practice, or to more complex interaction effects within the so-called peer review system.

Myth of the super-reviewer

A thorough review of a complex paper would require deep familiarity with prior work - not just the work referenced by the authors, but also relevant work ignored by them. It would require reasonable awareness of the authors' own work, to check that they were not merely rehashing previous stuff, and to check that they had addressed any objections or other issues arising from their own previously published work. It would require confident mastery of the notations and techniques (such as statistics) deployed in the paper. A super-reviewer, if one existed, would be able to detect inconsistencies and gaps in the argument, to surface conflicts with established terminologies or theories, and to anticipate a wide range of objections from various sources.

The myth of the super-reviewer distorts the behaviour of editors and programme chairs (who expect each reviewer to cover all aspects of a paper) and reviewers themselves (who may feel unable to reject papers without adequate justification).

Note that the software industry, for all its faults, has developed techniques for peer review that overcome some of these problems, and allow good use to be made of the actual knowledge and capabilities of a team of reviewers. Sadly, these techniques are not widely enough practised, even by software engineers.

Positive thinking

Perhaps aware of the bad feelings that accompany a destructive review, many reviewers attempt to identify the positive aspects of the paper as well as the areas for improvement. These attempts are worthy, and usually come from a desire to be helpful and positive, but they may lead to the false impression that the paper is better than it actually is, and reduces the force of genuine intellectual objections.

Single-cycle review

In many situations, there is only a single-cycle review. The revised paper doesn't go back to the (same) reviewers, and the editor cannot check in detail whether the revisions meet the requirements of the reviewers. There is little incentive for authors to engage seriously with the reviewers' comments.

My preference as a reviewer is to provide a detailed account of the shortcomings and imperfections of a paper, and indicate some things that would have to be improved before publication, as well as some further suggestions. It is galling to see my reviewing work wasted, as the paper subsequently gets published without any sign that my requirements or suggestions have been heeded.

Casual review

As reviewers learn the imperfections of the system, it becomes increasingly difficult to maintain a conscientious stance. If detailed comments are likely to be ignored, if conditional acceptance is converted by overworked editors into unconditional acceptance, then perhaps it's not worth the effort to review a paper properly.

Automatic review

The quickest way to review papers is according to a pre-programmed set of responses.

	If it's hard-going, then it might be profound. I don't have the time to work out whether it makes any sense, and to find enough flaws to justify rejecting it, so I'd better accept it.
	The paper goes outside my specialist area, so I don't have sufficient grounds for rejecting it.
	The paper doesn't conform to my preconceptions about subject or method, so it must be rubbish.
	The paper is obscure, so it must be rubbish.
	I agree/disagree with the findings, so the paper must be good/bad.

Tactical review

Reviewers may also sometimes allow their own personal agenda to influence their reviews.

	If this gets published, it will make my work look good in comparison.
	If this gets published, it will undermine my work.
	If this gets published, it will create a precedent for future publication in this area.
	If this gets published, it will foreclose further publication in this area..
	I'm outraged that this research got funding in the first place. I hope that no more funding gets wasted on this stuff. So although the paper itself is okay, I shall reject it, because I reject the research programme that sponsored it.

Of course, reviewers are unlikely to acknowledge such thoughts, even to themselves.

veryard projects > knowledge management > trahison des clercs