Why do academic publications lack rigour? Taylor on the problem of woozles

Tweet

Do you know what a ‘woozle’ is? It’s a creature that pops up in the Winnie the Pooh series. Winnie, assisted by Piglet, tracks the woozle around the hundred acre wood. But it doesn’t really exist. Winnie, it turns out, is tracking his own footsteps and the more he tracks them, the more footprints appear, and the evidence he finds for the existence of the woozle. It takes an intervention from Christopher Robin to make him see the light.

Academia, according to James Stacey Taylor, is replete with woozles. These academic woozles are not fictional creatures they are, rather, fictional ‘facts’: propositions or claims that are taken to be true simply because they are repeated and cited so often without anyone checking to see whether or not they are true. One of the most infamous academic woozles is the claim that spinach is rich in iron, even though it is not. This woozle allegedly influenced the creator of the Popeye cartoon and was debunked many years ago by TJ Hamblin. But, somewhat ironically, it turns out that this debunking of the woozle may itself be a woozle since Hamblin cites no sources in support of it.

Some academic woozles are innocuous in nature. For example, some involve the misattribution of authorship, such as the scientific papers attributed to ‘M.A. Cantab’ (actually a degree award from Cambridge University). Some are more serious, such as the spinach claim (which could negatively influence diet). Taylor’s book Markets with Limits is a lengthy debunking of a recent woozle perpetuated by Jason Brennan and Martin Jaworski. In an influential 2015 paper — ‘Markets without Symbolic Limits’ — Brennan and Jaworski claimed that many objections to the commodification of certain human practices — e.g. selling sex or bodily organs — are symbolic or semiotic in nature. In other words, people object to the symbolic meaning of commercial transactions, not their harmful consequences or social effects. Taylor argues that this is wrong. Very few of the leading objections to commodification are premised on a symbolic claim of this sort, and this includes the objections made by people to whom Brennan and Jaworski attribute such a claim.

Taylor documents this in meticulous detail in his book and explains why he thinks Brennan and Jaworski’s woozle derails the academic debate about the ethics of commodification. Full disclosure: I am one of the people Taylor criticises for perpetuating the woozle. In my writings on the ‘Symbolic-Consequences’ argument in the sex robot debate, I have referenced Brennan and Jaworski’s paper, and used it as the basis for critiquing a number of objections to the creation of sex robots. In my defence, the validity of my argument does not depend on whether Brennan and Jaworski’s characterisation of the commodification debate is correct. The validity of my argument depends on whether I accurately characterise the arguments of those who object to the creation of sex robots (or, at least, the handful of objectors I wrote about in those papers). But Taylor is fair to point out that I simply parrot Brennan and Jaworski’s claim that symbolic objections to commodification are common. I never bothered to check to see if their insistence on this was correct.

More interesting than Taylor’s debunking of this particular woozle, however, is his theory as to why woozles of this sort are common in academia, particularly in academic philosophy. In brief, he argues that modern scholarship is too beholden to market norms. In other words, academics publish lots of articles because they are rewarded for doing so (money, promotion, fellowships etc), but these rewards do not depend, much, on the scholarly rigour of what they are publishing. He argues that it would be better if we replaced these market norms with a set of academic norms that rewarded scholarly rigour instead.

In the remainder of this article, I want to analyse and evaluate Taylor’s argument. As you will see, the short summary in the preceding paragraph doesn’t quite do justice to the nuance of his position. I will try to unpack that nuance in what follows and explain why, even though I agree with a lot of it, I am not entirely convinced. At the very least, my experience of academia, and academic publishing suggests that the situation is more complicated than Taylor seems to allow and that his suggested solution — that the goal of publication should be to enhance ‘understanding’ — has its own problems.

1. The Incentive Problem in Academic Publishing

Taylor critiques the use of market norms in academic publishing but, as best I can tell, he never precisely defines what he means by a ‘market norm’. You can, however, infer what he means from other things that he says. The market norms in question relate to the incentives faced by academics. His argument starts from the observation that academics are rewarded for publishing (lots of) work in top ranked journals. The more they do this, the more ‘external rewards’ they receive. The problem is that in doing so they have very little incentive to ensure that the work they publish is accurate, and neither do the referees or gatekeepers to academic publishing.

Here is the high-level summary of the argument from the opening chapter of Taylor’s book:

An academic’s professional success is largely judged by the number and quality of her publications, with the usual proxy for the latter being the reputation of the venue in which they appeared…But while academics have an incentive to publish, they have little incentive to serve as peer reviewers — and even less incentive to engage in the time-consuming work of checking submitted manuscripts to ensure that they are exegetically accurate. It is thus only to be expected that exegetically inaccurate work will enter the literature. 

(Taylor 2022, 2)

 

Once the inaccurate work enters the literature it can take on the guise of ‘fact’ or ‘received wisdom’ simply through repeated, uncritical, citation.

So I take it that when he refers to market norms in academia, he his referring to the fact that academics are incentivised to publish lots of work in order to obtain professional success (external rewards) and not incentivised to publish work for its own sake or the sake of its academic merit. In other words, I take it that, for Taylor, academic publishing guided by market norms is academic publishing that is pursued largely for external reward, not for intrinsic academic merit. It is not that ‘academic’ incentives are absent from the system. Taylor is clear in stating that normative systems often coexist within the same institutions and govern the same behaviours (see page 149 of his book). But the ‘market’ incentives dominate or usurp these academic incentives, at the present time.

Assuming I have got it roughly correct, it is worth breaking Taylor’s argument down in more detail. The main conclusion of the argument is that academic publishing guided by market norms will contain errors (he specifically focuses on exegetical errors but presumably the argument could be expanded to include other kinds of error). The argument in support of this works something like this:

• (1) If academics are externally rewarded for X, then, if they are prudentially rational, they will invest time in whatever maximises X and underinvest time and effort in whatever detracts from X

• (2) At present, academics are externally rewarded for publishing (a lot) of work in top venues.

• (3) Being academically rigorous, in particular by checking references and ensuring exegetical accuracy, slows down the rate at which academics can publish in top venues.

• (4) Therefore academics, if they are prudentially rational, will underinvest in rigour and exegetical accuracy.

• (5) Therefore, it is inevitable that exegetical errors will be present in academic publications and will be perpetuated by other academics for the reasons stated in premises (2) and (3).

Taylor doesn’t offer this formal reconstruction of the argument in his book. This is entirely my own interpolation. There is obviously a logical gap between (4) and (5), but I think it is easy to see how the gap could be filled and so the leap should be relatively uncontroversial. The controversial premises are (2) and (3). Let’s see what Taylor has to say about them.

On premise (2), here’s what Taylor says about the external rewards associated with publication:

The more publications an academic has in top venues, the more likely it is that she will be able to secure a tenured or tenure-track position in an academic department at a prestigious research institution. As an academic’s research profile rises, she could expect to receive lucrative job offers from institutions that compete with her own. Even if she does not accept such offers, she could use them as leverage to increase her remuneration from her current employer. A record of continuous and active publication in top venues will also make it more likely that she will receive invitations to give paid talks at other institutions, as well as grants, Fellowships, and other benefits that constitute the “spice of academic life”. 

(Taylor 2022, 142 - references omitted)

 

Many of these rewards are indirect in nature — i.e. you don’t receive a specific reward for each and every publication but rather you make it more likely that you will receive such rewards if you have a good publication profile. However, some institutions do offer direct financial incentives for individual publications. For instance some universities give ‘summer bonuses’ for publications and some give direct cash rewards for each and every publication. Taylor gives examples of these in the book.

Moving on to premise (3), Taylor says a lot about this — about why, specifically, academics underinvest in checking references and ensuring exegetical accuracy — but the gist of his argument is captured by the following passage:

An ambitious academic motivated by the desire for external reward would also be unlikely to spend much time ensuring that her citations to others’ work was accurate. She would believe — with some justification — that if her arguments were interesting, referees would not reject her paper on the technical grounds that her citations were inaccurate. Instead, they might only [assuming they can identify the errors] require that she revise them prior to publication…Similarly, a prudentially rational academic who aims to publish as much and as good as possible would be disinclined to verify that she had quoted sources accurately if nothing in her arguments turns on the precise phrasing used. And to increase her publication rate, she might eschew providing sources altogether, simply asserting that an argument that she addresses is one that is “widely held” or “common in the literature”.’ 

(Taylor 2022, 143-144 - references omitted)

 

That’s how Taylor defends the argument. Is it any good?

2. Evaluating Taylor’s Critique

The first thing to note about Taylor’s argument is that it is conditional on an academic being prudentially rational and hence being motivated to obtain external rewards. I’m sure that many academics are prudentially rational, at least to some extent, and that, as a rule of thumb, we can imagine that many are motivated to obtain external rewards. That said, my experience suggests that there are a lot of academics who are not particularly motivated by external rewards and not that bothered about publications. Admittedly, these tend to be academics that already have their foot in the door, so to speak. They have stable, permanent employment and enjoy their work-life balance. Consequently, they don’t overinvest in increasing their rate of publications. I think of myself as falling within this camp, even though I possibly do publish more than I need to. While more money would be nice, I earn more than enough to get by, and, to be frank, climbing the greasy pole doesn’t seem that appealing to me since it would, invariably, involve taking on more responsibilities and losing a lot of my research time (one of the odd paradoxes of academic life: the more senior you are the less time you have to do the writing and research that you enjoy).

Setting that to the side, however, I agree with a lot of what Taylor says. As an academic, I certainly feel that there is pressure to publish, particularly early in one’s career. I also agree that there is a perception that publication record is king when it comes to promotion and progression. That said, I am not convinced that premise (2) is all that persuasive, at least in the form I outlined above (and bear in mind Taylor does not formulate his argument in the way that I have done). I don’t doubt that there are some external rewards attached to publication, but I am not sure that they are compelling or universally felt. I am also confident that others things matter, perhaps more, than publication when it comes to achieving professional success as an academic. There are six points, in particular, that I would make.

First, I think Taylor’s claims about external rewards may apply primarily to US universities, and, specifically, to research intensive US universities. There are many other kinds of academic institutions and publication is not always treated as king at them. To take the obvious example, there are some teaching-focused institutions — small liberal arts colleges in the US — where publication takes a backseat to teaching.

Second, even in universities that value research, there are often other pathways to professional success. The academic institutions with which I am familiar tend to divide academic careers into three components: teaching, research and administration/contribution. Some of those institutions have formal promotion criteria that recognise each element equally. I have found that people that focus on administrative duties, particularly on taking up management or leadership roles, tend to do better than those that are publication-oriented. In addition to this, in these institutions ‘research’ is often taken to include much more than publication. It also includes things like number of PhD students, research funding, conferences organised and so on. I know lots of people that score highly on those metrics and have relatively sparse publication records.

Third, direct financial incentives for publication are, in my experience, rare and often administratively complex. I have seen attempts to implement them, but they often involve heated debates about what counts as a ‘top’ publication venue and they don’t often involve direct cash payments but, rather, payments into research accounts or other institutional accounts that people can use towards organising conferences, traveling or other job-related expenses. It is often administratively burdensome to access these funds since they require an audit trail and, given that the sums involved are typically small (less than €1000) it rarely seems worth the effort to me. Admittedly, I might be an outlier given my aversion to paperwork and form-filling, but I don’t think this aversion is that unusual. Suffice to say, unless these direct rewards are extremely large, they wouldn’t incentivise me.

Fourth, lucrative job and fellowship offers due to publication record may be common in some places but, in my experience, they are not particularly common. I’m only working from personal anecdote here but I have a reasonable publication record and I can safely report that I have never received any lucrative job offers as a result (If you want to send me one, please do). I think I have received maybe two fellowship offers, neither of which was paid, and, given family and other commitments, it was impossible to take them up. When it comes to fellowships, grants and the spice of academic life, my sense is that academic reputation matters more than publication record. Publications may be a good way of achieving reputation, but they are not foolproof and there are other means of doing so. Indeed, I suspect that having the right academic pedigree — being educated in highly ranked institutions and knowing the right people — matters a lot more than publications.

Fifth, in my experience, universities care more about what brings prestige and income into the university than they do about publication records. Academics that can win lots of research funding or attract students to lucrative postgraduate programmes do better than those with long lists of publications. Publication record might help someone to win funding or attract students but, again, it is not a guarantee. Many people win large research grants with relatively minimal publication records. I can, for instance, think of several people that have won multi-million euro research grants from the EU with half a dozen publications in top venues. I have been a reviewer on several such grants and they rarely afford significant weight to prior publication records. What’s more, the people that win these grants have received lucrative job offers and promotions as a result. Why? Because these grants bring money and prestige directly into the university (universities often take a % overhead from each grant) and they often get the benefit of more staff without having to directly pay for them (since salary is usually paid directly from the grant).

Sixth, and finally, I think there are probably significant diminishing marginal returns to publication. I suspect getting your first ten or so publications in top venues will do a lot more for your career than getting, say, fifty or sixty such publications. At a certain point in time, it becomes impossible for people to keep track of everything you have written and there is less gain to be wrought from each additional article. At this point in time, it probably makes sense to slow down one’s rate of publication and not to keep going at the same pace.

I readily concede that most of these objections stem from anecdote and personal experience and, again, I don’t doubt that academics do receive some external rewards for publications in top venues. I just think it is easy to overestimate this phenomenon and that its motivational power is less salient than one might suppose. Again, focusing purely on my own experience, I very rarely think about external rewards when I write and publish articles. They seem so diffuse and disconnected from what I do. I suppose, in the back of my mind, I think that having a long list of publications won’t hurt my career, but I’m also pretty convinced, at this stage, that it doesn’t help it all that much either either. I would probably do better if I invested less time in publication and more in, say, writing funding applications and taking on more senior leadership roles within my institution.

That’s just to focus on premise (2). What about premise (3)? There, I think Taylor is on more solid ground. Having seen publication from both sides — author and peer reviewer — I think it is correct to say that there is little incentive to check references and ensure exegetical accuracy. At least within philosophy, success in publication tends to depend more on novelty and cleverness of argument than it does on exegetical accuracy. It’s not that these things are irrelevant. If I am a reviewer and I know that someone is misrepresenting the literature, then I will point that out. But if I don’t already know it, I am not going to check to see that each and every source cited is accurately represented.

Furthermore, if I am an author, I do care about getting things right and I do try to accurately represent the sources I cite, but there are many things that I cite that are tangential to my main argument and so I am inclined to be less rigorous about them. For instance, in my citation of Brennan and Jaworski’s paper, for which Taylor takes me to task, it wasn’t important to me whether they had accurately characterised the debate about commodification. I was citing them for a different reason, specifically in support of the claim that symbolic meaning is not fixed and should not be treated as such in ethical debates. Their article does make the case for this, pretty well in my opinion (and Taylor seems to agree), and that’s what mattered to me. I was sloppy when it came to the rest of what they were saying, but that’s because it didn’t seem that important.

Even though premise (3) is on more solid ground, there are some potential objections to it. One could argue that even though it is unlikely that one will be taken to task for not ensuring exegetical accuracy, there is a risk that this will be picked up by someone else and that one’s professional reputation will be ruined as a result. This risk might be particularly acute if one acquires a reputation for sloppiness or is shown up for repeated mistakes. Certainly, careers have been, effectively, ended in other academic disciplines when people have been found to have fabricated research data or engaged in other forms of research fraud. The chances of this happening might be small, but if the punishments are sufficiently large (as I think they sometimes are), then a small chance of a big punishment might incentivise people to be more accurate.

Taylor seems unconvinced of this, particularly when it comes to exegetical errors. He thinks the adverse effects for those that make them are minimal and that the benefits of pointing them out tend to accrue to others by enabling them to publish more work pointing out these errors. So, in a perverse sense, errors add fuel to the fire of market-oriented publication:

Errors in academic work provide an opportunity for others to publish work correcting them. The more serious or widespread the error, the more likely it is that work written to correct it will be published. By increasing publication opportunities errors benefit academics by increasing their opportunities for professional advancement. 

 

(Taylor 2022, 171)

 

Still, as we will now see, Taylor’s solution to the problem focuses on increasing the penalties attached to exegetically sloppy work, and increasing the rewards associated with identifying and ensuring exegetical accuracy.

3. Solving the Incentive Problem

To this point, I focused on Taylor’s critique of the current publication system and its tendency to encourage the propagation of academic woozles. So what’s the alternative? How can we get out of the rut in which we are stuck?

The first thing is to acknowledge that there is something perverse about the present system insofar as it has lost sight of what the true purpose of academic research actually is:

The primary purpose of [existing] publications is not to advance the understanding of the issues that they address. Instead, it is to secure professional advancement…for their authors. But the primary purpose of academic research cannot be to function as a sorting mechanism to allocate the extrinsic rewards of academic research. The primary reason for an activity to be performed cannot be to determine who should receive the rewards of performing it. 

(Taylor 2022, 172).

 

Which raises the question: what is the primary purpose of research? Taylor equivocates a little bit here, conceding that some research might be valued for its consequences outside the academy, but that a lot of research, in particular philosophical research, is primarily focused on enhancing our understanding of the issues being addressed by that research. Enhancing understanding should, he argues, be the goal of academic publication:

I will argue in this chapter that academic research as it is practiced today should not primarily be directed by the norms of the market. I will argue that the purpose of academic research lies in its potential for furthering understanding. This will be more likely achieved if academic researchers direct their efforts to this end rather than seeing their research as being of mainly instrumental value as a means to secure their own professional success. 

(Taylor 2022, 159)

 

Ensuring exegetical accuracy — that you have properly understood the views of others — is a key part of enhancing understanding. So how can we go about redirecting academic minds toward that goal? The answer, somewhat ironically, lies in changing the present incentives. Taylor offers a number of practical suggestions but there are two main ones.

First, he thinks we should change the incentives of peer reviewers. Instead of relying on voluntary labour, we should pay peer reviewers for spotting errors in the manuscripts they review. What’s more, he thinks that these payments should be large (a high, per error rate of payment) and be made by the authors who submit their manuscripts for review:

The payments that are made to referees who detect and document errors should come from the authors in whose manuscripts the errors are detected…Authors could be required to place a certain amount of money in escrow at the time they submit their manuscripts. Alternatively (since this initial approach would disadvantage financially poorer academics), they could be required to sign an enforceable contract to pay any costs associated with error detection within a specified period of time… ….Referees might judge that the small bounty offered for the detection and documentation of error would be insufficient to compensate them for the time that they would spend on this. To rectify this, the bounties offered…would need to be fairly substantial. But the higher the bounty paid per error, the greater the incentive authors would have to avoid error in their work. 

(Taylor 2022, 187-188)

 

The other major policy reform suggested by Taylor is to stop rewarding academics for productivity and start rewarding them for high quality, low error publications. His main suggestion for doing this is to introduce a medium-term reward scheme for academics that produce high quality publications:

…short term bonuses [for productivity] should be replaced by medium-term incentives…which are presented to academics who produce a certain number of high-quality publications within the award period. To qualify for these incentives, eligible academics would be required to provide copies of the requisite number of publications to the award body…Academics who submit work that exceeds a certain pre-determined level of “excusable” inaccuracy would become ineligible to receive the incentive offered. 

(Taylor 2022, 189)

 

To prevent this reward system from being abused by sloppy but high productivity academics, Taylor suggests introducing a set of penalties to disincentivise them from submitting their sloppy work:

…academics eligible to apply for such incentives should be faced with significant disincentives to submit inaccurate work. This could take the form of the bounty-hunting scheme outline above…This scheme could also be coupled with the imposition of further penalties on academics who submit work that exceeds a certain pre-determined level of “excusable” inaccuracy. 

(Taylor 2022, 189)

 

Taylor has some other policy proposals too, but these are the main ones. What should we make of them?

4. Evaluating Taylor’s Proposals

It is difficult to know where to begin. Suffice to say, I don’t find myself persuaded by Taylor’s suggestions but I will start on a positive note. The current publication system is, frankly, bizarre. I cannot understand why we have for-profit publishers extracting a rent from a largely publicly funded system of research. I don’t understand how these for-profit publishers get away with relying on the free labour of researchers (journal editors, peer reviewers) to maintain their profitability. I also cannot understand why publication in for-profit journals is valued so highly by academics and institutions even if, as noted in the previous section, this valuation can be overstated. In other words, I readily concede that we have a bizarre incentive system right now and so Taylor’s new incentive system, as bizarre as it may seem to me, is not beyond the realms of possibility.

Still, I have a number of concerns about it.

First, the goal of ‘understanding’ seems very fuzzy to me. I’m not sure what it entails. At a minimum, it would need to be cashed out in terms of other features such as ensuring empirical adequacy, theoretical parsimony, predictive success, and, of course, exegetical accuracy. But I am not sure what the full list of features would be and they may well vary across disciplines. Furthermore, when cashed out in those terms, I don’t think it will capture what academic research should be about, at least not entirely. We don’t just care about understanding, we also care about policy reform, technological innovation, novelty, insight and so on. The present system may focus too much on novelty, but to eliminate novelty entirely would, to my mind, be a mistake. To put it more succinctly, there are many things for which academic research could be optimised. Understanding is part, but not all, of the picture.

Second, if we reduced understanding to ensuring exegetical accuracy, then I think we would have a less than ideal research system. I appreciate that Taylor does not reduce understanding to exegetical accuracy, but since he focuses so heavily on exegetical accuracy, it’s easy to come away with the impression that this is what he thinks matters most. It doesn’t and, in fact, a publication system that was fixated on exegetical accuracy would, to my mind, be sub-optimal. I can think of nothing worse than navel-gazing philosophy that focuses primarily on ensuring that we get the ‘correct’ interpretation of old writings. There is a lot of that about anyway and it seems to me to miss the point of philosophy. The point of philosophy is to ask and answer philosophical questions. These are the questions that motivated the likes of Kant, Hume, Plato and Descartes, and so figuring out what they said is valuable, but the primary focus should be on the questions and not on interpreting what these historical figures had to say.

Third, I think this system would encounter a number of practical snags, not least of which is arriving at some agreement about what counts as an error in understanding. There may be some clearcut errors of understanding or exegesis, but oftentimes there is room for reasonable disagreement about what sources say and scholarly debate often thrives on such disagreement. I am always surprised, and sometimes enlightened, by the fact that people interpret things that I wrote in a way that I did not intend. To impede scholarship on the grounds that its interpretation of its sources is not perfectly correct would, to my mind, be a mistake. It is often through debates about interpretation that we learn. Indeed, I learned a lot from Taylor’s book for this very reason. And it is also no surprise that it takes him an entire book to make his case.

Fourth, I imagine that the bounty-hunting system would be difficult to implement. I can’t imagine people voluntarily accepting it and I can’t see how we could get from our current set of norms to it. In any event, if it were implements, I suspect it would favour wealthy researchers at wealthy institutions and hence perpetuate inequality and lack of diversity in scholarship. Taylor’s concession to this point, suggesting that less wealthy researchers could have some deferred payment scheme misses the mark for me. Wealth researchers will be able to employ teams of research assistants to fact check their articles in advance of publication. It is unlikely that their rate of publication will be slowed down. Poorer researchers will not be able to do this. They may judge the financial risks of publication to be too high, particularly if they are not entirely sure what might be singled out as an ‘error’ in their manuscript. Perhaps you could resolve this problem by paying bounty hunters from other sources, but I suspect this wouldn’t have the disincentive effect desired by Taylor due to a moral hazard effect.

Fifth, I imagine that there would be a significant amount of overhead and bureaucracy associated with the reward scheme. In a sense, the UK REF schemes involved a variation of this, panels of researchers reading hundreds of articles to judge them on quality and not quantity. These exercises cost significant sums of money for minimal benefit. It is hard to see why any institution would implement this system when the marginal value of publication is so small. Still, stranger things have happened.

There are other criticisms one could make. Another obvious one is that Taylor’s proposal doesn’t eliminate market norms from the system. It simply replaces one set of external rewards with another: academics are still primarily motivated (or assumed to be motivated) to attain financial reward and avoid financial penalty. They are not pursuing academic rigour for its own sake. To be fair, Taylor has an answer to this criticism. He claims that if an activity has external rewards attached to it, or a transactional element to it, it does not follow that it is commodified or market oriented in a problematic way. Some use of market incentives can be beneficial and not corrupting. This may be a good point, but I’m not sure that it rescues the proposal.

To conclude, there is merit to Taylor’s criticisms and, strange as they may seem, at least he is proposing some alternative to the current system. I’m just not convinced that they would result in a better system of academic research and publication.

Subscribe to the newsletter