“We are dwarfs standing on the shoulders of giants”. This metaphor from Charles of Chartes (12th century – nanos gigantum humeris insidentes) illustrates the importance to build up on previous knowledge for each type of intellectual effort. In today’s science, a major and formal part of this effort is achieved by citations in published papers. This blog is about the quality of these citations.
“To Cite” may be in the top 10 of the verbs I hear the most frequently at work. “How should I cite”, “can I cite this?”, “do I need a citation for this sentence?” and so on.
These questions are mainly coming from students and young scientists. It seems like the more experienced ones know how to do… Seems like.
One can be cited for diverse reasons, either to support a statement, contradict a result or validate a hypothesis proposed earlier. It can also be that a paper is cited just wrongly, just not at the right place.
In the past, I observed here and there some of my papers cited in an unexpected way, but I didn’t pay much more attention to it. But after having several Msc students writing up in their thesis and coming to me with the same questions, I had a closer look.
We had a little group exercise at the occasion of a retreat: I selected one of my papers published a couple of years ago and we looked at the papers citing it. We wanted to evaluate if the citation was making sense or were misused. The results were quite surprising, I propose to share with you the main outcomes.
You will find some details about the selected paper below. This paper has been well cited in terms of numbers (65 as of today in google scholar), mainly because it was one of the first to show a soluble fraction from this type of organic matter, with potential repercussions at global scale.
Out of the 50+ papers citing it at that time, I easily found 40 of them, all peer reviewed in indexed journals. We read them and looked how the paper was cited (see the figure above). First surprise, we considered that only in 55% of the cases the citation was appropriate (Fine like this), meaning there was a problem in 45% of the citations… Worth, in 17.5 % of the cases, we wondered if the authors read the paper further than the title (Did you read the paper?). The citation was related to one keyword of the paper, or some vague association of ideas, but nothing can relate to the content of the work.
Then come three different types of errors:
– You’re going too far (10 %): the authors are over interpreting our results and are placing them at a much larger scale, or associating processes to our findings which were not explicitly demonstrated. This is obviously a common issue with citations: one build on the others findings, hypothesis and speculations, without a clear statement of the way it was initially reported.
– You got me wrong (10%): the quote contains an error: it reports a wrong method, wrong data, a mistake in the unit… This leads most of the time to a wrong interpretation. For example, in our paper, the amount of soluble compounds were very small, but with a comma put at the wrong place, the quantities became quite substantial.
– Cause and consequence (7.5%): here the problem is a bit more complicated to decipher: out of our observations, the authors who cited our study drew a series of logical constructions based on “if this then that”. This has for me a sophism structure, implying from observations on a reduce number of data a very general behaviour. This category as well as the “you’re going to far” fall into the trap of “confirmation bias” that may be so spread out in my field that it is worth another blog entry.
These three erroneous citation behaviours may create long term bias in the interpretation of the paper. In this specific case, the problem is limited to a small piece of science, still, it is worth keeping an eye on it.
About the paper
It was a short communication I published in 2011 in soil biology and biochemistry: Abiven, S., Hengartner, P., Schneider, M.P.W., Singh, N., Schmidt, M.W.I. (2011) Pyrogenic carbon soluble fraction is larger and more aromatic in aged charcoal than in fresh charcoal. Soil Biology and Biochemistry 43, 1615-1617.
It was based on a simple experiment: we took two charcoals, one freshly prepared in our lab and another one picked up in the field 10 years after a fire event. We shacked them into water, filtered them and analysed the water solution for fire-derived organic matter using a molecular marker method. Main results are in the title of the paper.