Monday, January 28, 2013
Sunday, June 1, 2008
A report from the IMU in cooperation with ICIAM and IMS. On the use and misuse of citation data and statistics in assessing research.
This is a report about the use and misuse of citation data in the assessment of scientific research. The idea that research assessment must be done using "simple and objective" methods is increasingly prevalent today. The "simple and objective" methods are broadly interpreted as bibliometrics, that is, citation data and the statistics derived from them. There is a belief that citation statistics are inherently more accurate because they substitute simple numbers for complex judgments, and hence overcome the possible subjectivity of peer review. But this belief is unfounded.
• Relying on statistics is not more accurate when the statistics are improperly used. Indeed, statistics can mislead when they are misapplied or misunderstood. Much of modern bibliometrics seems to rely on experience and intuition about the interpretation and validity of citation statistics.
• While numbers appear to be "objective", their objectivity can be illusory. The meaning of a citation can be even more subjective than peer review. Because this subjectivity is less obvious for citations, those who use citation data are less likely to understand their limitations.
• The sole reliance on citation data provides at best an incomplete and often shallow understanding of research—an understanding that is valid only when reinforced by other judgments. Numbers are not inherently superior to sound judgments.
Using citation data to assess research ultimately means using citation‐based statistics to rank things— journals, papers, people, programs, and disciplines. The statistical tools used to rank these things are often misunderstood and misused.
Saturday, November 10, 2007
What’s the price of a journal?
For that matter, what is the price of a car or a novel or a loaf of bread? All these things are frequently discounted, but we don’t throw up our hands and claim that they don't have a "real” price. Yet on several occasions recently, I’ve heard people say that we can't tell the price of journals because they are often discounted.
When the editorial board of the journal Topology resigned and began a competing journal, Elsevier wrote: “Because the majority of our subscribers purchase this journal in a larger set of journals, most are paying a fraction of the institutional subscription price.” I’ve heard similar arguments from other publishers, who like to compute the “price” of a journal by dividing the total revenue by the number of “subscribers”. But that’s not the price! It’s the “average revenue per subscriber."
The (list) price of a journal is set by the publisher, and it’s plainly visible to anyone who examines annual price lists. For some journals, there may be a two-tiered price, one for institutions and one for individuals, but in every case there is a price. Just as for cars or novels or bread, journals may be sold at a discount. But it's important to remember that publishers discount journals for business reasons, not because, in a sudden fit of remorse, they want to lower the price. Journals are sometimes discounted to agents, who consolidate them to help libraries purchase from multiple publishers. They are discounted to institutional members of scholarly societies as a member benefit, in return for dues. And journals are discounted to subscribers who buy bundles of journals, often making a commitment to buy for several years. In each case, the publisher is discounting journals in order to gain some advantage -- it's a simple business arrangement.
There is nothing wrong with discounting journals; it's good business. But it doesn’t change the price. Indeed, the price is the starting point for all discounting arrangements, defining the terms of a bargain: I’ll return a portion of the price in return for some action on your part – consolidating, being a member, or purchasing a bundle. Confusing the discounted price with the actual price ignores one half of the bargain.
We should pay attention to the list price of a journal because inevitably some subscribers (quite often, most) pay the list price. But there are other reasons not to let publishers substitute the "average revenue per subscriber" for the price. The average revenue is a quotient, and publishers control both the numerator and the denominator.
Unlike the list price, we must rely on the publisher to tell us the numerator, that is, the total revenue for a journal. Calculating total revenue sounds straightforward until one realizes that when selling bundles, large publishers apportion revenue among many journals – a somewhat mysterious process that isn’t easily discovered. For many publishers, the total revenue assigned to a particular journal is a very fuzzy number indeed.
The denominator is even more problematic. How many subscribers does a journal have? If a publisher adds many journals to bundles at no charge, the number of subscribers will quickly rise. But adding unwanted (and frequently unused) journals to bundles doesn't really change the number of subscribers to each journal in any meaningful way. Allowing publishers to use these arrangements to calculate either the average price per journal (for an institution) or the average revenue per subscriber (for the publisher) is like allowing politicians to count all those people who might have voted for them (but didn't vote) in an election. And allowing publishers to tell us the "real" price of their journals is like asking car salesmen to dictate the "best" price for their cars.
Scholars face a crisis today caused by high journal prices. If they are going to make headway in addressing that crisis, they have to get smarter about journals and more sophisticated about business practices. They can't allow publishers to redefine the problem by redefining the price. That's neither smart nor sophisticated.