![[IOP]](http://images.iop.org/cws/icons/themes/phw/colophon-small.png){kind=icon}
By James Dacey
Comparing apples and oranges. (Courtesy: iStockphoto/Raymond Gregory)
In an opinion article just published on this website, the historian and philosopher Robert Crease discusses the pitfalls when trying to measure the “value” of artefacts in culture and society. The article was inspired by an appearance Crease made at a group debate at the Museum of Modern Art (MoMA) in New York City. The panel were trying to identify ways to measure the museum’s impact on culture and the wider economy.
Crease refers to something known as Goodhart’s law, which he explains as follows.
“Named after the British economist Charles Goodhart, who devised it in 1975, the law essentially says that once a measure is chosen for making policy decisions, it begins to lose value as a measure. Goodhart applied it to banking policy, but in other fields, too, measurement can distort not only the practice being measured, but also perception of the goal.”
One example of Goodhart’s law is trying to measure researchers’ quality by the number of papers they produce. As soon as you define this as a measure, researchers could work within the system by churning out unnecessary numbers of low-quality papers. Likewise, there may be many brilliant scientists out there who go undervalued, because they do not score highly in the recognized assessments of quality. The most widely used being the h-index, which attempts to combine the quantity and impact of research papers.
Clearly, in evaluating researchers there is no silver bullet. Researchers can contribute to science in myriad ways, and attempting to compare them against each other involves a heck of a lot of subjectivity. But in the real world, this is precisely what universities and funding agencies have to do. And in the process, they do resort to these metrics and value judgements. In this week’s Facebook poll, we would like you to imagine you have to rank a bunch of researchers, by answering the following question:
What is the best way to measure a researcher’s contribution to science?
Number of publications
The novelty of their research
The frequency with which their work is cited
The prizes they have won
The applications that have sprung from their research
Something else (please post a comment on the Facebook poll, or on this blog)
In last week’s poll, we asked you the question “Should scientists speculate openly in the mainstream media about new science results?” Of the people who responded, 67% chose the option “yes, this would make science appear more exciting”. The remaining 33% went for the option “No, it is only useful to present clear-cut results”. We came up with the poll question after the theoretical physicist and blogger Matt Strassler had accused scientists involved in the AMS dark-matter experiment of spinning their results to journalists. He believes the scientists made them sound more exciting than they were.
Thank you for you participation and we hope to hear from you again this week.
From the first five items above only the novelty of their research and, in part, the applications that have sprung are adequate. Other two must be the generality of the contribution and how fundamental is the discovery.
I largely agree. Of the list provided, novelty and, in part, applications are the most important. But more important than either is how fundamental the discoveries made by the researcher are. These matters can be better judged not by publications algorithms but by the good sense of academics with a lot of experience in the field. That is why good science departments do not need to use such contrived formulas. Their members are able to tell the wheat from the chaff. Of course, even the best evaluation at any one time can be overruled by future experts. Generality normally comes with fundamental research. Prizes do offer the reassurance that experts have already evaluated how important the research is. That might be helpful.
Well said Gonzalo. What is the best way to measure a researcher’s contribution to science?. There isn’t one. It’s a fallacy to even think you can “measure” a researcher’s contribution to science. You can judge it, but you can’t “measure” it. So the answer isn’t even Something else. It’s Buzzzzz. Does not parse.
The question is tricky, if not selfdefeating. If I happen to be among the most frequent answers, I do not appear much original.
It hss been estimated that Einstein’s contribution to science turns out twice that of Dirac, but Einstein’s fame appears hundred times larger than Dirac’s!
Applications that have sprung from a research seems to be the only unbiased measure.
Trackback: Blog - physicsworld.com