This site uses cookies. By continuing to use this site you agree to our use of cookies. To find out more, see our Privacy and Cookies policy.
Skip to the content

Share this

Free weekly newswire

Sign up to receive all our latest news direct to your inbox.

Physics on film

100 Second Science Your scientific questions answered simply by specialists in less than 100 seconds.

Watch now

Bright Recruits

At all stages of your career – whether you're an undergraduate, graduate, researcher or industry professional – brightrecruits.com can help find the job for you.

Find your perfect job

Physics connect

Are you looking for a supplier? Physics Connect lists thousands of scientific companies, businesses, non-profit organizations, institutions and experts worldwide.

Start your search today

Blog

What is the best way to measure a researcher’s contribution to science?

By James Dacey

Photo of an apple and an orange on a balance

Comparing apples and oranges. (Courtesy: iStockphoto/Raymond Gregory)

In an opinion article just published on this website, the historian and philosopher Robert Crease discusses the pitfalls when trying to measure the “value” of artefacts in culture and society. The article was inspired by an appearance Crease made at a group debate at the Museum of Modern Art (MoMA) in New York City. The panel were trying to identify ways to measure the museum’s impact on culture and the wider economy.

Crease refers to something known as Goodhart’s law, which he explains as follows.

“Named after the British economist Charles Goodhart, who devised it in 1975, the law essentially says that once a measure is chosen for making policy decisions, it begins to lose value as a measure. Goodhart applied it to banking policy, but in other fields, too, measurement can distort not only the practice being measured, but also perception of the goal.”

One example of Goodhart’s law is trying to measure researchers’ quality by the number of papers they produce. As soon as you define this as a measure, researchers could work within the system by churning out unnecessary numbers of low-quality papers. Likewise, there may be many brilliant scientists out there who go undervalued, because they do not score highly in the recognized assessments of quality. The most widely used being the h-index, which attempts to combine the quantity and impact of research papers.

Clearly, in evaluating researchers there is no silver bullet. Researchers can contribute to science in myriad ways, and attempting to compare them against each other involves a heck of a lot of subjectivity. But in the real world, this is precisely what universities and funding agencies have to do. And in the process, they do resort to these metrics and value judgements. In this week’s Facebook poll, we would like you to imagine you have to rank a bunch of researchers, by answering the following question:

What is the best way to measure a researcher’s contribution to science?

Number of publications
The novelty of their research
The frequency with which their work is cited
The prizes they have won
The applications that have sprung from their research
Something else (please post a comment on the Facebook poll, or on this blog)

In last week’s poll, we asked you the question “Should scientists speculate openly in the mainstream media about new science results?” Of the people who responded, 67% chose the option “yes, this would make science appear more exciting”. The remaining 33% went for the option “No, it is only useful to present clear-cut results”. We came up with the poll question after the theoretical physicist and blogger Matt Strassler had accused scientists involved in the AMS dark-matter experiment of spinning their results to journalists. He believes the scientists made them sound more exciting than they were.

Thank you for you participation and we hope to hear from you again this week.

This entry was posted in General and tagged . Bookmark the permalink.
View all posts by this author  | View this author's profile

5 comments

  1. Walter Cabrera -Febola

    From the first five items above only the novelty of their research and, in part, the applications that have sprung are adequate. Other two must be the generality of the contribution and how fundamental is the discovery.

  2. Gonzalo Munevar

    I largely agree. Of the list provided, novelty and, in part, applications are the most important. But more important than either is how fundamental the discoveries made by the researcher are. These matters can be better judged not by publications algorithms but by the good sense of academics with a lot of experience in the field. That is why good science departments do not need to use such contrived formulas. Their members are able to tell the wheat from the chaff. Of course, even the best evaluation at any one time can be overruled by future experts. Generality normally comes with fundamental research. Prizes do offer the reassurance that experts have already evaluated how important the research is. That might be helpful.

  3. John Duffield

    Well said Gonzalo. What is the best way to measure a researcher’s contribution to science?. There isn’t one. It’s a fallacy to even think you can “measure” a researcher’s contribution to science. You can judge it, but you can’t “measure” it. So the answer isn’t even Something else. It’s Buzzzzz. Does not parse.

  4. The question is tricky, if not selfdefeating. If I happen to be among the most frequent answers, I do not appear much original.

    It hss been estimated that Einstein’s contribution to science turns out twice that of Dirac, but Einstein’s fame appears hundred times larger than Dirac’s!

  5. roy t james

    Applications that have sprung from a research seems to be the only unbiased measure.

Leave a comment

Your e-mail address will not be published. Required fields are marked *

Guidelines

  • Comments should be relevant to the article and not be used to promote your own work, products or services.
  • Please keep your comments brief (we recommend a maximum of 250 words).
  • We reserve the right to remove excessively long, inappropriate or offensive entries.

Show/hide formatting guidelines

Tag Description Example Output
<a> Hyperlink <a href="http://www.google.com">google</a> google
<abbr> Abbreviation <abbr title="World Health Organisation" >WHO</abbr> WHO
<acronym> Acronym <acronym title="as soon as possible">ASAP</acronym> ASAP
<b> Bold <b>Some text</b> Some text
<blockquote> Quoted from another source <blockquote cite="http://iop.org/">IOP</blockquote>
IOP
<cite> Cite <cite>Diagram 1</cite> Diagram 1
<del> Deleted text From this line<del datetime="2012-12-17"> this text was deleted</del> From this line this text was deleted
<em> Emphasized text In this line<em> this text was emphasised</em> In this line this text was emphasised
<i> Italic <i>Some text</i> Some text
<q> Quotation WWF goal is to build a future <q cite="http://www.worldwildlife.org/who/index.html">
where people live in harmony with nature and animals</q>
WWF goal is to build a future
where people live in harmony with nature and animals
<strike> Strike text <strike>Some text</strike> Some text
<strong> Stronger emphasis of text <strong>Some text</strong> Some text
WordPress Appliance - Powered by TurnKey Linux