How are scientists assessed?

I put this blog together after having a twitter exchange with Mick Watson and reading his excellent piece about how not to sack your professors.

Scientists are assessed primarily by the quality of their research output.  A secondary assessment, but one which is tightly linked to publication quality is the  ability to generate research income.  I will come to that later…

Research output is primarily regarded as articles in peer reviewed publications, usually in scientific journals, though there are other methods of publishing.  In order to be published in a journal, an article must pass “peer-review” by multiple scientists who are expert in the same field as the submitted work.   The whole process of getting from a first manuscript to a final published article can be lengthy, involve multiple revisions and often require the author to go and do more research to satisfy requests from the peer-reviewers (often called “referees”).   I’ve explained this whole process elsewhere, but will  repeat it in a later post here.

However, what constitutes “quality”? In general this means being published in high-impact journals. A high-impact journal is one that is read by a lot of people and so includes a lot of articles that are cited by other articles. One measure of journal quality is to look at its impact factor. This is a number that reflects the number of citations that the journal receives. The simple view is that scientists that publish in journals that have high-impact are doing research that is widely respected. If you only publish in obscure, little read journals then your work is less regarded and so you are not such a good scientist. Unfortunately, this is a very simplistic view since some subject areas are not as trendy as others and so are less likely to appeal to high impact journals like Nature and Science. A further simplistic way to assess scientists is to count their total citations – how often do people cite their papers? If you work in a popular field, your citations are likely to be higher than if you work in a subject area that is less popular. This doesn’t make your work any the less important, or your quality as a scientist smaller, but a pure numerical measure of quality based on citations might be unfair unless carefully normalised against citations within your field.  These factors make assessing quality in a scientists output very difficult and how to do it best is a subject of continual debate.

In the UK every 5 years, there is a Research Assessment Exercise (RAE) for Higher Education Institutions (HEIs). The most recent RAE exercise (now called REF for Research Excellence Framework) published results at the end of 2014.   The RAE/REF process aims to assess all academics (not just scientists) within the context of their field and so give a fairer estimate of quality.  In the latest REF, each academic had to submit four “outputs” for assessment. For scientists, these were typically research papers.  The “outputs” were assessed by a panel of peers and graded, then the results published as departmental summaries, so departments never know how well outputs from individual academics rated.   The RAE/REF is important in the UK since the results directly affect the funding given by central government to individual departments over the next 5-year period

Like any household, UK Universities have to work within a budget.  They must spend less than their income in order to break even and keep some surplus for future investment in people, buildings and so on.  Income is critical to keeping the lights on and keeping everyone paid.  The income is not fixed, but depends on a number of factors such as:

  1. Number of students and where they come from.  All students generate fees income, but international students pay more than home students.  As a consequence, most institutions actively market themselves across the world in order to raise the proportion of non-EU students.
  2. Research grant income.  Academics who win research grants bring income to the university, not only to pay the salaries of the staff on the grants and to buy equipment and consumables, but also in overhead money on the grant.  The overhead money is what keeps the lights on and pays for infrastructure costs and essential administration.
  3. Funds from industry/licensing.  This can be very lucrative since industry funding normally attracts a bigger overhead.  If an institution develops something with high commercial value then this can add significantly to the university income.
  4. Funding directly from government through the REF.  After grading departments every 5-years, the size of this funding component is linked to the ranking of the department.  The formula for this differs in each 5-year cycle.

So to increase income, institutions can choose to attract more high-paying students, but to do this they have to invest in good teaching staff and student facilities.  They can choose to attract researchers who are successful at publishing “high-quality” papers  and hence usually also successful at winning grants and thus bringing in overhead income.  They can also attract staff who have strong industry connections.  Most institutions do a bit of all of this.  On the research side, there is a bit of a frenzy of recruitment coming up to a REF year as departments aim to hire research academics who will provide high quality outputs for the REF and so not only be able to fund their own research from external sources, but also boost the likely income from central Govt through the REF formula.

Few institutions have large reserves of cash they can afford to dish out to all their staff to help with their research, but have to think strategically in order to develop new areas with the limited uncommitted funds they have.  Should they fund Ph.D. studentships to help newly appointed young Academics get their work going?  Should they use the funds to support some core equipment that needs replacing or updating?  Perhaps some building work needs doing to allow a new star scientist to install their expensive equipment?  Or perhaps a longstanding and successful member of staff has just failed to get their core programme grant renewed and needs some bridging funding while they try again in the next round or with a different funding agency?

It can be difficult for department heads to make decisions about who they support and how.  It is clearly complicated and a simple “metrics” based approach is not going to be fair or work effectively for every individual.  While there have been some reports in the media about seemingly arbitrary income-based decisions leading to sackings or even suicide, I find it hard to believe that these tough and tragic events are ever due to a single factor.   The ability to generate research grant income is clearly an essential part of being a scientist since without funding you can’t do anything.  It is also true that few universities can afford to fund research programmes in full from their own resources.  What they can do, like the institution I work at,  is provide the environment that supports their staff to be as successful as they can in the current, highly competitive world of academic science research.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s