It’s almost a year ago that we started the peer review audit. In this blog, our senior research policy manager, Andy reflects on the main outcomes from the audit and why he thinks AMRC members really are ‘raising the standards of research funding’.
Let’s wind back time to May 2015. There was an ‘awakening’! Many of our members were waiting in earnest at the prospect of having to fill in a ‘peer review audit’. Some didn’t quite know what it meant. Others didn’t know it was happening. A few audit veterans had vague recollections from the previous one in 2010. Was it an exam? What would happen if they ‘failed’? Would they end up in peer review prison?
Let’s face it, peer review isn’t perfect but it’s really important –
- It’s a criterion of AMRC membership - we check new member’s policies when they join us, but also ask existing members to tell us how they undertake peer review every five years through the audit to ensure the quality hallmarks are maintained.
- The government also recognises AMRC membership as an indicator of quality. Because of the high-quality peer review processes, AMRC members’ funding in universities attracts additional quality-related (QR) funding from the charity research support element of QR (known as the charity research support fund or CRSF).
- Similarly, under AcoRD the Department of Health in England and the devolved nations meets some of the additional study support costs of charity-funded research in the NHS, but only for those charities that are members of AMRC.
These are all big reasons for doing peer review properly. But in May 2015, it was a challenging time for the whole sector. It felt like not week went by without charities being mentioned in the news – not all of it positively. To me at least it felt like there was a sense uncertainty - no one was quite sure what was around the corner…
Things started to look up
When we finally kicked off on May 26th there was a hive of activity as many charities started grappling with questions, digging out policies and checking the wording around exactly how they defined a ‘conflict of interest’. Time was tricking – there only 6 weeks to get this right but there was a real sense of pride and care in how members were responding to the questions. And a bit of friendly completion helped to oil the wheels!
Winding forward a few weeks, many started to submit their entries and AMRC began reading them. Submissions came thick and fast and by August, pretty much everything that was ready to go. Things were looking up!
Moving forward a few weeks again to October 2015, we held the peer review audit committee meeting. This was it! All those weeks of hard work were coming to a head and members would finally find out the outcome. The committee meeting itself was a mammoth day – we went through all the responses and deliberated carefully as to whether a charity passed or failed the audit. It’s also fair to say that the committee were tough – and rightly so. If AMRC is to promote best practice and ensure the quality hallmark is maintained, it’s vital that the process is thorough and meticulous.
What came out of the meeting was an overwhelming positive story that many of our members were going above and beyond the basics to ensure their peer review processes were as robust as they could be. There were some fantastic examples of charities being transparent with all of their all policies – even the ones like rotation, terms of reference and quorums for their committees. There were other examples where charities ensured their committees were balanced by asking a broad range of experts – including patients – to help undertake peer review. Others had new approaches to managing conflicts of interest. It was genuinely fascinating.
Some members did however fail the audit and committee gave clear actions as to what needed to be undertaken to improve their peer review practices. These charities were given until 31st March 2016 (around 5 months) to make these changes, and AMRC’s Executive Council gave a clear steer members needed to have completed their actions by this time to maintain their membership status.
All the findings, and where applicable, actions, were sent to members via feedback letters. Where members passed, they were issued with a lovely certificate – something that we hope is proudly on display on your websites and in your offices!
There are still some challenges
Moving on to Spring 2016, the vast majority of members that had initially failed the audit have completed the actions originally put to them. This is no small victory and we were really pleased that most of these members took any criticisms on the chin and worked with us to sort out the issues. A small number of charities were however unable to undertake the actions. These members have left AMRC and we hope to continue working with them outside of membership.
You can read about the main findings from the peer review audit in our report and briefing:
The headline figures are:
- 116 charities were invited to take part in the audit
- 4 charities resigned during the audit, mainly because membership was no longer applicable or because they had temporarily stopped funding research
- 4 charities were demitted from the Association – 2 because they did not fill in the audit, and 2 because they were unable to complete the actions by the March deadline.
As well as the positives, the audit highlighted the difficulties our members face undertaking peer review. We have used the information to revise the guidance around implementing the principles of peer review. There are also new challenges the sector faces – particularly as we see increasingly complex co-funding partnerships with new organisations and AMRC will continue to work with our members to ensure that decision making in new models of funding is consistent with our principles of peer review – something that we will be discussing in a workshop on July 13th.
But the future is bright
We hope that the peer review audit has been a useful exercise for all members. We have certainly found it valuable and our chief executive, Aisling Burnand sums up the audit from our perspective:
AMRC members work tirelessly to ensure the money so generously raised by the public is put to the best possible use. Maintaining high standards of research funding is vital – this underpins the AMRC quality hallmark. Our audit allows us to ensure that funding standards are being upheld.
Overall, we have much food for thought as we move forward to the next audit in 2020. By then, it’s likely the environment will have changed again. We hope members will embrace these changes and we will continue to support them sharing learning and identifying best practice as it develops.
If anyone has any questions on the audit, please feel free to contact me.