The ARC ERA and EI Action Plan sets out the next steps the ARC will take to implement these recommendations
ERA and EI Action Plan The Australian Research Council’s response to the ERA EI Review
ISBN 978-0-6484847-6-9 © Commonwealth of Australia 2021
All material presented in this publication is provided under a CC Attribution-NonCommercial- NoDerivatives 4.0 International (CC BY-NC-ND 4.0) licence with the exception of the Commonwealth Coat of Arms, the Australian Research Council (ARC) logo, images, signatures and where otherwise stated. The details of the relevant licence conditions are available on the Creative Commons website as is the full legal code for the CC Attribution BY-NC-ND 4.0 licence. Requests and enquiries regarding this licence should be addressed to ARC Legal Services on +61 2 6287 6600. Images: Icons via Vecteezy.com.
Message from the CEO
Over the past year, the ARC engaged an Advisory Committee (Committee) of leaders and experts from the university, industry, government and community sectors to conduct a comprehensive review of its national university research assessments – ERA and EI. The Committee, chaired by Emeritus Professor Michael Brooks, recently provided me with a report on the findings of the review and I am pleased to be presenting the report on the ARC website alongside the The Committee’s report sets out a strong agenda for the ARC to move forward with the ERA and EI programs. The Committee made 22 recommendations to improve the value of research assessment for Australia, streamline the programs through smarter uses of data and technology, and maintain best practice in research assessment into the future. The recommendations set out a new vision statement for ERA and EI and provide a timely update and alignment of the objectives of each program. The recommendations also set out practical measures for the ARC to adopt to make participation in the programs easier for universities, and to ensure the outcomes are rigorous and more beneficial for a wider range of stakeholders. I am pleased to accept the report’s recommendations. ARC response through this Action Plan.
The hard work of the Committee was supported by several expert working groups as well as the input and feedback from a wide-range of stakeholders through public and targeted consultations. This support was essential to the review and is remarkable in light of the challenges arising from COVID-19. I thank the Committee, the working groups, and the university and end-user stakeholders for The challenges of COVID-19 also highlight the need for the ARC, universities and other stakeholders to work together as we implement the Committee’s recommendations. This Action Plan is the first step in this process. It sets out next steps the ARC will take in implementing changes to ERA and EI, and indicative timeframes. Some changes will be immediate, while others will take place in later rounds. I look forward to their efforts. working with the university sector to ensure ERA and EI provide the benefits that the Committee envisaged.
Professor Sue Thomas Chief Executive Officer Australian Research Council
“The report incorporates considerable consultation with stakeholders and reflects the challenges faced by universities, researchers and end-users.”
ERA AND EI ACTION PLAN | 1
New Vision and Objectives for ERA and EI A new, unified vision statement and revised objectives for ERA and EI will support the programs’ capacity to inform and promote excellent, engaged and impactful research in an everchanging research environment. The new vision statement clarifies the overarching purpose of ERA and EI as companion programs that form a holistic national assessment system, in which research excellence, engagement and impact are integral and complementary aspects of Australian university research performance. The new vision for Australia’s research assessment is:
ERA AND EI ACTION PLAN | 2
ERA Objectives
PROMOTING EXCELLENCE Rigorously assess research quality to promote pursuit of excellence across all fields and all types of research
INFORMING DECISIONS Provide a rich and robust source of information on university research excellence and activity to inform and support the needs of university, industry, government and community stakeholders
DEMONSTRATING QUALITY Provide government and the public with evidence of the quality of research produced by Australia’s universities
ENABLING COMPARISONS Allow for comparisons between Australian universities and against world standard for all discipline areas
ERA AND EI ACTION PLAN | 3
EI Objectives
PROMOTING ENGAGEMENT AND IMPACT Rigorously assess research engagement and impact to promote better practice engagement with, and translation of research into benefits for, end-users and the Australian community more broadly INFORMING DECISIONS Provide a rich and robust source of information on the strategies, practices and benefits of university research engagement and impact to inform and support the needs of university, industry, community and government stakeholders
DEMONSTRATING SUCCESS Provide evidence that Australia’s universities are undertaking valued engagement with research end- users, embedding practices of engagement and impact, and producing research with social, economic, environmental and cultural benefits
ENABLING COMPARISONS Noting the need for an appropriate calculation methodology, provide a basis for discipline-level comparisons between Australian universities of research engagement and impact performance
ERA AND EI ACTION PLAN | 4
WHAT DO YOU NEED TO DO NOW? ♦ Start using the ANZSRC 2020 FoR codes to code research outputs and other data ♦ Ensure research is appropriately assigned to the FoR Division 45 Indigenous studies in ERA 2023 and EI 2024 What do universities need to know? ♦ Stop collecting ERA applied measures data for ERA submissions, noting the need for appropriate evidence requirements for EI ♦ Note that the ERA rating scale and benchmarking are being reviewed – the ARC will establish an expert working group to assist ♦ Note that EI engagement indicators will be optional, chosen by universities as appropriate for each FoR, and included only within the narratives ♦ Note that the EI definition of research end-user will be amended to include publicly funded research organisations WHAT DO YOU NEED TO PLAN FOR? ♦ Providing input into consultations and working groups: ♦ the ARC will be moving to a by-line method (post-2023) as the primary criteria for research output eligibility ♦ the EI rating scale will be revised to provide increased granularity and descriptors that are informative to all participants ♦ the definitions for approach to impact, and additional guidance for impact, engagement and research end-user as part of consultations on draft submission guidelines ♦ the development of a calculation methodology for the number of impact case studies required in EI, to be proportionate to the size and capacity of the discipline area ♦ Stronger encouragement to report Open Researcher and Contributor (ORCID) iDs and Digital Object Identifier (DOI) and link to research outputs ♦ Consider ways to help the ARC increase the size, quality and diversity of the ERA peer reviewer pool WHAT DO YOU NEED TO BE AWARE OF? ♦ The ARC will work with other government data custodians to use data from existing datasets where possible (e.g. HERDC/HESDC) ♦ ERA will move to annual data collection (post-2023), following the implementation of sufficient streamlining measures ♦ Future ERA submissions (post-2023) will be partially pre-filled using existing data sets ♦ The ARC will work to publish as much ERA and EI data as possible, including publishing research output metadata with FoR apportionment, for transparency ERA AND EI ACTION PLAN | 5
Changes to ERA and EI
CONTINUAL IMPROVEMENT Streamlining measures, reporting cycle, research excellence for the future, optional quantitative impact indicators
EI 2024 Self-determined EI indicators, clearer guidance and definitions, more impact studies for some UoAs, expanded EI rating scale
IMMEDIATE CHANGES New vision and objectives, greater data access and insights, streamlining measures, revised rating scales, Indigenous research
FUTURE ROUNDS Collect once approach to data submission, data harvesting for partial pre-filling of submissions, adoption of by-line method, annual collection
ERA 2023 Removal of applied measures, adoption of new ANZSRC codes, new ERA benchmarks and rating scale, ORCID iD and DOI uptake, improvements to evaluation methodology
ERA AND EI ACTION PLAN | 6
What is the ARC doing immediately?
New vision and objectives
Greater data access and insights
ACTION
Streamlining measures
Revised rating scales
Indigenous research
DESCRIPTION ♦ The ARC will adopt new objectives for ERA and EI to align the programs under a single vision.
♦ This ARC project is ongoing. Stakeholder discussions will commence in the 2nd half of 2021. ♦ Reporting requirements will be included in ERA 2023 and EI 2024 submission guidelines. increase the value of ERA and EI data by providing current and future data in a more accessible and transparent format to existing and new end-users. publishing the metadata for ERA research outputs with FoR apportionments following an ERA round. promote greater insights from ERA and EI data, especially for government and other end-users. ♦ The ARC will work to ♦ This will include ♦ The ARC will work to
♦ The ARC will consult with stakeholders on the revision of rating scales for the next rounds of ERA and EI. ♦ An expert working group will provide advice on recalibrating the ERA rating scale, the peer review and citation analysis assessment benchmarks, and the definition, appropriateness and application of the ‘world standard’ benchmark. ♦ The ARC will also revise the EI rating scale and rating descriptors, in consultation with stakeholders.
♦
The ARC will implement the new ANZSRC FoR Division 45 Indigenous studies in ERA 2023 and EI 2024. The ARC will assess Indigenous research through peer review. The ARC will work with universities on leadership in implementing Division 45.
♦ The ARC will undertake work on measures to streamline the submission process for universities, including use of existing data sets and new technological approaches. ♦ This will include aligning ERA data requirements with Department of Education, Skills and Employment (DESE) higher education data collections such as Higher Education Research Data Collection (HERDC) Higher Education Staff Data Collection (HES DC ).
♦
♦
TIMEFRAME* ♦ New vision and objectives will be
♦
♦
Consultation with universities on rating scales for ERA and EI will commence in the 2nd half of 2021.
♦ The ARC will continue to work with universities and Indigenous researchers to support the implementation of Division 45.
The ARC has commenced discussions with the DESE on the use of HERDC and HESDC data.
adopted immediately.
An expert working group will be established for ERA ratings, which will report in the 1st half of 2022. New ERA and EI rating scales will be published alongside submission guidelines in 1st half of 2022. ♦ ♦
*Timeframes are indicative
ERA AND EI ACTION PLAN | 7
What changes will be made for ERA 2023?
Implementation of new ERA benchmarks and rating scale
Removal of applied measures
Adoption of new ANZSRC 2020 codes
ACTION
ORCID iD and DOI uptake Improvements to evaluation methodology
DESCRIPTION
The ARC will remove the requirement to collect applied measures for ERA 2023: plant breeder’s rights patents registered designs research commercialisation income NHMRC endorsed guidelines ♦ Universities no longer need to collect this data for ERA purposes.
ERA 2023 and EI 2024 will use the revised ANZSRC FoRs to determine the UoE and UoA respectively.
As mentioned in the previous table, the ARC will seek expert advice through a working group to review the ERA rating scale, benchmarks and definition and appropriateness of ‘world standard’ for peer review and citation methodologies.
♦ The ARC will encourage greater uptake of Open Researcher and Contributor (ORCID) iDs and Digital Object Identifiers (DOIs) linked to research outputs for ERA 2023.
♦
♦ Key changes will be in areas of: Indigenous Research, Technology, and Medical and Health Sciences.
The collection of these unique identifiers now will allow for greater automation and verification of future ERA submissions.
♦ The ARC will aim to increase the size, quality and diversity of the peer reviewer pool. ♦ Assessment committees will have authority to exclude units of evaluation or research outputs where significant miscoding has occurred and request a recalculation of citation profiles. ♦ New assessment advice will be published in ERA evaluation and peer review handbooks in the 2nd half
TIMEFRAME*
♦ This change will be immediate.
♦ This change will be
♦ New ERA rating scale and information on benchmarking will be published alongside
♦ Reporting requirements will be included in ERA 2023 submission
immediate.
guidelines.
submission guidelines in the 1st half of 2022.
of 2022.
*Timeframes are indicative
ERA AND EI ACTION PLAN | 8
What changes will be made for EI 2024?
Develop a definition of approach to impact and additional guidance ♦ The ARC will develop a definition for approach to impact. Clarity on the definition of approach to
Number of impact studies proportionate to UoA size and capacity
Self-determined EI Indicators
Expanding definition of research end-user
Implementation of expanded EI rating scale
ACTION
DESCRIPTION
♦ The definition of research end-user will be expanded to include publicly funded research organisations, and will focus on the nature of the impact or the intent of the activity, rather than the nature of the organisation.
♦ The ARC will develop a calculation methodology to determine the number of impact studies to be submitted per unit of assessment. ♦ The number will be proportionate to the size and capacity of the discipline area. ♦ Any increase will be modest.
♦ The ARC will work to develop an expanded, more granular and
♦ The exception to this will occur where the ARC may be able to capture data to provide streamlining benefits such as data for HDR student co- supervision with research end-users. the indicators that demonstrate the effectiveness of each UoA’s engagement activities, to be included in narratives. ♦ The ARC will allow universities to determine
meaningful rating scale for engagement, impact and approach to impact.
impact will provide for clearer expectations for universities and assessors.
♦ Additional guidance will be developed drawing on examples from previous impact studies and engagement narratives.
TIMEFRAME*
♦
This change will be immediate. ARC will work with DESE on the opportunities that may be available to use HDR data as an engagement indicator. Further advice will be included in the EI 2024 submission guidelines by the 1st half of 2022.
♦ The new definition of research end-user will be used in the EI 2024 submission guidelines by the 1st half of 2022.
♦
The ARC will consult with universities during the 2nd half of 2021. The new definition of approach to impact will be used in the EI 2024 submission guidelines by the 1st half of 2022.
♦
The ARC will consult with universities during the 2nd half of 2021 to develop a calculation methodology for determining the number of required impact studies. The calculation methodology will be used for EI 2024.
♦
The ARC will consult with universities during the 2nd half of 2021. The new rating scale will be included in the EI 2024 assessment handbook in the 2nd half of 2022.
♦
♦
♦
♦
♦
*Timeframes are indicative
ERA AND EI ACTION PLAN | 9
What work will be done for future rounds?
Collect once approach to data submission
Data harvesting for partial pre-filling of submissions
ACTION
Adoption of by-line method
Annual collection
DESCRIPTION ♦
The ARC will move to a ‘collect once’ approach to minimise the need to provide the same data for multiple government data collections, reducing the reporting burden for universities. This will include aligning ERA data requirements with DESE Higher Education Data collections such as HERDC/HESDC.
♦
The ARC will work with universities, government departments and citation providers to enable greater harvesting of data from existing data sources. The ARC will explore the potential use of automated FoR assignment to research outputs. Work will commence in 2nd half of 2021 and be ongoing until submissions are due for the ERA round after ERA 2023. The ARC will continue to evaluate data harvesting and pre-filling technologies each round, and incorporate into the ERA data collection processes as appropriate.
♦ The ARC will adopt a by-line method for determining research output eligibility post ERA 2023. An expert working group will address complexities around implementation.
♦ To reduce the intensive reporting burden of previous rounds, the ARC will commence a transition to annual collection of ERA research output and other data, following the implementation of sufficient streamlining measures.
♦
♦
♦
♦
ARC will consult with universities and establish an expert working group due to report in the 1st half of 2022. Universities will be advised of the outcomes of the report at this time. By-line eligibility for research outputs will be included in ERA submission guidelines after ERA 2023.
♦
The ARC will consult with universities with timing dependent on the outcomes of other streamlining measures. Annual reporting will be implemented in ERA submission guidelines after ERA 2023.
TIMEFRAME*
♦ Discussions with DESE have commenced.
♦
The ARC HERDC and HESDC staffing and income data will be used for the ERA round after 2023.
♦
♦
♦
*Timeframes are indicative
ERA AND EI ACTION PLAN | 10
What continual improvement will be undertaken?
Research excellence for the future
Optional quantitative impact indicators
ACTION
Streamlining measures
Reporting cycle
DESCRIPTION ♦ The ARC will raise awareness that activities such as annual collection cycles are contingent on sufficient streamlining being in place.
♦ The ARC considers that three- yearly evaluation cycles are desirable for ERA and EI to ensure timely data. ♦ ERA and EI reference periods will remain unchanged if the reporting burden for ERA and EI is offset by streamlining for ERA submissions.
♦ The ARC will continue to explore new methods that streamline the submission and evaluation processes of ERA and EI. ♦ The ARC will continue to monitor excellence and investigate how they may be incorporated into future ERA and EI rounds. ♦ The ARC will continue to monitor and develop technologies that better showcase the data generated by the assessment process. international best practice understandings of research
♦ Quantitative impact indicators will remain optional for the assessment of research impact in EI 2024. ♦ The ARC will continue to investigate and monitor international developments.
♦ The ARC will monitor the
effectiveness of streamlining measures following implementation.
TIMEFRAME*
♦ The ARC will consult with
♦ The ARC will consult with
♦ Ongoing.
♦ Requirements for impact evidence and indicators will be included in the EI 2024 submission guidelines by the 1st half of 2022.
universities and review the effectiveness of streamlining measures post-ERA 2023.
universities post-ERA 2023 to inform options on ERA and EI reporting frequency.
*Timeframes are indicative
ERA AND EI ACTION PLAN | 11
arc.gov.au > Excellence in Research for Australia arc.gov.au > Engagement and Impact assessment
Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Page 10 Page 11 Page 12 Page 13 Page 14Made with FlippingBook flipbook maker