Background

The Australian Council for Educational Research (ACER) was engaged by the RANZCP Board in July 2019 to conduct an independent review of its examinations and associated processes. The Board accepted ACER’s report (January 2020) and the recommendation to decouple the then Essay-style examination into its two components and offer them as separate independent examination – Modified Examination Question (MEQ) examination and Critical Essay Question (CEQ) examination. The RANZCP Board further requested that the role and the format of the CEQ examination be reviewed to consider its effectiveness and fitness for purpose. 

RANZCP consultation process

In 2021-2022, RANZCP conducted a comprehensive consultation process with numerous relevant stakeholders and committees, including trainees regarding the format and the function of the CEQ assessment in the RANZCP Program. This consultation process showed that stakeholders’ views regarding the function of the CEQ and the assessment format were diverse and varied. 

The views of the Board and other stakeholders who participated in the consultation support that the skills assessed by the CEQ are critical and should be retained. However, the format of the assessment should be reviewed and modified. The Education Committee at its meeting on 
 9 December 2022 agreed that the set of skills and competencies examined by the CEQ should continue to be promoted and assessed but in ways that can be considered in the context of more contemporary assessment methodology and in an integrated format with future workplace-based program of assessment activities, while still addressing the burden of assessment for candidates and assessors.

ACER engagement

The Australian Council for Educational Research (ACER) has been formally engaged by the College in May 2023 to review the findings from the extensive consultation with stakeholders regarding the format and the function of the CEQ.

Considering the diversity of strong views of the relevant stakeholders, it was difficult for the Education Committee to reach a consensus and an objective outcome in relation to the format of the CEQ. To maintain the continuity of the project that ACER initially undertook with the RANZCP in 2019, the College has taken the initiative to appoint the ACER as an independent entity to provide an external review of the CEQ examination and its role and provide a set of recommendations for the way forward. 

The intended purpose of the CEQ is to assess written, analytical, integrative, critical and reflective thinking skills. RANZCP is interested in exploring other formats for assessing these important skills.  This move aligns with shifts in contemporary medical specialist training and assessment, where varied assessment formats are utilised to provide longitudinal assessment data and evidence of learning, more integrated with work-based training and as part of a whole program of assessments. 

As part of the review, RANZCP is also seeking guidance from ACER in exploring approaches to improving the quality of the MEQ examination. 

Scope of the ACER review

Broadly the scope of the ACER review will be:

• Evaluate the 2022 consultation findings and feedback received by the RANZCP from the relevant committees, trainees and other relevant Fellow stakeholders. 
• Provide a clear re-articulation of the Board-confirmed function (content and the skills) being assessed through the current CEQ format and provide an assessment of the current CEQ format in the context of the contemporary assessment environment (including consideration of an online format) and its fitness for purpose. 
• Review the alignment of the RANZCP curriculum learning outcomes and learning activities focused on the development of these skills (with consideration to timing in the program) with the CEQ assessment. 
• Identify contemporary directions for assessing the skills assessed by the CEQ, making reference to how these skills are assessed by other Specialist Medical Colleges and educational institutions in Australia and internationally (where publicly available). Such comparisons will particularly focus on psychiatry and in tertiary institution assessments where essay writing/critical review are assessed. 
• Conduct consultations with stakeholder representatives, to seek further information and gather evidence-based data around the format of the CEQ. 
• Identify any relevant alternative assessment formats that are fit for purpose for assessing the critical thinking skills assessed by the CEQ. This will include a couple of concrete examples of the relevant assessment formats/tools that could be used instead of the CEQ or alternatively suggested modifications of the current essay format for the CEQ. 
• Develop recommendations for possible assessment formats that can evaluate the skills with consideration to: 
  1. concerns in relation to the ‘burden of assessments’
  2. the transition by the RANZCP from a single high-stakes assessment to a more integrated, holistic, longitudinal Program of Assessments, based on contemporary assessment trends and the directions being encouraged by the Australian Medical Council
  3. feasibility of implementation of the recommended assessment format(s)
  4. logistics issues. 

In relation to the MEQ examination, ACER will:

• Review the MEQ assessment with a view to quality improvement for this examination and evaluate the probable causes of low MEQ pass rates in the past few years despite the decoupling of the Essay- style exam in 2021 (in line with the ACER recommendations in its January 2020 report). 
• Provide advice in relation to the suitability to offer the MEQ online and what challenges this may entail.

Timelines

After the preliminary evaluation of all resources including the background papers on the CEQ Review and the consultation process, the ACER plans to provide an Interim Report by the end of May 2023 in line with the project scope.

Further to this preliminary evaluation, the ACER will conduct consultations with the nominated individuals to identify perceived key strengths and weaknesses of the CEQ format, and to gather insights about potential new formats for the CEQ. These are planned for early June 2023.

The ACER will also provide the RANZCP with some sample assessment formats for review and broad rubric criteria.

The ACER is committed to ensure that ultimately any recommendations they provide in relation to written examinations are sound, justified and evidence based.

The final report and recommendations from the ACER is expected in early August 2023. Any potential transitional implications stemming out of the recommendations will require a lead-time for the formal implementation of any approved outcomes.