On Friday 28 April, the College held an Assessments Stakeholder Forum, with the attendance of 38 members, to discuss our future program of assessments, the OSCE and its alternatives, and to reflect on the recently received member feedback. Participants included representatives from Education, Examinations and Training Committees, trainee representatives, supervisors, Directors of Training, Branch Committees, members of the Board, staff, and the Chief Executive.

Concerns raised through the petitions and correspondence received by the College in relation to the removal of OSCEs were discussed and various views were expressed on the assessment rigour, role of the OSCEs, challenges of integration of some summative assessments into the workplace environment, and the emerging role of the supervisors as assessors. Mindful of the concerns from some member groups, the complexities underpinning the success of the new program of assessments and its implementation need to be addressed through an agreed communication strategy, development of supervisor supports and engagement processes, and in the context of modern Education theory.  

Professor Lambert Schuwirth challenged our thinking regarding the educational value of single high stakes assessments including OSCEs, highlighting the developments in assessment over the last 20 years where progression decisions have been based on assessment from multiple points and longitudinal information up to that point, that constitutes a combination of workplace-based and centrally administered assessments.

Participants reaffirmed the core underlying principles (formed at the Stakeholder Forum of 3 February):

• Educational value and patient safety (which will be separated into 2 individual principles)
• Use of both evidence and expert opinion in making outcome determinations 
• Feasibility 
• Well-being and support for all stakeholders
• Competence as a narrative 
• Longitudinal assessment 
• Front-loading 
• Concepts of fairness for trainees and SIMGs
• Acknowledge major change implications 

And, further to a discussion, added the following:

• Developmental perspective (descriptors)
• Continual lifelong learning
• Evaluation/ QA of the assessment process
• Acceptability and credibility
• Transparency

In the afternoon, participants debated the pros and cons of various options and possible models of assessment (as presented in the papers for the SHF), including maintaining the OSCEs. Support was provided for a move towards more holistic and programmatic models which ensures that candidates are assessed on core psychiatry skills, where the assessments are used to inform progression judgements and where the program also includes summative assessments of clinical skills independent of a trainee’s supervisor. 

Consensus was reached that the OSCE will not form part of the future assessment strategy.

Based on the evaluation of Options’ Pros & Cons and the discussions, two options were short-listed for potential development of the Program of Assessments, with a clear direction for the conclusion of the OSCEs. 

Selected Options

(note these are yet to be further conceptualized):

O3: Modified CCA format – Clinical Competency Portfolio Review (CCPR) - proposed

Some of the types of assessments that may form part of this option are:

• Portfolio Review (PR) retained as per CCA with same criteria 
• Additional data points for the PR assessment 
• One Stage 2 Observed Clinical Activity (OCA) – assessed by an external supervisor
• Two Stage 3 OCAs – one assessed by an internal supervisor/second by an external supervisor (SOCA)

O4: Integrated Assessment Pathway (WBA) – proposed

• Integrates workplace-based assessments with central summative assessments
• Multiple data points to consider by progression panels for progression decisions
• Multiple points data enabling consistent and programmatic assessment decisions 
• Consideration of existing assessments of OCAs and extending this to SOCAs, and EPAs and ITAs holistically – the focus on aggregated assessments progression judgements
• Possibility of introducing some different types of assessments in the workplace to target competencies not generally assessed elsewhere
• Different progression decision points for implementation

While there was a demonstrated support for these options, it was evident that there is a need to address challenges related to supervisor buy-in and role, calibration, and external assessor input. While work is progressing to develop a detailed articulation and program design of the two options selected in relation to assessment activities - summative and formative assessment, calibration and delivery, alignment to the assessment framework and curriculum. 

A more detailed proposal will be developed for further consultation and feedback, through attendance at the various Branch and stakeholder groups meetings, and another Stakeholder Forum (SHF) planned in the second half of the year. In the meantime, while this significant body of work is undertaken, we encourage members and trainees to continue to engage with their representatives to relay the discussion outcomes from the SHF and provide feedback on the direction agreed on 28 April to shape our competency-based program. 

Meanwhile, we are developing an engagement strategy, with the aim to provide relevant communications and socialize the outcomes of the Stakeholder Forum to all College stakeholder groups, committees, and Branches across Australia and NZ. We are building a website portal to maintain documents, articles, and development updates to keep members informed and establish an effective reach, as well as planning to do virtual and/or face-to-face ‘roadshow’ with various stakeholder cohorts. 

While we are working towards a feasible, acceptable, and robust model of assessment that meets the specialist competencies, it is important to assure our members that a final decision and model are yet to be determined and agreed upon, as we continue to work alongside our committees and members to shape our future assessment strategy.