Case Study

Four Steps to retrain a colleague in Difficulty

In 2023, a colorectal unit in England, with Trust and Lapco support, designed a re-training programme to help a colleague (the subject) whose laparoscopic colorectal practice had been suspended.

Key steps of the process:

1. We delivered a Lapco Train the Trainer course to six colorectal consultants, including the subject, to help elevate and standardise their training.

2. Training of the subject by his colleagues in the OR was monitored with the Lapco formative assessment system.

3. Post completion of training, the subject submitted 2 videos of procedures which were analysed with the Lapco Summative Assessment system.

4. The subject, with Trust agreement and Lapco sign-off, continued with independent laparoscopic colorectal surgery.

Formative Assessment was used To Determine Independence

Education assessment is a means to objectively determine progress in training. The Formative Assessment tool helps surgeons and trainees objectively measure performance during a mentored surgical training procedure. A series of assessments (followed by summative review) can determine ability for a trainee to perform a procedure independently.

The Delphi Method was used to construct this form, a systematic stepwise process that uses a panel of experts in the subject to reach a consensus to develop professional guidelines.

Each key step of the surgical procedure e.g. ‘setup’, ‘exposure’, ‘reconstruction’ is graded from 1-6 (see below) to measure the amount of support the trainee received from their mentor during the case to ascertain their competence.

•Grade 1 = Not performed, step had to be done by trainer

•Grade 2 = Partly performed, step had to be partly done by trainer

•Grade 3 = Performed, with substantial verbal support

•Grade 4 = Performed, with minor verbal support

•Grade 5 = Competent performance safe without guidance

•Grade 6 = Proficient, could not be done better

Both the trainee and trainer independently ‘score’ trainee performance immediately after the procedure. Completion of the form also allows constructive performance enhancing feedback after the case and determines areas for improvement for the next case.

Example stages of a formative assessment form, a typical form is comprised of seven stages

As a surgeon’s time is precious, it takes less than 2 minutes to fill the form with a mobile smart device.

What is Competence in Surgery?

Competence:

Competence in surgery is a combination of independence, efficiency and performance with minimum errors.

Errors in surgery:

Inconsequential errors – minor tissue damage with little or no after effects e.g. electrosurgical damage to a non-critical structure. Consequential errors – tissue damage that, if not recognised and corrected, will result in an adverse outcome for the patient e.g. damage to bowel.

Time:

Balancing precision and speed in surgery is essential for achieving optimal outcomes. Focusing on task-specific training with timed practice rather than rushing through complete procedures can greatly enhance the effectiveness of surgical training.

Formative Assessment Outcome Chart

Anterior Resection Learning Curve- Formative Assessment

Formative educational assessment occurs during training. In surgery the role of the trainer is to bring the trainee to the point of competence progressively and safely, and with increasing independence. Hands on training in the OR can be prefaced or accompanied by forms of simulated training in the skills lab.

Anterior resection and right hemicolectomy learning curve

Anterior resection and right hemicolectomy are common colorectal procedures. Prior experience of laparoscopic surgery and open colorectal surgery will all influence the number of cases required in training to achieve competence. In each case (above) the subject achieved competence in 6-7 cases, as you would expect in this particular instance with their prior experience of laparoscopic and open colorectal surgery. Less experienced trainees, more cases would be typically required.

CUSUM learning curve chart to visualise competency

Proficiency gain curves using cumulative sum (CUSUM) charts are powerful tools for monitoring and improving skills over time. CUSUM charts are used to detect small shifts. They plot the cumulative sum of deviations from a target value. For proficiency gain, this target is an expected level of performance!

When multiple cases are recorded using the Lapco Learning Platform, a ‘proficiency gain or learning curve’ will be generated to demonstrate the trainee’s progression towards safe, competent and independent performance of a surgical procedure.

This system can be used when surgeons are being trained in a new technique or to use a new technology such as robotic surgery.

Once the trainee surgeon is consistently averaging scores of 5 or above, they have reached competency in that procedure and can be considered for ‘Summative Assessment’ using video analysis.

By way of example, this graph compares 3 trainees (yellow, red and blue). For each trainee, the broken lines represent scores uploaded by the trainees and the solid lines for scores from the trainers. This allows scores to be compered between trainers and trainees. Dr Pierre (blue) overestimates their level of competence. Dr Millie Joanna (yellow) and Mr Bob (red) both underestimate their progress.

Methodological BackgrounD

Content validity

Content validity examines the relevance of an assessment tool. For this, a selective literature search on operating techniques in LCS was performed using internet sources, educational videos, and books. Based on these sources, a generic task analysis for laparoscopic colorectal resections and a scoring system was created. The proposal was sent to members of the educational committee of the National Training Program, represented by 12 expert laparoscopic surgeons, and an educationalist, from 10 different centres in the United Kingdom. The draft was discussed and amended via two committee telephone conferences, and the final version was approved by all members.

Construct validity

Construct validity measures the ability of the assessment to measure the task. This was calculated by comparing four different experience groups. Trainees who performed more than 15 consecutive cases were extracted, and their individual data were split into four experience groups (cases 1–5, 6–10, 11–15, and Due to a lack of a ‘‘gold standard’’ method, it was not possible to establish criterion- related validity. Both trainees and trainers had to fill in the same scoring sheet independently, which facilitated the assessment of interrater reliability.

Transitioning from Formative to Summative Assessment

As reflected in graphs above, the subject’s curve started to flatten for both procedures. Following consultation between the Trust, the subject and Lapco, the subject performed two separate cases independently which were submitted for summative assessment.

Summative Assessment determines the level of competence to perform a surgical procedure

Lapco summative assessment is a video-based platform that helps to determine the trainee’s level of competence to perform a surgical procedure at the end of their formative training process.

With the Lapco Learning platform, trainees can upload videos to get independent and objective feedback from our experts using the Lapco summative assessment system.

Summative assessment uses structured and validated Lapco methodology to analyse generic and task-specific skills to result in a reliable score for surgical competence at the end of training

This can be followed by an online debriefing session to help the trainee to understand their performance during procedures.

Summative Assessment: four key criteria

Ergonomics:

Optimizing the surgical environment, tools, and techniques.

Accuracy:

The precision with which surgical procedures are performed, including the correct execution of techniques and adherence to surgical plans.

Safety:

Measures to protect both the patient and the surgical team.

Final Product:

The final product refers to the quality of patient outcomes and the overall effectiveness of surgical interventions.

Summative Assessment: Right Hemicolectomy and Anterior Resection

The above represents two of the four stages of one of the subject’s procedure scored using the summative assessment from. For the purpose of the subject’s retraining programme, right hemicolectomy and anterior resection (common colorectal procedures) were selected for independent assessment. In both cases, the subject received high ‘pass’ scores.

The Retraining Process was a success!

Retraining consultants in difficulty presents significant challenges. The Lapco method promotes a supportive team approach to help remedy technical and non-technical problems, and an objective framework to reliably observe the re-acquisition of skills in a colleague’s colorectal practice.

This model should be considered in situations where difficulties are identified in the practice of a colorectal surgeon.