ANZCTR - Registration

The safety and scientific validity of this study is the responsibility of the study sponsor and investigators. Listing a study does not mean it has been endorsed by the ANZCTR. Before participating in a study, talk to your health care provider and refer to this information for consumers

Trial registered on ANZCTR

Registration number

ACTRN12617000017325

Ethics application status

Approved

Date submitted

16/12/2016

Date registered

5/01/2017

Date last updated

22/12/2021

Date data sharing statement initially provided

23/07/2019

Type of registration

Retrospectively registered

Titles & IDs

Public title

Evaluation of NetworkZ: Can multidisciplinary team-training improve the safety of surgery in New Zealand?

Scientific title

Evaluation of the NetworkZ programme: Can multidisciplinary surgical simulation team-training improve teamwork processes and the safety of surgery in New Zealand?

Secondary ID [1]

Nil known

Universal Trial Number (UTN)

U1111-1189-3992

Trial acronym

NetworkZ

Linked study record

Health condition

Health condition(s) or problem(s) studied:

Preventable surgical harm and death

Condition category

Condition code

Surgery

Other surgery

Anaesthesiology

Other anaesthesiology

Intervention/exposure

Study type

Interventional

Description of intervention(s) / exposure

NetworkZ is team-training intervention designed to improve information sharing between team members, improve communication strategies and develop mutual respect and trust between Operating Room (OR) staff.
The educational framework used is the Salas Model of effective teams. Whole operating room teams participate in immersive, simulated surgical cases where the challenges to teamwork and communication are amplified in order to demand optimal processes and expose deficiencies. Each simulation is followed by a structured debrief, led by trained and accredited simulation instructors, exploring reactions, clarifying scenario details to ensure a shared understanding of what happened, exploring what actually happened and why, and drawing lessons on communication and teamwork to apply to clinical practice. The debrief lasts approximately 45 minutes. Simulation based training is supported by communication skills training comprising four separate 30 minute workshops on the principles of teamwork and shared mental model; speaking up; closed loop communication; and structured recap. Workshops include videos, slide presentations and small group discussion. The communication skills training is delivered by trained and accredited NetworkZ instructors at the DHB.
The simulator used is a 3G Laerdal Simman, set in a simulated or real operating room environment, and integrated with realistic custom-designed surgical models of trauma or pathology requiring a specific clinical response. All patient notes, laboratory investigations, equipment, drugs, fluids and consumables such as surgical drapes, gowns, gloves and masks will be provided as in a real theatre case.
The simulated cases will be modelled on real life cases, and include up to 25 different surgical scenarios in the disciplines of general surgery, ENT/ORL, urology, plastics and orthopaedics. A full team training intervention comprises 1 full day of simulations, debriefs and discussion. In each day, there will be 4-5 simulations and debriefs.
Simulations will be chosen from the 25 available scenarios based on the participants attending the session. As nursing, technician and anaesthesia staff work across all of the surgical disciplines portrayed in the scenarios. choice of scenario will be based on the discipline of the surgeon or surgeons attending any particular course, also ensuring that participants have not previously been exposed to that scenario.
The intervention will be delivered at each District Health Board (DHB) in New Zealand and may be run either in existing operating theatres or in a purpose built simulation facility. The intervention will be progressively rolled out across all 20 DHBs over a 5 year time frame, in four cohorts with five DHBs in each. Cohorts are matched for DHB size.
Cohort 1 training commenced on 3 March 2017. Cohort 2, 3, and 4 training will commence at 12 month intervals between 2018 and 2020. The intended start date for each new cohort is 1 February. In practice there may some delays in the start date in each cohort.
Delivery of the intervention is by trained, accredited instructors. The Project team will deliver the initial courses at each DHB and train local staff as instructors through a structured training programme with ongoing monitoring and support. Course quality will be monitored and assured through a central accreditation process, instructor networks and instructor exchange.
The NetworkZ team training days will be ongoing on a regular basis in each DHB, with an expectation that all OR staff will participate in a full course at least once every five years. We aim to train the majority of NZ OR staff during the course of this project and estimate that this will be over 6000 operating theatre staff.
The intervention will run from 3 March 2017, beginning in Cohort 1, and will continue until the training is running in all four Cohorts (i.e. all DHBs in New Zealand).
The pre-intervention period will be defined as the period of 12 months prior to the commencement of training in that Cohort. The implementation period will be 15 months following the commencement of training in that Cohort. The post-intervention period will be 12 months following the implementation period in that Cohort.
Our primary outcome measure, Days Alive and out of Hospital will be measure at 90 days post surgery for all patients undergoing surgery in New Zealand public hospitals. The measurement will be from 12 months prior to the intervention in Cohort 1 to 12 months after the implementation period in Cohort 4 (plus 90 days for surgical case data). Because of the 90 day post-surgery follow-up we will continue to collect data for each DHB for 90 days after the measurement period has ended

Intervention code [1]

Other interventions

Comparator / control treatment

We are using a stepped wedge cluster design with staged roll-out of the intervention across the 20 DHBs in New Zealand in four cohorts at 12 month intervals, over a period of 4 years and including a transition phase (implementation period) of 15 months.
Using this design, the DHBs will act as their own controls as the intervention progressively rolls out across cohorts.

Control group

Active

Outcomes

Primary outcome [1]

Days Alive and Out of Hospital at 90 Days. This data is downloaded from a national data base. All hospital admissions, discharges and readmissions are entered into the National Minimum Data Set (NMDS) as well as complete details of patient identity, hospital, diagnosis, patient co-morbidities, procedures undertaken. In addition, mortality is entered into another national database which can be linked to the NMDS. The files of all patients undergoing the surgical procedures of interest will be provided to the research team by the Ministry of Health, from which we will be able to calculate Days Alive and Out of Hospital.

Timepoint [1]

90 days post surgery for all included surgical cases from 12 months prior to the start of training in Cohort 1 up until 12 months after the implementation period of the intervention in Cohort 4.

We also intend to undertake an interim analysis of the impact of the programme on Days Alive and Out of Hospital (DAOH) scores, based on data available at 17 March 2020. The interim analysis will inform funding applications and future programme planning. The outcome of the interim analysis will not alter the conduct of the trial or the final analysis. The final analysis will continue to follow the methodology described here and in the protocol publication in BMJ Open. The final analysis of the impact of the programme will be conducted in 2022, 12 months after the transition period for the final cohort (Cohort 4), as outlined in the original protocol description.

Primary outcome [2]

Days out of hospital at 90 days post-surgery calculated from the NMDS.

Timepoint [2]

90 days post surgery for all included surgical cases from 12 months prior to the start of training in Cohort 1 up until 12 months after the implementation period of the intervention in Cohort 4.

We also intend to undertake an interim analysis of the impact of the programme on Days Out of Hospital (DOH) scores, based on data available at 17 March 2020. The interim analysis will inform funding applications and future programme planning. The outcome of the interim analysis will not alter the conduct of the trial or the final analysis. The final analysis will continue to follow the methodology described here and in the protocol publication in BMJ Open. The final analysis of the impact of the programme will be conducted in 2022, 12 months after the transition period for the final cohort (Cohort 4), as outlined in the original protocol description.

Primary outcome [3]

Mortality and days alive post-surgery calculated from data from the Ministry of Health Mortality Data Set, linked through patient National Health Index, to the National Minimum Data Set.

Timepoint [3]

90 days post surgery for all included surgical cases from 12 months prior to the start of training in Cohort 1 up until 12 months after the implementation period of the intervention in Cohort 4.

We also intend to undertake an interim analysis of the impact of the programme on days alive scores, based on data available at 17 March 2020. The interim analysis will inform funding applications and future programme planning. The outcome of the interim analysis will not alter the conduct of the trial or the final analysis. The final analysis will continue to follow the methodology described here and in the protocol publication in BMJ Open. The final analysis of the impact of the programme will be conducted in 2022, 12 months after the transition period for the final cohort (Cohort 4), as outlined in the original protocol description.

Secondary outcome [1]

Post-operative complications recorded in the National Minimum Data Set

Timepoint [1]

90 days post surgery for all included surgical cases from 12 months prior to the start of training in Cohort 1 up until 12 months after the implementation period of the intervention in Cohort 4. Data on postoperative complications is entered by coders at DHB level into the NMDS. This will be measured from day 1 post surgery to 90 days post-surgery.

We also intend to undertake an interim analysis of the impact of the programme on days alive scores, based on data available at 17 March 2020. The interim analysis will inform funding applications and future programme planning. The outcome of the interim analysis will not alter the conduct of the trial or the final analysis. The final analysis will continue to follow the methodology described here and in the protocol publication in BMJ Open. The final analysis of the impact of the programme will be conducted in 2022, 12 months after the transition period for the final cohort (Cohort 4), as outlined in the original protocol description.

Secondary outcome [2]

This is a composite secondary outcome of Accident Compensation Corporation (ACC) treatment injury claims including volume, cost, and categorisation from ACC claims data base.

Timepoint [2]

Measured continuously from one year to the start of training on Cohort 1, to one year after the end of faculty assisted training in Cohort 4. This covers the period for all included claims from 12 months prior to the start of training in Cohort 1 up until 12 months after the implementation period of the intervention in Cohort 4.

We also intend to undertake an interim analysis of this data in mid 2020.

Secondary outcome [3]

Change in scores in a custom-designed teamwork processes survey.

Timepoint [3]

A pre-measure in each cohort, and post-measure 12-15 months after the start of the intervention

We also intend to run an interim analysis of the completed Cohort 1 and 2 Cohorts based on data available at mid March 2020.

Secondary outcome [4]

Change in safety culture score measured by the Surgical Safety Culture Survey

Timepoint [4]

This survey is being collected at a national scale in 2017, 2019 and 2021.

We intend to run an interim analysis comparing 2017 and 2019 data.

Secondary outcome [5]

Change in Quality of administration of the WHO Safe Surgery Checklist, measured by World Health Organisation Behaviourally Anchored Rating Scale (WHOBARS).

Timepoint [5]

A pre-intervention observational measure in each cohort up to eight months before the start of training; and a post-intervention observational measure in each cohort about 12-15 months after the start of training in each DHB in the cohort.

We intend to run an interim analysis of the completed Cohort 1 and 2 Cohorts based on the data available at mid March 2020.

Secondary outcome [6]

Organisational implementation process (qualitative exploration and understanding)

Timepoint [6]

Data collected from each DHB during the period from five to twelve months after the start of training.

Secondary outcome [7]

NetworkZ effectiveness and application to clinical practice (qualitative exploration and understanding)

Timepoint [7]

OR staff interview data, collected from each DHB during the period from three to six months post the start of training.

Secondary outcome [8]

Uptake within the DHBs measured as the number and proportion of relevant staff who have received the training (adjusted by the number of trained staff transferred in and out of the DHB).

Timepoint [8]

From 12 months prior to the start of training in Cohort 1 up until 12 months after the implementation period of the intervention in Cohort 4.

Eligibility

Key inclusion criteria

Eligibility for the training and staff surveys and observations: All staff working in ORs to provide general, orthopaedic, urological, plastics, and ENT/ORL surgery in NZ. The main staff groups who will be included are: surgeons, surgical trainees, anaesthetists, anaesthetic trainees, anaesthetic technicians, anaesthetic technician trainees, and theatre nurses. We will also include some staff who have a managerial interest in the implementation of NetworkZ training in their DHB, including directors of surgery, directors of nursing, and quality managers.
Eligibility for patient outcome measure: all patients having surgical procedures with anaesthesia that are listed in the procedural codes of the Health Quality and Safety Commission (HQSC) Perioperative Mortality Review Committee (PMRC), involve the previously defined surgical disciplines and are operated on in a public hospital operating room in New Zealand

Minimum age

No limit

Maximum age

No limit

Sex

Both males and females

Can healthy volunteers participate?

Yes

Key exclusion criteria

Exclusion criteria for training and evaluation: Some junior medical staff that have not commenced specialised training in anaesthesia or surgery, such as house officers, may be present in the teams that we observe and rate, but we will not seek to gather survey information from them in relation to our two staff safety culture surveys. The same will apply to Health Care Assistants, who may or may not be involved in OR work.
Exclusion criteria for surgical cases: minor procedures (i.e. patients who had their operation under local infiltration only).

Study design

Purpose of the study

Educational / counselling / training

Allocation to intervention

Randomised controlled trial

Procedure for enrolling a subject and allocating the treatment (allocation concealment procedures)

The allocation of DHBs to Cohorts was as follows: The DHBs were allocated to five groups according to population. The first Cohort of five DHBs was selected based on convenience and to ensure inclusion of DHBs of appropriate sizes and geographical distribution. The remaining DHBs were randomly allocated to Cohorts 2, 3 and 4 using a stratified allocation to ensure each Cohort contained the same distribution of large to small DHBs. Due to the pragmatic nature of quality improvement interventions, DHBs unable to participate in the training at their allocated time are able to swap with a DHB of a similar size in a subsequent Cohort.
Methods used to generate the sequence in which subjects will be randomised (sequence generation)
Randomised number generator in Excel for DHB allocation.

Methods used to generate the sequence in which subjects will be randomised (sequence generation)

Randomised number generator in Excel for DHB allocation

Masking / blinding

Open (masking not used)

Who is / are masked / blinded?

Intervention assignment

Other

Other design features

Stepped Wedge Cluster design

Phase

Not Applicable

Type of endpoint/s

Safety/efficacy

Statistical methods / analysis

The Primary Outcome measure (DAOH at 90 days), will be calculated from the National Minimum Data Set and the Mortality Data set.
For all patients alive at 90 days, the DAOH score is 90 minus the number of days in hospital from the date of the surgical procedure. Any part of a day in hospital will count as one day. The maximum score for any patient included in the study is 89 because the day of the procedure will count as one day in hospital. A death prior to leaving hospital will count as zero. A death within the 90 day period when the patient was discharged home from hospital will count as the number of days out of hospital during the 90 day period.
A second operation within the 90 day period will be treated as the same event with the exception of any situation where the first operation is a minor diagnostic procedure leading directly to the definitive procedure (e.g. biopsy under GA confirming Ca, followed by definitive surgery within the 90 day period).
We will include only patients in the surgical disciplines receiving the intervention, patients undergoing regional or general anaesthesia, and patients receiving an intervention listed as relevant to the concerns of the HQSC’s PMRC.

The study design will be the transition form of the stepped wedge cluster design [1}, where the transition phase will be a 15-month period of implementation of the intervention following the start of training. The total period of data collection for the DAOH analysis will be from 12 months prior to the first training day in the first DHB in Cohort 1 until 12 months after the end of the implementation period in Cohort 4. Data from the implementation (transition) phase will not be used in the analysis given that it cannot be assigned to either intervention or control arms.
Patients’ mean DAOH scores will be classified into pre-intervention (control) and post-intervention (intervention) for analysis.

We will use a non-parametric Mann Whitney rank sum tests as our primary test for an overall difference between distributions of DAOH90 before and after implementation of NetworkZ, specifying p <0.05 as significant. We will then perform quantile regression at the 0.1, 0,25, 0.75, 0.9 quantile to characterize any difference between the two distributions. This quantile regression will include time (month and year of procedure), cohort, DHB (within cohort) and intervention as factors in the model.

We will conduct a similar comparative analysis for Days Out of Hospital (for survivors); for post-surgery mortality; a defined set of post-surgery complications (for example wound infection, renal impairment, thrombotic event).

We will use a similar approach to the analysis of the ACC claims data base for treatment injury claims (number and costs).

[1] Hemming, K., et al., The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ : British Medical Journal, 2015. 350.

The teamwork perceptions surveys will be provided to participants with identifiable numbers, which will enable us to conduct paired t-tests of pre- and post-intervention survey responses from the same individuals.

For both the SSCS and teamwork perceptions surveys we will establish the "dose effect" of intervention by gathering and analysing data on which staff have received the training.

For each cohort we will analyse observational data collected via the WHOBARS tool. We will then conduct pre- and post-intervention t-tests on the WHOBARS scores. Because our observational and training records will enable us to identify the proportion of observed staff, and OR staff in each DHB, who have been trained. We will be able to investigate the relationship between these variables (i.e. the dose) and WHOBARS scores (the effect).

Quantitative measures will be compared between pre- and post-rollout phases.

Qualitative data from the interviews will undergo framework analysis to identify barriers and enablers to the implementation of NetworkZ training by DHBs, and improved communication by OR staff. The framework for this analysis will be the Organising for Quality model [2]. The themes identified will be analysed at the level of entire cohorts.

[2] Bate, P,. et al., Organising for Quality: The Improvement Journeys of Leading Hospitals in Europe and the United States. Oxford & New York, Radcliffe Publishing, 2008.

Recruitment

Recruitment status

Recruiting

Date of first participant enrolment

Anticipated

Actual

18/11/2016

Date of last participant enrolment

Anticipated

Actual

Date of last data collection

Anticipated

31/07/2022

Actual

Sample size

Target

6000

Accrual to date

1800

Final

Recruitment outside Australia

Country [1]

New Zealand

State/province [1]

nationwide

Funding & Sponsors

Funding source category [1]

Government body

Name [1]

Accident Compensation Corporation

Address [1]

PO Box 242
Wellington 6140

Country [1]

New Zealand

Funding source category [2]

Government body

Name [2]

Lotteries Health Research Fund New Zealand

Address [2]

The Department of Internal Affairs
Community Operations 147 Lambton Quay
Wellington
Post code 6011

Country [2]

New Zealand

Funding source category [3]

Other Collaborative groups

Name [3]

Australia and New Zealand College of Anaesthetists

Address [3]

PO Box 6095,
Melbourne,
Victoria 3004,

Country [3]

Australia

Primary sponsor type

Individual

Name

Professor Jennifer Weller

Address

Centre for Medical and Health Sciences Education
School of Medicine
The University of Auckland
Private Bag 92019
Auckland 1142

Country

New Zealand

Secondary sponsor category [1]

None

Name [1]

Address [1]

Country [1]

Ethics approval

Ethics application status

Approved

Ethics committee name [1]

Health and Disability Ethics Committees

Ethics committee address [1]

Ministry of Health
PO Box 5013
Wellington 6011

Ethics committee country [1]

New Zealand

Date submitted for ethics approval [1]

15/08/2016

Approval date [1]

03/10/2016

Ethics approval number [1]

16/NTB/143

Summary

Brief summary

This project evaluates the effectiveness of the NetworkZ training intervention for operating room (OR) staff in all DHBs in New Zealand (NZ). The aim of the intervention is to improve outcomes for surgical patients, and improve teamwork in the OR.
Our primary hypothesis is that NetworkZ will lead to improved scores for Days Alive and out of Hospital (DAOH) at 90 days. The NetworkZ training is for staff in general, orthopaedic, urological, plastics, and ENT/ORL surgery in all DHBs in New Zealand. Each training session will be provided by accredited instructors to multidisciplinary teams of about 10 staff. The training comprises realistic simulations of challenging surgical cases, debriefs of the cases, and discussions of specific teamwork and communication strategies. The training focusses on shared mental models, structured communication, and mutual respect and trust.
The NetworkZ training will be rolled out through all NZ DHBs in four cohorts of five DHBs over a period of five years. We will conduct a concurrent evaluation throughout using a stepped wedge cluster research design. This design will allow us to compare pre- and post-intervention cohorts, and enhance our capacity to separate training effects from long term trends and changes in the DHBs.
Staff interviews will seek to identify factors that assist or inhibit the implementation of NetworkZ training in the different DHBs.
Primary Outcome Measure:
Days Alive and Out of Hospital (at 90 days). This looks at patients who are alive AND the number of days they are out of hospital, in the 90 days following surgery. We will also analyse mortality, and days out of hospital separately. The data for this analysis will be drawn from the NMDS.
Secondary Outcome Measures:
Post-surgical complications from the NMDS.Number and costs of post-surgical treatment injury claims, from ACC data base.
Safety Climate pre-post intervention survey using the Surgical Culture Safety Survey developed and validated by the Harvard School of Public Health.
Teamwork perceptions pre-post intervention survey, using a tool we have custom-designed to measure NetworkZ training processes.
The quality of implementation of the Safe Surgery Checklist pre-post intervention, measured by the WHOBARS observation instrument, which we have previously developed and validated.
The number and proportion of OR staff trained in the DHBs.
Interview data exploring the organisational implementation process in the DHBs and how staff apply the training in their OR work. (Framework Analysis).

Trial website

www.networkz.ac.nz

Trial related presentations / publications

Public notes

Publications arising from the NetworkZ pilot study of 20 courses in two DHBs are listed below (Note: The NetworkZ programme was originally known as MORSim (Multidisciplinary Operating Room Simulation)),
1. Weller JM, Civil I, Torrie J, Cumin D, Garden A, Corter A, Merry AF. Can team training make surgery safer? Lessons for national implementation of a simulation-based programme. New Zealand Medical Journal 2016. October 14th. Volume 129 Number 1443
2. Weller J et al. Improved scores for observed teamwork in the clinical environment following a multidisciplinary operating room simulation intervention. New Zealand Medical Journal 5 August 2016, Vol 129 No 1439 ISSN 1175-8716
3. Cumin, D., Skilton, C., & Weller, J. (2016). Information transfer in multidisciplinary operating room teams: a simulation-based observational study. BMJ Quality & Safety. 2016. doi:10.1136/bmjqs-2015-005130
4. Weller, J., Cumin, D., Torrie, J., Boyd, M., Civil, I., Madell, D., . . . Merry, A. F. (2015). Multidisciplinary operating room simulation-based team training to reduce treatment errors: a feasibility study in New Zealand hospitals. The New Zealand Medical Journal, 128(1418), 40-51.
5. Nakarada-Kordic, I., Weller, J., Webster, C., Cumin, D., Frampton, C., Boyd, M., Merry, A.F. Assessing the similarity of mental models in operating room team members and implications for patient safety: a prospective, replicated study. BMC Medical Education 2016. 16 (1), 229

Attachments [1]

/AnzctrAttachments/371766-HDEC Letter 16NTB143 Approved FULL Application.pdf (Ethics approval)

Attachments [2]

/AnzctrAttachments/371766-Participant Information Sheet_v5.pdf (Participant information/consent)

Attachments [3]

/AnzctrAttachments/371766-Email invitation and consent.pdf (Participant information/consent)

Contacts

Principal investigator

Name

Prof Jennifer Weller

Address

Auckland City Hospital, 2 Park Road, Grafton (Support Building 599, Level 12)
Private Bag 92 019
Auckland 1142

Country

New Zealand

Phone

+64 9 923 9459

Fax

+64 9 373 7641

[email protected]

Contact person for public queries

Name

Prof Jennifer Weller

Address

Auckland City Hospital, 2 Park Road, Grafton (Support Building 599, Level 12)
Private Bag 92 019
Auckland 1142

Country

New Zealand

Phone

+64 9 923 9459

Fax

+64 9 373 7641

[email protected]

Contact person for scientific queries

Name

Prof Jennifer Weller

Address

Auckland City Hospital, 2 Park Road, Grafton (Support Building 599, Level 12)
Private Bag 92 019
Auckland 1142

Country

New Zealand

Phone

+64 9 923 9459

Fax

+64 9 373 7641

[email protected]

Data sharing statement

Will individual participant data (IPD) for this trial be available (including data dictionaries)?

No/undecided IPD sharing reason/comment

The primary outcomes are based on hospital admissions data collected by a third party. The researchers working on this study do not have authorisation to share this data with other parties.

What supporting documents are/will be available?

Doc. No.	Type	Citation	Link	Email	Other Details	Attachment
7898	Study protocol	Weller J, Long JA, Beaver P, et al. Evaluation of the effect of multidisciplinary simulation-based team training on patients, staff and organisations: protocol for a stepped-wedge cluster-mixed methods study of a national, insurer-funded initiative for surgical teams in New Zealand public hospitals. BMJ Open 2020;10:e032997. doi: 10.1136/bmjopen-2019-032997

Results publications and other study-related documents

Documents added manually

No documents have been uploaded by study researchers.

Documents added automatically

Source	Title	Year of Publication	DOI
Embase	Evaluation of the effect of multidisciplinary simulation-based team training on patients, staff and organisations: Protocol for a stepped-wedge cluster-mixed methods study of a national, insurer-funded initiative for surgical teams in New Zealand public hospitals.	2020	https://dx.doi.org/10.1136/bmjopen-2019-032997
Embase	Towards a safer culture: implementing multidisciplinary simulation-based team training in New Zealand operating theatres - a framework analysis.	2019	https://dx.doi.org/10.1136/bmjopen-2018-027122

N.B. These documents automatically identified may not have been verified by the study sponsor.

Trial Review

Documents added manually

Documents added automatically