ANZCTR - Registration

The safety and scientific validity of this study is the responsibility of the study sponsor and investigators. Listing a study does not mean it has been endorsed by the ANZCTR. Before participating in a study, talk to your health care provider and refer to this information for consumers

Trial registered on ANZCTR

Registration number

ACTRN12620000695909

Ethics application status

Approved

Date submitted

11/05/2020

Date registered

22/06/2020

Date last updated

7/12/2022

Date data sharing statement initially provided

22/06/2020

Date results information initially provided

7/12/2022

Type of registration

Prospectively registered

Titles & IDs

Public title

The assessment of experimental artificial intelligence (AI) algorithms for the diagnosis of skin tumours against human performance

Scientific title

The assessment of experimental artificial intelligence (AI) algorithms for the diagnosis of skin tumours against human performance

Secondary ID [1]

None

Universal Trial Number (UTN)

U1111-1251-8995

Trial acronym

Linked study record

Health condition

Health condition(s) or problem(s) studied:

Skin tumour diagnosis

skin cancer

melanoma

Condition category

Condition code

Skin

Dermatological conditions

Cancer

Malignant melanoma

Cancer

Non melanoma skin cancer

Intervention/exposure

Study type

Observational

Patient registry

False

Target follow-up duration

Target follow-up type

Description of intervention(s) / exposure

The condition observed is the diagnosis and management of skin tumours. The ground truth can take up to a year to be confirmed. Two groups of patients are recruited - 1. Those with baseline total body photographs taken 1-4 yrs prior to examination. Here, WHOLE BODY EXAMINATION occurs, and all discrete skin lesions are recruited with a > 4 mm longest diameter except for clinically non-suspicious amelanotic actinic keratoses and multiple lesions consistent with an inflammatory skin eruption or ephelides. In addition, all discrete lesions > 3 mm that are chosen for excision or monitoring by ground-truth assessment are recruited. 2. Patients undergoing routine excision or biopsy of pigmented skin lesions. Here, only these INDIVIDUAL LESIONS are examined.

Clinicians with a variety of clinical expertise (dermatology residents/registrars, specialists) will be recruited with their level of expertise (years using dermoscopy, clinician classification) recorded. One novice clinician (dermatology resident/1st yr registrar) and one expert clinician (specialist in pigmented lesion clinics) will examine the patient.

The diagnostic algorithms that will be COMPARED WITH THE CLINICIANS DIAGNOSIS/MANAGEMENT AND GROUND TRUTH are artificial intelligence-based that can be used on mobile phones. In our study, images are taken from the mobile phone of recruited patient lesions by the study researchers and analysed by cloud computing. The result is then returned to the researchers. Doctors and patients are not given the results. The output of these algorithms are Diagnosis (from 7 categories; 1. Melanoma 2. Melanocytic nevus 3. Pigmented Basal cell carcinoma 4. Pigmented Actinic keratosis / Bowen’s disease (intraepithelial carcinoma) 5. Benign (pigmented) keratotic lesion 6. Benign Vascular lesion 7. Dermatofibroma) and Management (dismiss, dermoscopy monitor or biopsy). However, the algorithms DO NOT influence patient management and are not revealed to the attending clinicians or patients.

The following AI algorithms from MetaOptima will be used to generate the above results
1. 7-class classification
* ISIC 2018 challenge winning model
* An ensemble of Taxnet models (standard backbone network with an RNN)
* An ensemble of Lesion Localization with Bias Unlearning models (LLBU)
2. Management decision
* An ensemble of Taxnet models (standard backbone network with an RNN) to get ben/mal predictions and map to Dismiss | Monitor | Biopsy classes
* An ensemble of Lesion Localization with Bias Unlearning models (LLBU) to get ben/mal predictions and map to Dismiss | Monitor | Biopsy classes

The Active control = ground truth. Ground truth is determined by a descending hierarchy of:
• Histopathology (ie. biopsy changed lesion).
• Unchanged lesions on TBP = benign
* Changed melanocytic lesions undergo 3-month dermoscopy monitoring (excise if change)
• Subsequent unchanged 3-month digital dermoscopy OR insignificant changing long-term monitored lesions = benign
• In vivo confocal microscopy (if available).
• 2 of 2 independent readers viewing the dermoscopy images of changed lesions that are clinically benign (eg. Seborrheic keratoses, hemangiomas) = benign

Intervention code [1]

Diagnosis / Prognosis

Comparator / control treatment

The AI diagnosis and management will be compared to Clinician diagnosis (7 categories: melanoma, nevus, dermatofibroma, actinic keratosis/Bowen's, benign vascular lesion, pigmented BCC, benign pigmented keratotic lesion) and management (dismiss, monitor or biopsy). The diagnostic methods used by the clinicians are at their discretion (ie. those used by them in routine practice).
The Active control = ground truth. Ground truth is determined by a descending hierarchy of:
• Histopathology (ie. biopsy changed lesion).
• Unchanged lesions on TBP = benign
* Changed melanocytic lesions undergo 3-month dermoscopy monitoring (excise if change)
• Subsequent unchanged 3-month digital dermoscopy OR insignificant changing long-term monitored lesions = benign
• In vivo confocal microscopy (if available).
• 2 of 2 independent readers viewing the dermoscopy images of changed lesions that are clinically benign (eg. Seborrheic keratoses, hemangiomas) = benign

As well the AI-clinician results will be compared to an online image based reader study that used the same AI algorithms but were compared in an experimental setting (Tschandl P et al. An open, international diagnostic study comparing the accuracy of humans and machines for skin lesion classification. Lancet Oncol. In press).

Control group

Active

Outcomes

Primary outcome [1]

1. Correct Management Decision (MD): Two definitions will be analysed.
I. Biopsy = Correct for ground truth malignant lesions (melanoma, pigmented BCC, pigmented AK/IEC, other pigmented malignant)
Monitor (3 months) or Dismiss (includes long-term monitoring) = correct MD for ground truth benign lesions (benign as per study flow chart, or naevus, benign keratosis, benign vascular, dermatofibroma, other pigmented benign)

II. Biopsy or Monitor (3 months) = Correct MD for ground truth malignant lesions (melanoma, pigmented BCC, pigmented AK/IEC, other pigmented malignant)
Dismiss (includes long-term monitoring) = Correct MD for ground truth benign lesions (benign as per study flow chart, or naevus, benign keratosis, haemangioma, dermatofibroma, other pigmented benign)

The correct MD will be determined by the ground truth procedure as previously described.

Timepoint [1]

Ground truth will be up to 12 months following AI measurement.

Primary outcome [2]

2. Correct Diagnostic Category: Categorical classification of 7 diagnoses (see previous).
The correct diagnostic category will be determined by the ground truth procedure as previously described.

Timepoint [2]

Ground truth will be up to 12 months following AI measurement.

Primary outcome [3]

3. Balanced multiclass accuracy: This will be according to the ISIC 2018 challenge analysis https://challenge2018.isic-archive.com/task3/ where the balanced diagnostic accuracy is defined as the average sensitivities obtained for each diagnostic category separately.

Timepoint [3]

Ground truth will be up to 12 months following AI measurement.

Secondary outcome [1]

Nil

Timepoint [1]

Nil

Eligibility

Key inclusion criteria

A. Whole body examination
1. Patients with baseline total body photographs (TBP) taken within 1-4 years of examination
2. Gender: Male or Female
3. Age range: 18-99yrs
4. Modified Fitzpatrick I-III Skin Type
5. Willingness and ability to provide informed consent and to participate and comply with the study requirements.

B. Individual lesion examination
1. Patients undergoing an excision/biopsy of a pigmented skin lesion.
2. Gender: Male or Female
3. Age range: 18-99yrs
4. Modified Fitzpatrick I-III Skin Type
5. Willingness and ability to provide informed consent and to participate and comply with the study requirements.

Minimum age

18 Years

Maximum age

99 Years

Sex

Both males and females

Can healthy volunteers participate?

Key exclusion criteria

Not meeting all inclusion criteria for either group A or B.

Study design

Purpose

Screening

Duration

Cross-sectional

Selection

Convenience sample

Timing

Prospective

Statistical methods / analysis

This first in human pivotal study is powered for the primary aim of comparing management and diagnosis of the clinician vs AI, according to published methods of equivalence and non-inferiority (Liu J et al. Stat Med 2002;21:231-45 Liu K et al. Stat Med 2004;23:545-59).
1) Summary Statements for comparing each test (AI and clinicians management) to Ground Truth – Equivalence test
A sample size of 3900 lesions (n=60 subjects with 65 lesions per subject) achieves 81% power at a 5% significance level using a two-sided equivalence test of correlated proportions when the standard proportion (proportion of a positive management decision) is 0.01, with the maximum allowable difference between these proportions that still results in equivalence (symmetrical equivalence limit) being 0.005 (0.5%).

2) Summary Statements for comparing the two test clinicians vs. AI management– Non-inferiority test
A sample of 3900 has more than 85% power to demonstrate the non-inferiority of AI algorithm for the diagnosis of pigmented skin lesions compared with the clinicians using a one-sided equivalence test of correlated proportions, an estimated standard proportion of 0.01 and a non in-inferiority margin of 0.005.

3) Summary Statements for comparing the two test clinicians vs AI diagnosis to ground truth
We have taken data from the reader study (Tschandl et al. Lancet Oncol) where there was an average of 7 lesions for every 30 that the best AI outperformed the experts (for diagnostic category 1-7). From this assumption, a sample size of 151 excised lesions achieves 80% power at a 5% significance level using a one-sided equivalence (non-inferiority) test of correlated proportions when the standard proportion is 0.730 (ie. estimating that 22/30 lesions for Human and AI have the same diagnosis), the maximum allowable difference between these proportions that still results in non-inferiority (the range of equivalence) is 0.10 (10%) ie. if the human diagnosis is less than 10% worse than the AI we classify the humans non-inferior to AI.
For the analysis we are planning to run Generalized Linear Mixed Models with two random effects: one random effect on the participants’ level (to account for variability within participant) and another random effect at assessment level (physicians including AI algorithm).

Recruitment

Recruitment status

Completed

Date of first participant enrolment

Anticipated

1/07/2020

Actual

22/09/2020

Date of last participant enrolment

Anticipated

1/07/2021

Actual

4/02/2022

Date of last data collection

Anticipated

1/06/2022

Actual

6/07/2022

Sample size

Target

211

Accrual to date

Final

190

Recruitment in Australia

Recruitment state(s)

NSW

Recruitment hospital [1]

Royal Prince Alfred Hospital - Camperdown

Recruitment postcode(s) [1]

2050 - Camperdown

Recruitment outside Australia

Country [1]

Austria

State/province [1]

Vienna

Funding & Sponsors

Funding source category [1]

Commercial sector/Industry

Name [1]

MetaOptima Technology Inc

Address [1]

1055 W Georgia St #2275
Vancouver, BC, Canada
V6E 3P3

Country [1]

Canada

Funding source category [2]

Other

Name [2]

Sydney Melanoma Diagnostic Centre

Address [2]

Royal Prince Alfred Hospital
Missenden Rd.
Camperdown 2050 NSW

Country [2]

Australia

Primary sponsor type

Commercial sector/Industry

Name

MetaOptima Technology Inc

Address

1055 W Georgia St #2275
Vancouver, BC, Canada
V6E 3P3

Country

Canada

Secondary sponsor category [1]

Other

Name [1]

Sydney Melanoma Diagnostic Centre

Address [1]

Royal Prince Alfred Hospital
Missenden Rd.
Camperdown 2050 NSW

Country [1]

Australia

Ethics approval

Ethics application status

Approved

Ethics committee name [1]

Royal Prince Alfred Hospital

Ethics committee address [1]

RESEARCH ETHICS & GOVERNANCE OFFICE
ROYAL PRINCE ALFRED HOSPITAL
Missenden Road CAMPERDOWN NSW 2050

Ethics committee country [1]

Australia

Date submitted for ethics approval [1]

31/05/2019

Approval date [1]

12/06/2019

Ethics approval number [1]

X19-0066 HREC

Summary

Brief summary

The purpose of this study is to compare various artificial intelligence algorithms used for the diagnosis and management of skin tumours against clinician diagnosis and management.

Who is it for?
You may be eligible for this study if you are an adult and either have previously had taken whole body photographs or have been assessed as requiring a biopsy of a skin lesion by your attending specialist..

Study details
Following your routine examination by your attending specialist you will be examined by two other doctors and have images taken of some of your skin lesions by a mobile phone. None of this will effect your management and any tests performed will only be related to your attending specialist's requests.

It is hoped that this study will help determine if artificial intelligence can accurately be used for the diagnosis and management of skin tumours.

Trial website

Trial related presentations / publications

Public notes

Contacts

Principal investigator

Name

Prof Scott Menzies

Address

The Sydney Melanoma Diagnostic Centre
Royal Prince Alfred Hospital
Missenden Rd.
Camperdown NSW 2050

Country

Australia

Phone

+61 2 95158537

Fax

[email protected]

Contact person for public queries

Name

Prof Scott Menzies

Address

The Sydney Melanoma Diagnostic Centre
Royal Prince Alfred Hospital
Missenden Rd
Camperdown NSW 2050

Country

Australia

Phone

+61 2 95158537

Fax

[email protected]

Contact person for scientific queries

Name

Prof Scott Menzies

Address

The Sydney Melanoma Diagnostic Centre
Royal Prince Alfred Hospital
Missenden Rd.
Camperdown NSW 2050

Country

Australia

Phone

+61 2 95158537

Fax

[email protected]

Data sharing statement

Will individual participant data (IPD) for this trial be available (including data dictionaries)?

Yes

What data in particular will be shared?

Images with ground truth

When will data be available (start and end dates)?

Following publication. The end date is estimated to be 2023.

Available to whom?

Following publication all members of public may apply to have access to the data. However, at this time, it is unclear whether a formal application will be required or the data will be on public domain.

Available for what types of analyses?

Assessment of future AI skin tumour diagnostic algorithms

How or where can data be obtained?

Details will be made available following publication. The data is owned by the investigators and contact will be made by email to the principal investigator ([email protected]) .

What supporting documents are/will be available?

Doc. No.	Type	Citation	Link	Email	Other Details	Attachment
7915	Study protocol					379808-(Uploaded-11-05-2020-14-11-03)-Study-related document.docx

Results publications and other study-related documents

Documents added manually

No documents have been uploaded by study researchers.

Documents added automatically

Source	Title	Year of Publication	DOI
Embase	Comparison of humans versus mobile phone-powered artificial intelligence for the diagnosis and management of pigmented skin cancer in secondary care: a multicentre, prospective, diagnostic, clinical trial.	2023	https://dx.doi.org/10.1016/S2589-7500%2823%2900130-9

N.B. These documents automatically identified may not have been verified by the study sponsor.

Trial Review

Documents added manually

Documents added automatically