Trial Review

The safety and scientific validity of this study is the responsibility of the study sponsor and investigators. Listing a study does not mean it has been endorsed by the ANZCTR. Before participating in a study, talk to your health care provider and refer to this information for consumers
Trial registered on ANZCTR


Registration number
ACTRN12620000488909
Ethics application status
Approved
Date submitted
16/03/2020
Date registered
20/04/2020
Date last updated
20/04/2020
Date data sharing statement initially provided
20/04/2020
Type of registration
Prospectively registered

Titles & IDs
Public title
Pivotal trial to evaluate an Artificial intelligence (AI) diabetic retinopathy grading classifier in the New Zealand population undergoing regular screening for diabetic retinopathy.
Scientific title
A prospective evaluation of the performance of an AI diabetic retinopathy grading algorithm to detect referable diabetic retinopathy (rDR) in the New Zealand diabetic retinopathy screening program.
Secondary ID [1] 300791 0
Nil known
Universal Trial Number (UTN)
U1111-1249-7630
Trial acronym
AEye
Linked study record
Nil

Health condition
Health condition(s) or problem(s) studied:
Diabetes 316656 0
Diabetic retinopathy 316657 0
Condition category
Condition code
Metabolic and Endocrine 314903 314903 0 0
Diabetes
Eye 314904 314904 0 0
Diseases / disorders of the eye

Intervention/exposure
Study type
Interventional
Description of intervention(s) / exposure
Currently, when patients attend for their routine diabetic retinopathy screening visit, a photograph is taken of their retina. This is typically a 20-30 minute appointment. Once the image has been obtained all the images thus collected are sent to a human grading team for review and grading. (The conventional grading pathway). This usually happens several days after the patient has attended for screening with the final result being issued to the patient and their GP a couple of weeks later.
During this current study the retinal images from those patients who give their consent, will also be read by an artificial intelligence classifier which has been trained to grade to the New Zealand MoH standard. (AI grading pathway). The inference of the AI classifier will be generated once the patient has left the clinic and thus it is envisaged that they will not be inconvenienced, or their clinic visit prolonged, if they choose to participate in this trial.
A masked observer will then collect the grades issued to each patient after these images have passed through the conventional grading pathway and compare them against the grades issued to each patient after the images have passed through the AI grading pathway.

At the time their consent is sought, patients will also be asked to complete a short questionnaire on their perceptions of artificial intelligence systems reading their retinal images. This questionnaire will be collected up by the team at the reception desk when they leave the clinic
Intervention code [1] 317115 0
Diagnosis / Prognosis
Comparator / control treatment
This study is designed to compare the accuracy of the grades issued by trained human graders to a trained artificial intelligence system. The trained human graders (the conventional grading pathway) will be considered the gold standard and will therefore form the comparator/control group against which the AI grades will be compared.
Control group
Active

Outcomes
Primary outcome [1] 323227 0
To compare the accuracy of the grades issued by trained human graders against those issued by the artificial intelligence system.
The grades used in this study are those issued by the MoH in their guidance Diabetic Retinal Screening, Grading, Monitoring and Referral Guidance published 2006. Found at https://www.health.govt.nz/system/files/documents/publications/diabetic-retinal-screening-grading-monitoring-referral-guidance-mar16.pdf
Timepoint [1] 323227 0
4 months
Secondary outcome [1] 381228 0
To assess the attitudes of patients with diabetes to the concept of having their images being graded by an AI classifier,
This will be assessed using the questionnaire developed by Ongena YP,et al (Patients’ views on the implementation of artificial intelligence in radiology: development and validation of a standardized questionnaire. http://link.springer.com/article/10.1007/s00330-019-06486-0) but reading diabetic eye screening rather than radiology.
Timepoint [1] 381228 0
This outcome will be assessed as the patient is waiting for their retinal screening to be performed and the questionnaire will be collected as they leave the clinic.

Eligibility
Key inclusion criteria
All patients with diabetes who are attending a DHB funded diabetic retinopathy screening program.
Minimum age
18 Years
Maximum age
No limit
Sex
Both males and females
Can healthy volunteers participate?
No
Key exclusion criteria
Vulnerable patients who are unable to give their consent.

Study design
Purpose of the study
Diagnosis
Allocation to intervention
Non-randomised trial
Procedure for enrolling a subject and allocating the treatment (allocation concealment procedures)
Methods used to generate the sequence in which subjects will be randomised (sequence generation)
Masking / blinding
Who is / are masked / blinded?



Intervention assignment
Other design features
Phase
Not Applicable
Type of endpoint/s
Statistical methods / analysis
Study success will be pre-defined as both sensitivity and specificity of the AI system in the New Zealand population. The hypotheses of interest are:
H0 : p<p0 vs: HA : p > p0
where p is the sensitivity or specificity of the AI system and p0 = 75% for the sensitivity endpoint and p0 = 77.5% for the specificity endpoint under the null hypotheses.
The alternative hypotheses are 85% for sensitivity and 82.5% for specificity, reflecting anticipated enrollment numbers and pre-specified regulatory requirements. One-sided testing will be further prespecified for both sensitivity and specificity; a one-sided 2.5% Type I error will be used resulting in a one-sided 97.5% rejection rule per hypothesis. To preserve Type I error, study success will be defined as requiring both null hypotheses to be rejected at the end of the study, e.g.
Pp(HA/Data) >0:975.
The primary sensitivity calculation will be performed using a logistic regression model including all participants with referrable retinopathy rDR without any baseline covariate adjustment, while the primary specificity calculation will be performed using a logistic regression model with enrichment as a baseline covariate. A Firth adjustment will be used to project sensitivity without any baseline covariate adjustment while the specificity will be projected using absent enrichment status to diffuse spectrum bias; enrichment will be used to increase the number of rDR cases based on stepwise increase of HbA1C levels and this approach is therefore expected to cause enrichment spectrum bias. Therefore, the specificity calculation will be prespecified to correct for such spectrum bias; no such correction will be prespecified for sensitivity analysis because the goal is to shift the frequency of disease to the more severe DR cases. No data imputation will be used for primary analyses.
Analyses will be based on the data from the study population: participants who have valid results on both the conventional grading and the AI grading protocols. Results will be reported as means, medians and with corresponding two sided 95% confidence intervals (CI).
Sample sizes for these hypotheses were calculated for at least 85% power and one-sided 2.5% Type 1 error requiring samples of 149 participants with referable DR and 682 with non referrable retinopathy.

Recruitment
Recruitment status
Not yet recruiting
Date of first participant enrolment
Anticipated
20/04/2020
Actual
Date of last participant enrolment
Anticipated
31/08/2020
Actual
Date of last data collection
Anticipated
30/09/2020
Actual
Sample size
Target
1000
Accrual to date
Final
Recruitment outside Australia
Country [1] 22438 0
New Zealand
State/province [1] 22438 0
Auckland

Funding & Sponsors
Funding source category [1] 305248 0
Charities/Societies/Foundations
Name [1] 305248 0
Diabetes New Zealand Auckland Branch. Graeme Mack Research award.
Country [1] 305248 0
New Zealand
Primary sponsor type
Individual
Name
Dr David Squirrell
Address
Ophthalmology clinical lead for the ADHB diabetic retinopathy screening program
Department of Ophthalmology
Greenlane Hospital,
Greenlane West,
Auckland 1051
Country
New Zealand
Secondary sponsor category [1] 305614 0
None
Name [1] 305614 0
Address [1] 305614 0
Country [1] 305614 0
Other collaborator category [1] 281239 0
Individual
Name [1] 281239 0
Dr Ehsan Vaghefi
Address [1] 281239 0
Department of Optometry,
University of Auckland
505 85 Park Road, Grafton,
Auckland 1023
Country [1] 281239 0
New Zealand

Ethics approval
Ethics application status
Approved
Ethics committee name [1] 305591 0
Auckland DHB Research Review Committee
Ethics committee address [1] 305591 0
Auckland DHB Research Office
Level 14, Support Bldg
Auckland City Hospital
PB 92024, Grafton,
Auckland 1148
Ethics committee country [1] 305591 0
New Zealand
Date submitted for ethics approval [1] 305591 0
31/07/2019
Approval date [1] 305591 0
06/08/2019
Ethics approval number [1] 305591 0
A+8218

Summary
Brief summary
We have, using data from the ADHB and CMDHB Diabetic retinopathy screening programs (ADHB approved studies A+8335 and A+8218, CMDHB approved study 947 DR Eye AI), developed a bespoke AI algorithm to grade referable diabetic retinopathy to the standard mandated by the MoH with a sensitivity of over 95% and specificity of over 92%. In this next phase of the project we will conduct a prospective evaluation of the results of the grades issued during routine screening and compare them against the grades issued by the AI algorithm.
Trial website
N/A
Trial related presentations / publications
Nil
Public notes

Contacts
Principal investigator
Name 100886 0
Dr David Squirrell
Address 100886 0
Department of Ophthalmology,
Auckland District health board,
Greenlane Hospital,
Greenlane West,
Auckland 1051
Country 100886 0
New Zealand
Phone 100886 0
+64 0211463841
Fax 100886 0
Email 100886 0
[email protected]
Contact person for public queries
Name 100887 0
Dr David Squirrell
Address 100887 0
Department of Ophthalmology,
Auckland District health board,
Greenlane Hospital,
Greenlane West,
Auckland 1051
Country 100887 0
New Zealand
Phone 100887 0
+64 0211643841
Fax 100887 0
Email 100887 0
[email protected]
Contact person for scientific queries
Name 100888 0
Dr David Squirrell
Address 100888 0
Department of Ophthalmology,
Auckland District health board,
Greenlane Hospital,
Greenlane West,
Auckland 1051
Country 100888 0
New Zealand
Phone 100888 0
+64 0211463841
Fax 100888 0
Email 100888 0
[email protected]

Data sharing statement
Will individual participant data (IPD) for this trial be available (including data dictionaries)?
No
No/undecided IPD sharing reason/comment


What supporting documents are/will be available?

Doc. No.TypeCitationLinkEmailOther DetailsAttachment
7364Ethical approvalemail from ADHB research office explaining that the ethics application for this project can be added to the existing ethics application    379457-(Uploaded-18-04-2020-05-58-20)-Study-related document.docx
7490Ethical approvalThe original ADHB ethics approval cited in the email attached above   379457-(Uploaded-18-04-2020-06-03-39)-Study-related document.pdf
7491Ethical approvalHDEC ethics approval   379457-(Uploaded-02-04-2020-08-56-07)-Study-related document.pdf



Results publications and other study-related documents

Documents added manually
No documents have been uploaded by study researchers.

Documents added automatically
SourceTitleYear of PublicationDOI
EmbaseA multi-centre prospective evaluation of THEIATM to detect diabetic retinopathy (DR) and diabetic macular oedema (DMO) in the New Zealand screening program.2023https://dx.doi.org/10.1038/s41433-022-02217-w
N.B. These documents automatically identified may not have been verified by the study sponsor.