An editorial article published in Ophthalmology (AAO) outlines a useful guide for the deciphering of studies in artificial intelligence (AI)(2019 Nov;126(11):1475-1479). In an AAO opinion piece it has been observed that a significant volume of research articles on AI has recently flooded the medical literature, with varying content. To assist readers on assessing the varying standards, methods, statistics, reporting metrics, and clinical translational value, editorial authors have suggested a number of brief pointers and insights in order to evaluate AI articles in ophthalmology.
The editorial article highlights 9 pointers which are both common sense and which explain how the field might evolve in the future. The key queries include:
• Is this study answering a question that matters?
• What are the core components in an artificial Intelligence system?
• What makes good training and testing datasets?
• What makes a good reference standard?
• What are the appropriate machine learning or deep learning techniques?
• What makes an excellent operational flow of an Artificial Intelligence system?
• How to assess the diagnostic performance clinically?
• What to watch for in an Artificial Intelligence system for clinical adoption?
• How can we guide Artificial Intelligence research better in the field?
According to the editorial, “deep learning” is a recently described AI machine learning technique applied to image analysis and this “allows an algorithm to analyse data using multiple processing layers to extract different image features”. In addition, the article cautions that the use of AI needs to consider and apply the relevant different regulations in each particular country. In commenting in the article on the international context, researchers have stated that there is a “variety of deployment models, and varying thresholds for referable DR (diabetic retinopathy)”. This also needs to consider what is a ”good reference standard”, and how this reference is to be derived depending on the users’ consensus and the relevant national guideline. Finally, in early 2019, the US FDA has proposed a framework entitled, “Proposed Regulatory Framework for Modifications for Artificial Intelligence/Machine Learning-Based Software as a Medical Device (SaMD)”, and this uses the International Medical Device Regulators Forum to consider AI-based software as a medical device.