Validation and evaluation as essentials to ensuring safe AI health applications

Artificial Intelligence (AI) is a rapidly growing technology within health informatics, but it is not subject to the rigor of scientific and safety validation required for all other new health techniques. Moreover, some functions of health AI cannot only introduce biases but can then reinforce and spread them by building on them. Thus, while health AI may bring benefit, it can also pose risks for safety and efficiency, as end users cannot rely on rigorous pre-implementation evidence or in-use validation. This review aims to revisit the principles and techniques already developed in health informatics, to build scientific principles for AI evaluation and the production of evidence. The Precautionary Principle provides further justification for such processes, and continuous quality improvement methods can add assurance. Developers should be expected to provide a robust evidence and evaluation trail, and clinicians and patient groups should expect this to be required by policy makers. This needs to be balanced with a need for developing pragmatic and agile evaluation methods in this fast-evolving area, to deepen knowledge and to guard against the risk of hidden perpetuation of errors.