Introduction

Deep learning algorithms have been widely used to address biomedical imaging problems in recent years. Common image-related tasks include detection of cell nuclei 1, semantic segmentation of tumors 2,3, and diagnosis of disease 4–7. In general, standard biomedical imaging tasks are straightforward, such as object detection, segmentation, or classification. One special aspect of medical images is that they are often taken as a series, in order to inform the clinician through different angles. For example, mammography is often taken from both the cranialcaudal view and the mediolateral oblique (MLO) view. How to integrate the imaging information from multiple views, or even multiple regions of the body is currently an open question in the field.
We are interested in studying the images of rheumatoid arthritis (RA), which is a chronic autoimmune disease that affects many regions of the body, and therefore contains many regions to be investigated when analyzing images 8. About 0.5 percent of adults are affected by RA worldwide with a higher prevalence in women than men 9. RA primarily targets and damages joints, including pain and swelling around the joint regions. Without timely diagnosis and appropriate treatment, RA can lead to severe and irreparable joint damages and disability 8
Many modern imaging techniques have been developed for the visualization and detection of joint damages in RA. The current standard quantification of RA damage is the Sharp/van der Heijde (SvH) scoring system with radiographic imaging10, including joint space narrowing and bone erosion. However, manually inspecting radiographic images and evaluating SvH scores is time-consuming and effort-intensive. The fast advancement of machine learning and artificial intelligence (AI) has opened a new avenue for automatic diagnosis of RA using computers 11. Deep learning methods have been developed to address the image-based RA scoring problems 12–15. Yet independent and comprehensive benchmarking are needed to evaluate the predictive performance of AI methods, especially on stringently held-out testing data. Moreover, beyond high predictive performance, a key question is to gain a deeper understanding of AI methods in RA. A typical presentation of RA is symmetrical symptoms in multiple large and small joints 16,17. Unfortunately, how to leverage the multiple images and reveal the working mechanism underlying a computational method is largely unexplored.
 Here we present Mandora (MAchine learNing Detection of jOint damages in Rheumatoid Arthritis), a machine learning approach for quantification of joint damages in RA based on radiographic images. Benefiting from cutting-edge deep learning techniques as well as conventional machine learning methods to exploit different types of information, this method automatically quantifies the degree of joint damage with high accuracy. It ranked first in scoring joint space narrowing in the recent DREAM Rheumatoid Arthritis Challenge, where state-of-the-art methods were systematically compared and benchmarked on independent testing data. Additionally, this method segments and highlights the joint space regions to assist further disease diagnosis in clinical practice. Most importantly, we leverage the multi-level symmetrical patterns in RA patients and integrated information across multiple images to improve performance and reveal the predictive relationships across joints, damage types, and left-right sides 18,19.