Introduction

Deep learning algorithms have been widely used to address biological and biomedical imaging problems in recent years [1–4]. Common image-related tasks include detection of cell nuclei [5–7], semantic segmentation of tumors [8–11], and diagnosis of diseases [12–17]. In general, standard computer vision tasks are straightforward, such as object detection, segmentation, or classification. A unique challenge posed by medical images is that there are often multiple related images in a series, which requires clinicians to analyze and diagnose diseases through different angles across images [18–20]. For example, mammograms are often performed from both the cranial caudal view and the mediolateral oblique view [21,22]. How to integrate information from multi-view images and multiple regions is currently an active research topic in the field [23–26].
Here we focus on images of rheumatoid arthritis (RA), which is a chronic autoimmune disease that affects many regions of the body [27]. About 0.5 percent of adults are affected by RA worldwide with a higher prevalence in women than men [28]. RA primarily targets and damages joints, including pain and swelling around the joint regions. Without timely diagnosis and appropriate treatment, RA can lead to severe and irreparable joint damages and disability [27]
Many modern imaging techniques have been developed for the visualization and detection of joint damages in RA. The current standard quantification of RA damage is the Sharp/van der Heijde (SvH) scoring system with radiographic imaging [29], including joint space narrowing and bone erosion. However, manually inspecting radiographic images and evaluating SvH scores is time-consuming and effort-intensive. The fast advancement of machine learning and artificial intelligence (AI) has opened a new avenue for automatic diagnosis of RA using computers [30]. Deep learning methods have been developed to address the image-based RA scoring problems [31–34]. Yet independent and comprehensive benchmarking are needed to evaluate the predictive performance of AI methods, especially on stringently held-out testing data. Moreover, beyond high predictive performance, a key question is to gain a deeper understanding of AI methods in RA. A typical presentation of RA is symmetrical symptoms in multiple large and small joints [35,36]. Unfortunately, how to leverage the multiple images and reveal the working mechanism underlying a computational method is largely unexplored.
Here we present Mandora (MAchine learNing Detection of jOint damages in Rheumatoid Arthritis), a machine learning approach for quantification of joint damages in RA based on radiographic images. Benefiting from cutting-edge deep learning techniques as well as conventional machine learning methods to exploit different types of information, this method automatically quantifies the degree of joint damage with high accuracy. It ranked first in scoring joint space narrowing in the recent DREAM Rheumatoid Arthritis Challenge, where state-of-the-art methods were systematically compared and benchmarked on independent testing data. Additionally, this method segments and highlights the joint space regions to assist further disease diagnosis in clinical practice. Most importantly, we leverage the multi-level symmetrical patterns in RA patients and integrate information across multiple images to improve performance and reveal the predictive relationships across joints, damage types, and left-right sides [37,38].