Deep learning prediction of joint locations based on full images

We designed a deep convolutional neural network for predicting joint locations. This neural network contains an encoder and a decoder with multiple convolutional layers, max-pooling layers, up-convolutional layers, and concatenation layers. The kernel sizes of convolutional layers, max-pooling layers, and up-convolutional layers are 7×7, 2×2, 2×2, respectively. The encoder has 8 convolutional layers as well as 4 max-pooling layers. The decoder has 8 convolutional layers and 4 up-convolutional layers. Meanwhile, the encoder and the decoder are connected through concatenation layers. Each convolutional layer is followed by a non-linear “ReLU” activation, except for the last convolutional layer that is activated by a “sigmoid” function. The batch normalization layer is applied before each convolutional layer to accelerate the training process. The cross-entropy loss was used together with the Adam optimizer. The neural network was first trained 50 epochs with the learning rate of 0.001, then trained 50 epochs with a smaller learning rate of 0.0001. The neural network was implemented using “Keras” (2.2.4) with the “Tensorflow” (1.14.0) backend in Python.

Deep learning prediction of joint damages based on image patches

We designed a special deep convolutional neural network for predicting joint damages based on image patches. Specifically, this neural network has a encoder-decoder-regressor architecture with two outputs: (1) segmentation of the joint space region, and (2) joint damage score. The encoder has 8 convolutional layers and 4 max-pooling layers. The decoder has 8 convolutional layers and 4 up-convolutional layers. The regressor has 8 convolutional layers and 4 max-pooling layers. Meanwhile, concatenation layers were used to link (1) the encoder and the decoder, and (2) the decoder and the regressor. The decoder outputs the joint space segmentation that is used as the input for the regressor as well. The regressor outputs the joint damage score. The kernel sizes of convolutional layers, up-convolutional layers, and max-pooling layers are 7×7, 2×2, 2×2, respectively. Each convolutional layer is followed by a non-linear “ReLU” activation. The batch normalization layer is applied before each convolutional layer to accelerate the training process. The last layer is a “Dense” layer that is activated by a “sigmoid” function. The cross-entropy loss was used together with the Adam optimizer. The neural network was first trained 10 epochs with the learning rate of 0.001, then trained 40 epochs with a smaller learning rate of 0.0001. The neural network was implemented using “Keras” (2.2.4) with the “Tensorflow” (1.14.0) backend in Python.

Tree-based learning of symmetrical patterns

To learn the symmetrical patterns among joints and damage types, we further build a tree-based machine learning model. Specifically, for each joint damage to be predicted, the image patch-based predictions of all joint damages from the neural network model are used as the input. The tree-based model automatically learns the non-linear relationships among all joints from hands and feet, and two types of joint damages. A total of 500 trees are used in the ensemble predictions with a maximum depth of 4. The tree-based model was implemented using the “etr” function of “scikit-learn” (0.21.2) in Python.

SHAP analysis

To investigate the interconnections among joints and damage types in our method, we performed the SHAP analysis of the tree-based model. For each joint damage of interest, the absolute SHAP values of all joint damages are calculated, representing the strength of contribution to determining the degree of damage. The joint-wise absolute SHAP values are shown as two heatmaps: (1) the contributions of joints from the same side, and (2) the contributions from the other side. The SHAP analysis was performed using “shap” (0.31.0) in Python.

Statistical analysis

To test whether two results are significantly different in cross-validation experiments, we performed the one-sided paired Wilcoxon signed-rank test using R (3.6.1).
Ethical approval
Ethical approval was done through Sage Bionetworks who distributed the data through the cloud environment.

Data availability

The radiographic images used in this study were downloaded from the challenge website:
https://www.synapse.org/#!Synapse:syn20545111/wiki/597243
The segmentation masks of the joint space regions are available on our website:
https://guanfiles.dcmb.med.umich.edu/Mandora/segmentation_joint_space

Code availability

The code of Mandora is available in the GitHub repository:
https://github.com/GuanLab/Mandora

Author contributions

HL and YG conceived and designed this project and wrote the paper. HL implemented the method, performed the experiments

Results

Overview of experimental design

Scoring joint damages in RA is a complex task with two subtasks: (1) object detection - we need to detect and locate multiple joints within an image, and (2) disease recognition - we need to predict the degree of damage through regression analysis. To solve this unique problem, we developed a multi-step pipeline (Fig. 1). We first built a deep convolutional neural network model to identify the location of each joint. Once we obtained the location, an image was cut into small patches that were centered around joints. Then the image patches were used as the input for a specially designed neural network to perform regression analysis of joint damage scores, as well as semantic segmentation of joint space regions. Notably, segmentation was not required, yet it significantly improved predictive performance. Finally, patients with RA are likely to develop symmetrical symptoms in both sides of hands and feet, and joint space narrowing and bone erosion often go hand in hand. We therefore developed a tree-based conventional machine learning model to integrate all available information from both sides and two types of damages. The step not only further improved the performance but also revealed cross-joint prediction relationships that are the nature of RA.