To leverage this unique characteristic feature and consider the multi-level associations in RA, we developed a conventional tree-based machine learning model. Specifically, for each joint, the image patch-based predictions of all joint damages from the previous step are treated as input features in the tree-based model. The associations and dependent relationships among joints are learned automatically in this machine learning model. It further significantly improved the predictive performance of joint space narrowing and bone erosion in both hand and foot, except for the Pearson’s correlation of erosion in the foot (Fig. 8). This model indeed improved the correlation from 0.693 to 0.711, yet the difference is not statistically different based on the one-sided paired Wilcoxon signed-rank test (p-value = 0.041).