Accurate predictions of the properties of interplanetary coronal mass ejection (ICME)-driven disturbances are a key objective for space weather forecasts. The ICME’s time of arrival (ToA) at Earth is an important parameter and one that is amenable to a variety of modeling approaches. Previous studies suggest that the best models can predict the arrival time to within an absolute error of 10-15 hours. Here, we investigate the main sources of error in predicting a CME’s ToA at Earth. These can be broken into two main categories: (1) the initial properties of the ejecta, including its speed, mass, and direction of propagation; and (2) the properties of the ambient solar wind into which it propagates. To estimate the relative contribution to ToA errors, we construct a set of numerical experiments of cone-model CMEs, where we vary the initial speed, mass, and direction at the inner radial boundary. Additionally, we build an ensemble of 12 ambient solar wind solutions using realizations from the ADAPT model. We find that each component in the chain contributes between ±2.5 and ±7 hours of uncertainty to the estimate of the CME’s ToA. Importantly, different realizations of the synoptic produce the largest errors. This suggests that estimates of ToA will continue to be plagued with intrinsic errors of ±10 hours until tighter constraints can be found for these boundary conditions. Our results suggest that there are clear benefits to focused investigations aimed at reducing the uncertainties in CME speed, mass, direction, and input boundary magnetic fields.