Ongoing efforts to characterize underwater dunes have led to a considerable number of freely available tools that identify these bedforms in a (semi-)automated way. However, these tools differ with regard to their research focus and appear to produce results that are far from unequivocal. We scrutinize this assumption by comparing the results of five recently published dune identification tools in a comprehensive meta-analysis. Specifically, we analyse dune populations identified in three bathymetries under diverse flow conditions and compare the resulting dune characteristics in a quantitative manner. Besides the impact of underlying definitions, it is shown that the main heterogeneity arises from the consideration of a secondary dune scale, which has a significant influence on statistical distributions. Based on the quantitative results, we discuss the individual strengths and limitations of each algorithm, with the aim of outlining adequate fields of application. Yet, the concerted bedform analysis and subsequent combination of results have another benefit: the creation of a benchmarking data set which is inherently less biased by individual focus and therefore a valuable instrument for future validations. Nevertheless, it is apparent that the available tools are still very specific and that end-users would profit by their merging into a universal and modular toolbox.