Modern society requires a sustainable, robust, and serviceable infrastructure to promote social welfare and boost economy. To support such infrastructure systems, an efficient health monitoring framework is needed which can promptly detect the presence of defects and perform associated rehabilitation and maintenance. In civil infrastructure, one of the most common types of defects is cracking, which evolves rapidly under the impacts of heavy traffic, aging of materials, and drastic environmental changes. In recent decades, image-based automated crack detection methodologies have been developed and extensively applied by professionals and researchers. Nevertheless, a few issues and challenges existing in this type of methodology are yet to be systematically investigated and properly addressed. In this study, an image-based condition assessment framework for roadway crack detection is developed. It consists of four topics: i) proposing a filter-based methodology that can address image disturbances to promote a robust image-based roadway crack detection; ii) performing a systematic study to investigate the impact from hyperparameter selection on the performance of deep convolutional neural network (DCNN) on roadway crack classification; iii) achieving pixel-level crack detection resolution on image data of real-world complexities through DCNN-based roadway crack segmentation; and iv) investigating the impact from heterogeneous image data on DCNN-based roadway crack detection and proposing heterogeneous image fusion strategies to address data uncertainties. Overall, experimental results and discussions show that the proposed crack detection framework is capable to properly address the issues under investigation and leads to improved and more robust crack detection performance than current image-based methodologies.