Timely diagnosis of autism spectrum disorder (ASD) is vitally important for improving the prognosis of young children with this condition. One of the greatest challenges facing healthcare providers for individuals with ASD and their families is shortening the time between when symptoms first appear and when an assessment for ASD is conducted. Current practice guidelines suggest pediatric screening should occur before 24 months of age for all children to help in detecting ASD as early as possible. Currently, screeners such as the Modified Checklist for Autism in Toddlers – Revised (M-CHAT-R) have been developed and validated for use in primary care settings. However, an underlying assumption behind screeners such as the M-CHAT-R is that parents are able to adequately understand the items on a screener questionnaire and relate those items back to their child’s behavior. Using an item response theory framework, the current study found that the majority of behaviors characteristic of ASD assessed during the screening process are easy or very easy for parents to correctly identify. This study also found that greater parental knowledge of both child development norms and knowledge of ASD helped parents to accurately identify symptoms of ASD, but only when these symptoms were severe. Results of the current study help to highlight a fundamental divide in screening wherein more severe cases of ASD are well captured by current screening measures, but mild, less severe cases of ASD may require closer examination in future studies on screening accuracy.