There is a critical need for high-quality clinical outcome assessments to capture the important aspects of communication ability of individuals with Angelman syndrome (AS). To center the perspective of caregivers, our team developed the novel Observer-Reported Communication Ability (ORCA) measure using best practice guidelines, with the goal of developing a measure that could be administered to caregivers directly without the need for a certified administrator for use in clinical trials. To refine the draft measure, we conducted two rounds of cognitive interviews with 24 caregivers and a quantitative study including 249 caregivers. The results from both studies support the overall content validity, construct validity, and the reliability of the ORCA measure for individuals with AS > 2 years old for use in research contexts. Future work should explore the responsiveness of ORCA measures to changes over time in a diverse sample.