Validity refers to an evidence-based claim about the trustworthiness of decisions made from context-specific performance data. Validity requirements for competency-based assessments in obstetrics and gynecology have not been defined in the literature. We explain why validity is intrinsic to any discussion about competency assessment and provide a model for obstetrics and gynecology programs to use in determining the essential validity evidence for various forms of assessments. The implications of decisions made from assessment results influence the requisite level and precision of validity evidence. Although validity evidence is essential, it is also flexibly tied to the implications of decisions made from assessment results and not all assessments require the same degree of validity. We propose a model for considering validity, and build a discussion around specific assessment examples targeting progressive levels of expertise along the training continuum.