Although randomized trials and observational studies are used as the evidentiary basis of clinical practice guidelines, they are not always in agreement. Limitations in the process of randomization in the former and the selective referral of patients for treatment as a consequence of clinical “risk stratification” in the latter are underappreciated causes for these disagreements. As a result, neither is guaranteed to correctly quantify treatment benefit. This essay reviews the operational differences between these alternative evidentiary sources and shows how these differences can affect individual clinical decisions, population-based practice guidelines, and national health policy. In conclusion, the process of evidence-based medicine can be improved by independent agencies charged with the responsibility to identify and resolve these differences.