In this critical narrative review, we examine the role of physical activity (PA), recreational and elite sports in the development of knee/hip osteoarthritis (OA), taking into account the role of injury in this relationship. The process of article selection was unsystematic. Articles were selected on the basis of the authors’ expertise, self-knowledge, and reflective practice. In the general adult population, self-reported diagnosis of knee/hip OA was not associated with low, moderate or high levels of PA. For studies using radiographic knee/hip OA as a primary outcome, the incidence of asymptomatic radiographic OA was higher for subjects with the highest quartile of usual PA than the least active subjects. The risk of incident radiographic knee/hip OA features was increased for subjects with a history of regular sports participation (for osteophyte formation but not joint space narrowing). This risk depended on the type of sport (team and power sports but not endurance and running), and certain conditions (high level of practice) were closely related to the risk of injury. The prevalence of radiographic OA was significantly higher, especially the presence of osteophytes, in former elite athletes than controls. The risk of OA was higher with participation in mixed sports, especially soccer or power sports, than endurance sport. However, the prevalence of clinical OA between former elite athletes and controls was similar, with less hip/knee disability in former athletes. Moderate daily recreational or sport activities, whatever the type of sport, are not a consistent risk factor for clinical or radiographic knee/hip OA. Risk of injury in different sports may be the key factor to understanding the risk of OA related to sport.