The aim of the present study was to evaluate how emotional, and relationship factors affect sexual desire in each gender, helping to justify gender differences in sexual desire. Two hundred and five men and 237 women answered a set of questionnaires assessing psychopathology, emotions during sexual activity, dyadic adjustment, and frequency of sexual desire. MANCOVAS (demographic variables as covariates) were conducted in order to assess differences between genders and groups of sexual desire (low/high desire). Findings indicated that women presented higher levels of psychopathology and significantly more emotions of hurt during sexual activity, whereas men presented more fear and shame in the same context. Moreover, psychopathology interfered in male and female sexual desire very similarly. However, emotions interfered in female and male sexual desire through different paths. Regarding dyadic adjustment, findings indicated that participants with high desire had significantly more dyadic consensus, cohesion, satisfaction, and affection. These findings were independent of gender, suggesting that men and women evaluate their relationship dynamics similarly. Findings suggest that therapeutic protocols for sexual desire difficulties should attend to dyadic adjustment and emotions during sexual activity, also considering how psychopathology and specific emotions act differently on sexual desire according to gender.