The integration of artificial intelligence (AI), particularly large language models (LLMs), into clinical prescription practices represents a transformative shift in healthcare, promising to offer enhanced therapeutic precision, reduced errors, and personalized care. However, rapid adoption is outpacing validation and governance. This narrative review critically examines the benefits, risks, and implementation challenges of AI in pharmacology, with an emphasis on prescribing use cases, highlighting the rapid adoption driven by physician constrained time and increasing complexity of medical knowledge, yet outpaced by validation frameworks. Key benefits include optimized decision support, as suggested by real-world systems like the Penda Health AI Copilot. However, inherent risks such as hallucinations, lack of explainability, spending inflation, and clinical de-skilling pose significant threats, potentially eroding trust and patient safety. We contrast regulatory philosophies [European Union (EU) precaution under the medical devices regulation (MDR) and AI Act vs U.S. flexibility via Food and Drug predetermined change control plans (FDA PCCP)] and spotlight a persistent grey zone in which general-purpose LLMs are used clinically without medical-device oversight. To operationalize trustworthy use, we foreground rigorous evaluation and outline an “architecture of trustworthy clinical AI” (retrieval-augmented generation, structured clinical prompting with abstention/uncertainty, and hybrid LLM–rule safety layers). Finally, we propose an outcomes-first paradigm adapted from French health technologies assessment (HTA) — service rendered by AI (SR-AI) and improvement of service rendered by AI (ISR-AI), with patient outcomes and system resilience as primary endpoints — and recommend a default stance that any generative AI used in health be treated as a medical device, with general-purpose LLMs permitted only via certified clinical wrappers that close the MDR-inconsistent gap.