One of the biggest challenges in drug development for Alzheimer’s disease (AD) is how to effectively remove deposits of amyloid-beta (Aβ). Recently, the relationship between microglia and Aβ has become a research hotspot. Emerging evidence suggests that Aβ-induced microglia-mediated neuroinflammation further aggravates the decline of cognitive function, while microglia are also involved in the process of Aβ clearance. Hence, microglia have become a potential therapeutic target for the treatment or prevention of AD. An in-depth understanding of the role played by microglia in the development of AD will help us to broaden therapeutic strategies for AD. In this review, we provide an overview of the dual roles of microglia in AD progression: the positive effect of phagocytosis of Aβ and its negative effect on neuroinflammation after over-activation. With the advantages of novel structure, high efficiency, and low toxicity, small-molecule compounds as modulators of microglial function have attracted considerable attention in the therapeutic areas of AD. In this review, we also summarize the therapeutic potential of small molecule compounds (SMCs) and their structure–activity relationship for AD treatment through modulating microglial phagocytosis and inhibiting neuroinflammation. For example, the position and number of phenolic hydroxyl groups on the B ring are the key to the activity of flavonoids, and the substitution of hydroxyl groups on the benzene ring enhances the anti-inflammatory activity of phenolic acids. This review is expected to be useful for developing effective modulators of microglial function from SMCs for the amelioration and treatment of AD.