Phellinus baumii extract (PBE) possesses considerable α-glucosidase-inhibited activity. This study investigated the hypoglycemic effect in vitro and in vivo using a glucose consumption assay in HepG2 cells, intragastric administration for ten weeks in STZ-induced mice, and intestinal flora fermentation in patients with type 2 diabetes to reveal the possible underlying mechanisms. PBE was prepared, including α-glucosidase-inhibited ethanol extract (EE) and aqueous extract (AE). In vitro , PBE promoted glucose consumption and enhanced glycogen content and hexokinase activity but lowered phosphoenolpyruvate carboxylase kinase activity in HepG2 cells. In vivo , PBE treatment significantly reduced the body weight (p  <  0.05) and fasting blood glucose levels of diabetic mice (p  <  0.01), with the lowest blood glucose level observed in the EE+AE group. Furthermore, the serum insulin levels and insulin resistance index (HOMA) of PBE-treated groups decreased significantly (p  <  0.01). Moreover, gene expression levels of the IRS-1/PI3K/AKT pathway were significantly upregulated by PBE treatment (p  <  0.01). In vitro fermentation demonstrated that EE significantly inhibited the production of H ₂ S and NH ₃ in the intestinal flora fermentation model in diabetic patients (p  <  0.05). In addition, the ratio of Firmicutes to Bacteroidetes was reduced, the growth of Lactobacillus and Prevotella 9 was promoted, and Pseudomonas aeruginosa was inhibited. This study provides new insights and clues for using PBE as a functional food and clinical drug for glycemic control.