In the present study, a combined finite/element approach is used to analyze failure characteristics of a rectangular hard rock with different sample height-to-width ratios based on the uniaxial compression test. The objective of this paper is to illustrate the transitional mechanisms from shear fracture to extension failure of brittle rocks and the occurrence conditions of surface-parallel slabbing. Numerical results show that failure modes have experienced the shear, tension with shear and surface-parallel slabbing failure with the decrease of sample height-to-width ratios. It is seen from the simulation results that the shear band in larger height-to-width ratios is composed of an amount of extension meso-cracks that are parallel to the loading direction, and the slabbing failure tends to be the result of the accessible propagation of extension meso-cracks due to short sample height, which validate the essence of extension failure in hard rocks. The good agreement between simulation and laboratory testing also shows the practicability and advancement of a combined finite/discrete element approach.