The success of in vitro fertilization (IVF) treatment for infertility majorly depends upon the selection of a healthy embryo by the embryologist which is highly subjective and depends on the expertise of the embryologist. This work introduces a comprehensive framework starting with the collection and pre- processing of the day 3 embryo and blastocyst images. It is followed by extraction of multifaceted information that includes color, edge, and other relevant features using local Descriptor, capturing the complex details necessary for precise embryo evaluation. Feature selection is done using the Extra Trees classifier and is followed by a one-dimensional Convolutional Neural Network (1D-CNN) for deeper feature extraction. The interpretability and predictive power of the extracted features is enhanced by 1D-CNN. Using a novel approach, the last layer of the 1D-CNN is replaced with an ensemble of classifiers to determine the quality of embryos. This ensemble technique leverages the unique strengths of each classifier used, providing a robust and comprehensive decision framework. The proposed method significantly outperforms existing approaches with an accuracy of 93% and 98% with blastocyst and day 3 embryo dataset, respectively. The research is undertaken in collaboration with Gynaecworld, the Center for Women's Health & Fertility, Mumbai.