The Brain-Computer Interface (BCI) system based on motor imagery (MI) is a hot research topic nowadays, which can control external devices through the brain and has a wide range of applications in rehabilitation, gaming, and entertainment. Due to the non-smooth, non-linear, and low signal-to-noise ratio of the MI EEG signal, it is challenging to accurately decode the MI task intention. A new end-to-end deep learning method is proposed to decode raw MI EEG signals without preprocessing, such as filtering and feature reinforcement. The 1D convolution is used to learn the time-frequency features in MI signals, and a four-branch fusion network is used as the main body to add a 1D CNN-AE block and 1D SE-block to enhance the algorithm's performance. Experiments on two publicly available datasets demonstrate that our proposed algorithm outperforms the current state-of-the-art methods. It achieves 86.11% and 89.51% on the BCI Competition IV-2a and the BCI Competition IV-2b datasets, respectively, and a 6.9% improvement in the generalizability test. The proposed data enhancement method can effectively alleviate the overfitting of the algorithm and improve the decoding performance. Further analysis shows that 1D convolution can effectively extract the features associated with the MI task.