Cardiac and respiratory movements pose significant challenges to image quality and quantitative accuracy in PET imaging. Cardiac and/or respiratory gating attempt to address this issue, but instead lead to enhanced noise levels. Direct four-dimensional (4D) PET image reconstruction incorporating motion compensation has the potential to minimize noise amplification while removing considerable motion blur. A wide-ranging choice of such techniques is reviewed in this work. Future opportunities and the challenges facing the adoption of 4D PET reconstruction and its role in basic and clinical research are also discussed.