The emerging coronavirus disease 2019 (COVID-19) has become a serious global public health threat. With more and more recovered patients, it is urgently needed for evaluation of the neutralizing antibody (NAb) in these patients. In this study, we collected blood samples from 49 patients recently recovered from COVID-19. Serum NAbs were measured using a novel surrogate virus neutralization test (sVNT). Factors associated with NAb titers were analyzed using Ordinary Least Squares regression model. The median age of the study participants was 37 years (IQR, 30.0–54.5) and 55.1 % (27/49) of which were male. The median time to blood collection (for NAb analysis) from illness onset, viral clearance and discharge were 43.0 days (IQR, 36.0–50.0), 27.0 days (IQR, 20.5–37) and 17.0 days (IQR, 15.0–33.0), respectively. Patients had a median NAb titer of 1: 40 (IQR, 1:15–1:120). NAbs were not detected in two asymptomatic children who quickly cleared the virus. NAb titers were higher in patients with older age (p = 0.020), symptomatic infection (p = 0.044), more profound lung involvement (p＜0.001), abnormal C-reactive protein level (p＜0.01) and elevated lactate dehydrogenase (p = 0.019). Multivariable analysis revealed that severity of pneumonia and having comorbidity positively correlated with NAb titers in recovered patients (p = 0.02), while use of corticosteroids negatively impacted NAb titers (p = 0.01). Our study suggests that some COVID-19 patients may not have detectable NAb after recovery. SARS-CoV-2 NAb titers are positively correlated with severity of COVID-19 pneumonia.