Treatments using combinations of drugs are becoming common in the oncology and the adequate identification of the maximum tolerated dose (MTD) in a Phase I clinical trial is essential. The combination setting, however, makes it more difficult to order combination levels according to the monotonically increasing toxicity. Recently, several dose-finding methods, relaxing the monotonicity assumption of toxicity have been proposed in the literature. Through a comprehensive simulation study, we assessed the performance of two methods which relax the monotonicity assumption and two methods which employ it. In the context of the monotonicity assumption violation, we explored the problem of the toxicity ordering misspecification and proposed clinical recommendations.

Motivated by a real clinical trial of a combined induction chemotherapy and immunotherapy for patients with high-risk neuroblastoma, we consider a simulation setting of eight toxicity scenarios that include a large variety of qualitatively different monotonic and non-monotonic dose toxicity relations. Four dose-finding methods were compared : the Bayesian Continual Reassessment Method (CRM), the Modifed Toxicity Probability Interval method (mTPI), the Bayesian Partial Ordering CRM (PO-CRM) and the No Monotonicity Assumption design (NMA).

Considering scenarios with correctly specified monotonic ordering, the CRM showed high proportion of correct selections (PCS) ranging from 54.7 % to 99.6 %. The mTPI was the most conservative and showed the highest PCS (64.8 %) when the dose-toxicity relation was steep and resulted in the least number of toxicities in all scenarios. The CRM and the mTPI failed to find the MTD when the monotonicity assumption was violated. In contrast, the PO-CRM and the NMA resulted in 29 %–39 % and 39 %–41 % of PCS, respectively. The NMA showed the robust performance in the non-monotonic cases and had the best performance (41.0 % PCS against 30 % by PO-CRM) when the dose levels preceding the MTD were highly toxic. The PO-CRM showed better performance when the MTD was located at the starting dose level (54 % PCS against 44 % by NMA) and when the dose-toxicity relation was flat (78 % and 99 % against 52 % and 91 % by NMA, respectively).

The ordering of dose levels can have the great impact on a Phase I clinical trial and should be carefully justified during the planning of the study. If a single monotonic order cannot be specified, the NMA or the PO-CRM must be considered with a limited set of orderings. The choice of the design must be guided by comprehensive simulations taking into account each scenario preferences and extensive discussions with investigators.