Optimization-based approach is developed in this paper in order to obtain the best combinations of long-lived transuranic isotopes charge to the subcritical reactor driven by accelerator. The most important isotopes were selected and nonlinear dynamic system for their concentration change is constructed. The solution of this dynamic system calculated at the end of operation time determines the radioactivity of the unloaded fuel. The goal of optimization is to reduce its total radiological impact by varying the initial nuclide densities, taking into account the physical constraints on subcriticality of reactor core. The problem is treated as nonlinear constrained programming, for which optimization algorithm based on quasi-Newton methods is suggested.