We propose a novel approach for the optimization of the H-2 norm for time-delay systems, grounded in its characterization in terms of the delay Lyapunov matrix. We show how the partial derivatives of the delay Lyapunov matrix with respect to system or controller parameters can be semianalytically computed, by solving a delay Lyapunov equation with inhomogeneous terms. It allows us to obtain the gradient of the H-2 norm and in turn to use it in a gradient-based optimization framework. We demonstrate the potential of the approach on two classes of problems, the design of robust controllers and the computation of approximate models of reduced dimension. Thereby, a major advantage is the flexibility: in the former class of applications, the order or structure of the controller can be prescribed, including recently proposed delay-based controllers. For the latter class of applications, approximate models described by both ordinary and delay differential equations (e.g., inhering the structure of the original system) can be synthesized.