The direct problem of optical microlithography is to simulate printing features on the wafer under the given mask, imaging system, and process characteristics. The goal of inverse problems is to find the best mask, imaging system, or process to print the given wafer features. In this study, we proposed the strict formalization and fast solution methods of inverse mask problems. We stated inverse mask problems (or “layout inversion” problems) as nonlinear, constrained minimization problems over a domain of mask pixels. We considered linear, quadratic, and nonlinear formulations of the objective function. The linear problem is solved by an enhanced version of the Nashold projections. The quadratic problem is addressed by eigenvalue decompositions and quadratic programming methods. The general nonlinear formulation is solved by the local variations and gradient descent methods. We showed that the gradient of the objective function can be calculated analytically through convolutions. This is an important practical result because it enables layout inversion on a large scale in order of operations for pixels.