Optical proximity correction (OPC) methods are resolution enhancement techniques used extensively in the semiconductor industry to improve the resolution and pattern fidelity of optical lithography. During the mask data preparation process, the mask pattern is first fractured into basic rectangles, and then fabricated by the variable-shaped-beam mask writing machine. The rectangle count included in the fractured pattern is preferable to be suppressed to reduce the mask fabricating time and cost. Recently, various pixel-based OPC (PBOPC) approaches have been developed to improve the resolution of optical lithography systems. However, these approaches fall short in controlling the rectangle count in the fractured pattern, thus deteriorating the manufacturability of the mask. This paper focuses on developing gradient-based PBOPC optimization algorithms to improve the resolution of optical lithography, while controlling the manufacturability of the mask. To achieve this goal, a topography filter is designed to analytically formulate the rectangle count in the fractured pattern during the optimization process. The manufacturability cost term is then introduced to constrain the complexity of the mask. Cost sensitivity is applied to speed up the proposed algorithms. A line search method is used to properly choose the parameters, and leads to superior resolution and manufacturability of masks.