Satellite image restoration from its degraded observation has a significant impact on the remote sensing industry. There are many potential applications that can directly benefit from this technique. A convolutional neural network (CNN) has been recently explored in image restoration and achieved remarkable performance. However, most deep CNN architectures in the literature do not properly handle the inherent trade-off between localization accuracy and the use of global context, which is vital for satellite images covering a ultrabroad area. We present a stacked lossless deconvolutional network (SLDN) for remote sensing image restoration. We fully exploit global context information while guaranteeing the recovery of fine details. Specifically, we design a lossless pooling by reformulating the pixel shuffle operator and incorporate it with a shallow deconvolutional network. The resulting lossless deconvolution blocks are stacked one by one to enlarge the receptive fields without any information loss. We further propose an attentive skip connection and progressive learning scheme to improve gradient flows throughout the SLDN. The SLDN can reconstruct high-quality satellite images without noticeable artifacts. An extensive ablation study is also provided to show that all the components proposed are useful for remote sensing image restoration. Experimental comparisons on various restoration tasks, including super-resolution, denoising, and compression artifact reduction, demonstrate the superiority of the proposed method over state-of-the-art methods both qualitatively and quantitatively.