The reconstruction-based coding methods utilize a dictionary to approximate an input feature as a sparse representation in image classification tasks by solving a least-squares problem with constraints. The input feature can be a handcrafted feature calculated over a small patch or a deep feature extracted by a pre-trained convolutional neural network model. As known, the contribution degrees of features extracted within different regions are distinct to the same classification task. Given this, we argue that for features with a large contribution degree, if their reconstruction errors induced by a reconstruction-based coding method can be decreased, more discriminative information will be retained to improve classification accuracy. Considering that different coding methods have the same lower bound of reconstruction error, we propose a visual dictionary refining method that adjusts only the lower bounds of the errors of features according to their contribution degrees. Concretely, given an initial dictionary used by a reconstruction-based coding method, the indexes (coding indexes) of the words from the dictionary for encoding each feature are obtained. Then, the initial dictionary is updated according to the contribution degrees and coding indexes of features. The highlight of our method is that it can be applied to many reconstruction-based coding methods without damaging their original constraints. Experimental results on 15-Scenes, Caltech-101, and UIUC-Sports show that the classification accuracies of four representative coding methods are improved by 0.17% to 1.5% after applying our method. These results exhibit the effectiveness and universality of our method.