A well-performed difference map is very important for the change detection of remote sensing images. However, due to the influence of the lighting conditions and the change of the sensor, the difference maps often have low contrast between changed and unchanged pixels, which makes it difficult for subsequent cluster analysis. A coupled distance metric learning (CDML) model is proposed to solve the problem. The model attempts to learn a pair of mapping matrices and transform bi-temporal image data into a common feature space in which the contrast between the changed and unchanged pixels will be further enhanced. First, a sample selection mechanism is proposed to select training samples with high accuracy. Then, these samples are used to learn a pair of mapping matrices by minimizing the sum of the distances between the unchanged samples and maximizing the sum of the distances between the changed samples according to the CDML. Finally, the original images are mapped to the same feature space respectively by the mapping matrices, and the difference is calculated by L2 norm. The final experimental results confirm the feasibility and effectiveness of the proposed model.