Modulation format identification (MFI) is widely used in elastic optical networks, such as optical performance monitoring and signal compensation. We propose an MFI scheme, through extraction of the amplitude probability distribution feature of the received signals and an analysis of the relative entropy, which is also known as the Kullback–Leibler divergence (KL). The modulation formats (MFs) can be distinguished via calculation of the KL value. Through this method, four commonly used transmission formats were investigated, namely, polarization-division multiplexing (PDM)-QPSK/16QAM/32QAM and 64QAM. A simulation transmission model was created, and the results were analyzed. The simulation results show that the MFs investigated can be distinguished when the optical signal-to-noise ratio (OSNR) is less than or equal to the corresponding theoretical 7% forward error correction (FEC) limit (bit error ratio  =  3.8  ×  10^−  3). In addition, two important factors, i.e., fiber nonlinearity and sample size, were investigated in relation to the proposed method. The results show that the method exhibits a certain tolerance to fiber nonlinearity and can be used to identify the MF at the FEC threshold even when the sample number is 1000. Furthermore, the experimental verifications were conducted to ensure the practicality of the proposed method.