Superlubricity provides a novel approach to addressing friction and wear issues in mechanical systems. However, little is known regarding improving the atomic force microscope (AFM) friction coefficient measurement resolution. Accordingly, this study established the theoretical formula for the AFM friction coefficient measurement and deduced the measurement resolution. Then, the formula was applied to the AFM probe with a rectangular cross- cantilever. The measurement resolution is associated with the dimensional properties of the AFM probe, the mechanical properties of the cantilever material, the properties of the position-sensitive detector (PSD), and probably the anti-vibration performance of the AFM. It is feasible to make the cantilever as short as possible and the tip as high as possible to improve the measurement resolution. An AFM probe for measuring an ultra-low friction coefficient was designed and fabricated. The cantilever’s length, width, and thickness are 50, 35, and 0.6 µm, respectively. The tip height is 23 µm. The measurement resolution can reach 7.1×10−6 under the maximum normal force. Moreover, the AFM probe was applied to measure the superlubricity between graphene layers. The friction coefficient is 0.00139 under 853.08 nN. This work provides a promising method for measuring a ∼10−5 friction coefficient of superlubricity.