The high-temperature rock drilling is a key process in the exploitation of the hot-dry-rock (HDR) geothermal energy. Understanding the abrasive characteristics of HDR is critical for drilling efficiency and cost. Therefore, this study investigates the wear mechanism and cutting/scratch evolutions of granite under temperatures from 25 °C to 500 °C by using the Cerchar abrasive test. The results show that: (1) The Cerchar Abrasivity Index (CAI) of granite decreases with the increasing temperature and three characteristic decreasing stages can be found. (2) The CAI of granite and the average cutting force exerted on its surface during sliding exhibit a similar trend as temperature increases, the high temperature treatment reduces the cutting force during sliding. (3) Observing the scratches, it is found that the cutting force increases sharply and aggravates the wear of the steel stylus when the steel stylus encounters minerals with higher hardness such as quartz or biotite. (4) There is a significant negative correlation between the instantaneous cutting force and the scratch depth during sliding. In other words, the higher the instantaneous cutting force, the smaller the scratch depth, and high temperature treatment can reduce the penetration depth of the steel stylus. These data and laws provide important reference for HDR geothermal development and drilling processes to improve drilling efficiency and reduce development costs.