Design/methodology/approach Wear debris analysis is considered to be one of the most effective methods to maintain the condition of mechanical equipment. In this paper, the friction and wear testing machine was used to design pin-disk rotation, pin-disk reciprocation and four-ball test to produce cutting, sliding, laminar and fatigue debris. A semi-online sampling system was designed to collect ferrographic images containing various fragments. The images were rotated and flipped to augment the data and enhance the generalization ability of the model. The data set required for data analysis is established. Using COCO pre-trained Mask R-CNN data set as a benchmark, the region proposal network (RPN) is trained with labeled wear debris images to enhance the ability of RPN to recognize background and wear debris. Two transfer learning scenarios are tested in the network head of the Mask R-CNN. Findings The results show that the deep convolutional neural network is suitable for the automatic classification and detection of wear fragments. Through transfer learning and proper training configuration, the ferrographic image recognition based on Mask R-CNN achieves high accuracy. Originality/value The results show that the deep convolutional neural network is suitable for the automatic classification and detection of wear fragments. Through transfer learning and proper training configuration, the ferrographic image recognition based on Mask R-CNN achieves high accuracy. Peer review The peer review history for this article is available at: https://publons.com/publon/10.1108/ILT-05-2024-0182/