To enhance the oxidation stability and tribological properties of soybean oil, we synthesized a novel environmentally benign, modified soybean oil (MSO) by incorporating matrine into soybean oil under solvent-free conditions. Furthermore, we experimentally investigated the molecular structure and tribological performance of MSO. Results reveal that the long-chain links of the N–H bonds is formed between matrine and soybean oil molecules through electrostatic interaction, rendering MSO with excellent thermal oxidation stability similar to matrine. The average friction coefficient and wear volume of MSO decreased by 40.5% and 76.4%, respectively, compared to soybean oil. Further, MSO exhibits excellent friction-reducing and antiwear properties compared to PAO. Raman and X-ray photoelectron spectroscopy analyses reveal that MSO containing nitrogen heterocycles is more inclined to penetrate the friction contact zone and form a physical adsorption film. Simultaneously, a chemical adsorption film comprising nitrogen-containing metal complexes and iron complexes was formed during the friction reaction.