Design/methodology/approach Helical gears of 20CrMnTi were lubricated with conventional oil and nanoadditive oils. An open helical gearbox with spray lubrication was tested under different speeds (200–500 rpm) and loads (20–100 N·m). Gear noise was measured by a sound level meter. Axial and radial vibrations were detected using an M+P VibRunner system and fast Fourier transform analysis. Vibration spectrums under conventional and nanolubrication were compared. Gear tooth surfaces were observed after testing. The experiment aimed to analyze the noise and vibration reduction effects of nanoadditive lubricants on helical gears and the sensitivity to load and speed. Findings The key findings are that nanoadditive lubricants significantly reduce the axial and radial vibrations of helical gears under low-speed conditions compared with conventional lubricants, with a more pronounced effect on axial vibrations. The vibration reduction is more sensitive to rotational speed than load. At the same load and speed, nanolubrication reduces noise by 2%–5% versus conventional lubrication. Nanoparticles change the friction from sliding to rolling and compensate for meshing errors, leading to smoother vibrations. The nanolubricants alter the gear tooth surfaces and optimize the microtopography. The results provide a basis for exploring nanolubricant effects under high speeds. Originality/value The originality and value of this work is the experimental analysis of the effects of nanoadditive lubricants on the vibration and noise characteristics of hard tooth surface helical gears, which has rarely been studied before. The comparative results under different speeds and loads provide new insights into the vibration damping capabilities of nanolubricants in gear transmissions. The findings reveal the higher sensitivity to rotational speed versus load and the differences in axial and radial vibration reduction. The exploration of nanolubricant effects on gear tribological performance and surface interactions provides a valuable reference for further research, especially under higher speed conditions closer to real applications. Peer review The peer review history for this article is available at: https://publons.com/publon/10.1108/ILT-07-2023-0220/