Rapid advances in robotics have placed urgent demands on more intelligent human-machine interaction technologies. Specifically, the way of establishing dual-way intuitive communication with a consistent sensory system can greatly enhance efficiency and reliability. Here, a bi-directional human-machine interface (HMI) is designed by applying starch-based hydrogel sensors. The whole system consists of a multi-modal wearable sensory exoskeleton with a haptic feedback module and sensory robotic hand. The sensory exoskeleton with strain-sensing glove and rotation-sensing arm can capture and project the motion of the entire upper limb. The system offers object recognition functions by utilizing a sensing array on the robotic hands and machine learning algorithms, which can identify the shape and hardness information. The recognized results can be delivered back to the operator via vision and vibrational haptic feedback, respectively. This dual-way intelligent sensory system shows potential application in many key fields such as the Internet of Things, teleoperation, and medical robotics.