Multi-Modal Mediated Touch
Multimodal social touch refers to the combination of various sensory modalities, such as touch, visual cues, and auditory stimuli, to enhance social interactions and communication. It encompasses the rich interplay of physical contact, facial expressions, body language, and vocal intonations that are integral to human connection. When engaging in multimodal social touch, individuals may experience a range of sensations, from the warmth of a handshake to the comforting embrace of a loved one. These tactile interactions not only convey emotional support, trust, and empathy but also serve as important nonverbal cues that can deepen social bonds and foster a sense of belonging. By integrating touch with other modes of communication, multimodal social touch amplifies the depth and richness of interpersonal exchanges, facilitating understanding and emotional connection in profound ways.
We designed a crossmodal vocalization-haptic system to allow users to communicate emotions to their partners. We explore affective context as a combination of user relationship (specifically the closeness between pairs of users), and user culture. We share the design and implementation of the crossmodal system that takes up to ten seconds of vocal expression (including humming or singing) from one user and transposes it into haptic signals to be displayed to twelve vibration actuators worn on the forearm of the second user. Our method of transposing musical vocal inputs captures the key signal features of rhythm, amplitude, time, and frequency. We present the results from a human subject study (N=20) involving 10 pairs of users with varying levels of closeness (ranging from siblings, friends, and strangers) to understand how our system supports affective communication. Our results show that audio parameters such as low-level and rhythm most strongly influence affective responses in our users. Additionally, the low-level vocal features are influenced by user demographic and the closeness between the pairs of users. The results suggest the impact of user closeness on affective communication.
Affective Communication
Touch as a modality in social communication has been getting more attention with recent developments in wearable technology and an increase in awareness of how limited physical contact can lead to touch starvation and feelings of depression. Although several mediated touch methods have been developed for conveying emotional support, the transfer of emotion through mediated touch has not been widely studied. This work addresses this need by exploring emotional communication through a novel wearable haptic system.
The system records physical touch patterns through an array of force sensors, processes the recordings using novel gesture-based algorithms to create actuator control signals, and generates mediated social touch through an array of voice coil actuators. We conducted a human subject study (N = 20) to understand the perception and emotional components of this mediated social touch for common social touch gestures, including poking, patting, massaging, squeezing, and stroking. Our results show that the speed of the virtual gesture significantly alters the participants’ ratings of valence, arousal, realism, and comfort of these gestures with increased speed producing negative emotions and decreased realism. The findings from the study will allow us to better recognize generic patterns from human mediated touch perception and determine how mediated social touch can be used to convey emotion. Our system design, signal processing methods, and results can provide guidance in future mediated social touch design.
Social Touch
Touch is an essential component of our interpersonal relationships, serving to communicate emotion, improve social bonding, and reduce feelings of isolation. Our lab studies how people naturally communicate to one another via touch with the goal of creating technology that will allow people to communicate over long distances in our increasingly digital world.
Humans use a variety of different gestures in social interactions, including squeezes, pats, and strokes. We have created a wearable device to create the sensation of a pleasant, calming stroke on the arm. Because it is mechanically difficult to create a long lateral motion in a wearable device, we instead create the illusion of lateral motion on the arm using an array of actuators that only have a small amount of vertical movement. These actuators, which are re-purposed excited speakers, are controlled to sequentially press into the user’s arm. We control the duration of each actuator’s motion and the delay between the onset of motion for adjacent actuators. By carefully tuning the duration and delay, we can create a pleasant and continuous sensation that creates the illusion of a hand stroking along the arm.
Human-Robot Social Touch
Touch between people is essential for forming bonds and communicating emotions. However, it is currently missing in human-robot interactions due to issues with reliability and safety. As robotics transitions to home and service sectors, it is increasingly important to design guidelines and models for human-robot social touch. This research aims to determine how variations in the motions a robot uses while patting the user’s forearm or shoulder affect perception and acceptance of the interaction [6]. We are using a Sawyer robot with a hand-like end-effector, varying the force, speed, location, and hold duration of the pat and trying to find safety, valence, arousal, and dominance related to each pat condition. Using these results, we propose guidelines for creating interactions that feel safe and non-dominant using low speed and low force trajectories. We also determine patting signals that can create a mechanical (low speed, high force), human-like (slow speed, hold at the end), or attention-getting (high speed, low force) sensation. These results will be useful in helping HRI designers create appropriate human-robot social touch interactions.