The world around us offers the most complete of the sensory experiences available. We perceive reality – as much as our limited and sometimes damaged senses allow – and are able to make decisions, learn or just enjoy what our surroundings offer. Something as mundane as a waterfall or a beautiful landscape is smell, hearing and even touch, as much as it is sight. There is however a human pursuit to not only understand how our senses work, but also imitate how nature already feeds us with the most amazing events. Hence the term ‘multisensory experience’ was coined.

According to Marianna Obrist and Carlos Velasco, researchers and experts in everything related to multisensory experiences, these are “impressions formed by specific events, whose sensory elements have been carefully crafted by someone.” In other words, how an event planner – be it a concert, an art exhibition, a VR simulation or a movie premiere –  selects which scenes to stimulate and how they will interact to achieve an entertaining, memorable and fun experience.

A great example of this type of experience (of course in the realm of hearing, since that’s what we love to do!) is ‘The Sound of the Sea’ by chef Heston Blumenthal in collaboration with gastrophysics Charles Spence. This dish, consisting of sashimi, tapioca edible ‘sand’ and sea foam, is served with a conch shell with an iPod in it. You put your headphones on, and then proceed to enjoy this Fat Duck’s signature delicacy, while listening to waves crashing and seagulls flying in a binaural experience that enhances all senses. There is taste, smell, touch, sight and of course, hearing.  

Simple, yet elegant, this multisensory experience is perfect to explain how one may use sound to enhance the experience of something as mundane as eating. But one can definitely go overboard, and include all sorts of technology like sensors, electronic controls and virtual reality…

A good example of this is ‘Tree’, a VR project which has been presented in festivals such as Sundance and Tribeca, as well as organizations such as the World Economic Forum and the United Nations. This experience lets the users place themselves in the shoes (or roots?) of a tree, from a seed state and into its grown form. In technical terms, the VR set has hand controls that let you experience the 360 degree ambience the developers created, as well as see your arms and body as branches. According to their site, people have even left crying from the experience. Particularly the ending which we won’t spoil!

There is so much to dissect about this particular project. Not only does it guide us to reflect upon our relationship with nature – as well as make us conscious of how little we know about the plant and tree experience – but it is a very interesting example of how entertaining and educative technology and multi sensory experiences can be when they join forces. And wait to see what they create when science gets involved in the equation.

Gastrophysics is one of the most exciting trends I have crossed paths within the last years. According to Charles Spence, head of the Crossmodal Research Laboratory at Oxford University, this branch of psychology is “the scientific study of those factors that influence our multisensory experience while tasting food and drink”. This is basically how our senses interact with taste. Associations between color of the plate and sweetness in a frozen strawberry mousse, assessment of how we can improve cutlery or the correlation between high pitch sounds and our perception of ‘crunchiness’ in chips, are all topics questioned and studied by gastrophysicists. One particular multisensory experience that caught my attention as per the application of this field of science in a project is ‘Hear the Taste’ by Ultra Nordic for Finnish airline Finnair.

Finnair wanted to improve the experience of their Chinese customers. Based on results found by gastrophysicists – including the fact that food tastes different while on an airplane several hundred meters off the ground – the airline modified their menu  and embarked to record sounds of Nordic nature to be played by customers while enjoying the carefully calibrated menu. Audiences at ground level could get an app that allowed them to take a picture of the meal, and an algorithm would suggest a soundtrack to accompany their food. Talk about innovation and multidisciplinarity. 

This example goes to show what we can achieve when art, science and technology meet. We can now answer questions not even asked before, in order to improve our experiences and even widen our knowledge of other cultures and societies. Multisensory experiences allow us to connect like never before. And music is right at the center of it, collaborating with the rest of the senses!

There are thousands of examples of how multisensory experiences are allowing us to create new and exciting ways of enjoying an event. We are slowly ceasing to be bound by the limits of technology, and approaching levels of interaction that help us connect like no other time in history. Multisensory experiences inspire us to create new forms of interaction between the art forms. They help us think about the future of our planet and reflect upon nature and how we relate to it. They are a bridge between cultures and how we learn about them. Sensory exploration (used for good) can lead to music and art of the XXI century, which must be entertaining, educational and responsible. After all, aren’t our senses how we access the world? 

Leave a Reply