How sound design is transforming UX

[Rss Bot 1]

Sound is a critical element when creating an engaging and compelling user experience.  If you need proof, try playing your favourite game or watching your favourite movie with the volume turned right down.

Ambient city noises, distant sirens, and low rumbling hums can drop you into a scene, even without the visuals. Sounds create the mood and set the pace of an interactive experience. Music enhances suspense, ramps up intensity, and ultimately triggers an emotional response, and does so much more powerfully than visuals can alone.

In this article, we'll explore the role of sound in web and app design. We'll look at how sound can enhance an experience, as well as what the different types of sound are, where to find sounds to use in your projects, and the audio libraries you should try out.

How can sound improve online experiences?

Immersive sites, such as world-building models, in-world web experiences and games can be greatly enhanced by sound effects and musical scores. Sites and applications with engaging animations, dynamic visuals or 3D, depend on solid sound design. But even the simplest sites can benefit from user feedback and meaningful signals through audio.

Otherworldly sounds immerse the user on Pottermore (click to visit the site)

In your projects, sounds can form part of the reward system in completing tasks. For example, sound effects that trigger with completing a puzzle make the act much more satisfying. A pleasant chime as a bar fills up, or as the user achieves a specific goal – these all work together to give short-term rewards, while the user works towards a larger goal.

Sound can provide audio feedback, to complement visual feedback. Buttons make noises when clicked, a clicking metronome can count you in before recording. Sounds can help reinforce that something was done as a further way to acknowledge the user’s actions. A way to say ‘yes’, your interaction was received. 

In some cases, sounds can call out something in a visually busy landscape. In time-critical applications, they can communicate something faster than visuals might alone.

Paper Planes uses audio to add a new layer of interaction (click to visit the site)

User interfaces, since the earliest days of the command-line, have used sound effects in the form of simple audible beeps, to warn of errors. And today, apps such as Twitter or Facebook use well-placed sounds for notifications and alerts. Skype uses audio to facilitate communications. 

As we move into technologies away from screens, speech and gesture interfaces are increasingly dependent on audio input and feedback. Assistant software and tools such as Alexa and Siri give audio feedback fuelled by powerful artificial intelligence. Audio cues and speech recognition can also be used to create a better experience for those with disabilities or learning difficulties.

With ever increasing support, bandwidth and networking power, now is the time to put the power of sounds and music to work in your sites and applications, through solid sound design.

What types of sound are there? 

Music/score
Imagine your favourite movie without the iconic score. Jaws without the ominous two-chord warning, Star Wars without the incredible score, driving the action from the first iconic frame. Great music has its own life; it’s essential to the success of the production. 

Ambient Sound
In film and game design this is also called asynchronous sound. It is not directly synced with other actions, but forms a mood bed or atmospheric soundscape. Crickets chirping, a crackling fire or sounds of the city at night. Ambient loops that set a tone for the experience. Of all the types of sound, this has the strongest ability to subconsciously transport the user into the world of the experience.

Sound FX
Every interaction, achievement unlocked and warning can be enhanced with sound effects. They can reinforce the tone of the experience, and communicate greater detail quickly to a user. A clear warning tone might signal a low battery, before the user notices the flashing bar. A gentle bell or satisfying click when pressing a button can encourage and make an action more enjoyable. The nice pop effect when you pull down on Twitter to refresh results is a great example. 

Voice
Voiceovers can humanise an experience, guide a user with instructions, and serve the functional role of communicating important information in the absence of visuals. For example, a navigational app for driving or an instructional tool. Pitch recognition for tuning an instrument or voice recognition to receive commands are just a couple of possibilities. 

Voice assistance through apps such as Alexa or Siri are rapidly becoming part of our lives, from using a phone, operating a vehicles or setting the mood with some music at home. Voice recognition in sites and apps is growing and will continue to grow.

The psychology of sound

Sounds are able to create powerful reactions on deep, instinctive levels. A baby’s cry can set a new parent into action without conscious thought, a sharp warning hiss of a snake can trigger adrenaline, and hearing that favourite song on the radio might bring a smile before you realise it. 

Scientists and thinkers have been pondering the relationship between sound and emotion for thousands of years. Modern day neuroscience has uncovered amazing insights into the relationship between our ears, minds and emotions.

Dramatic audio creates tension on The War of 1996 site (click to visit the site)

Deep within the brain, in the amygdala, we processes memories and emotional reaction, while the Hippocampus controls behaviour and helps form memories. Within these most primal sectors of the mind, responses are triggered when we hear sound. Breathing, heart rate, brainwaves and hormone secretions are continuously affected by incoming audio. Sounds are associated with experiences and the feelings connected to them, both good and bad. 

The right music or ambient sounds can instantly connect an audience to a flood of associated feelings

How someone feels at any given moment is very often affected directly by what they hear. People have the ability to remember hundreds, if not thousands of songs and voices. Sounds are remembered more easily because of the impression they make on your mind, and they also are much harder to ignore. Hearing someone say your name across a crowded room, but seeming to ignore everything else, is just one example.

Film and game-makers have capitalised on the psychological power of sound for years. They know how sound can evoke emotions. The right music or ambient sounds can instantly connect an audience to a flood of associated feelings. Everyone has that favourite game theme song or movie score. As soon as they hear it, they recall how they felt when they played it or saw it the first time. Perhaps it’s the iconic Super Mario Bros theme or the soundtrack from Azeroth in World of Warcraft. 

The right sound design, the right music, can not only set the mood, but emotionally connect to people in a way visuals alone simply cannot.

How can sound affect emotions?

Brain stem reflex
When loud or dissonant sounds are heard, they signal a potentially important and urgent event, causing us to react on an instinctive level. Notifications, beeps and sirens are examples.

Learned response or conditioning
When we have heard a sound repeatedly in a certain situation, it can often lead to an association between that sound and situation. Hearing it again can instantly elicit similar emotions or feelings. 

Emotional contagion
Perceiving emotions expressed by a piece of music. It doesn’t have to sound sad. Instead we recognise it as conveying ‘sadness’. Sound is so fundamental, that it often transcends languages. People from diverse cultural backgrounds often agree on whether a piece of music sounds happy or sad. This has led some to term music the ‘language of emotions’.

What are the challenges of using sound?

Download size and streaming limits
Sound will increase the weight of your page. We can overcome this a few ways. First, optimising sounds through good compression that balances quality, size and bit rate. MP3 format is going to give a good balance of these factors, and audio tools will give you options for levels of compression. Stream larger files instead like music and ambient tracks. Load FX for instant playback.

Unexpected intrusion
No one likes to be caught by surprise, or feel like they’re not in control of their experience. Give users the option to disable sound and control levels. Let them choose their experience, and they will often choose to listen.

Mobile limitations
Mobile offers new challenges. On most mobile devices, audio will not play until triggered by a user action. A simple solution is to start the experience with scroll or click event, that you can then bind your audio start to. Initialise ahead, and wait to fire load and play events the moment you get that first interaction.

Sound overload
It’s easy to overdo things when it comes to sound. Not every action needs a sound effect. Sounds should serve a purpose. Use music and effects that inform or enhance the experience. Just like a good visual designer, don’t be afraid to cut. Too much of a good thing can still be too much.

What are the major audio file formats?

MP3 - A compressed or ‘lossy’ format. Audio data is compressed and optimised for size and bit rate. This is the format you want to be using as often as possible.

WAV – The WAVE container format is an uncompressed audio file and has much larger sizes typically. Files in the WAVE container format typically end with the '.wav' extension.

FLAC – Uses lossless compression, meaning silence does not take up file space, but the uncompressed audio is retained. This is a larger file size, (smaller than wav), and not as well supported.

Ogg Theora Vorbis – Supported in desktop/mobile Gecko (Firefox), Chrome, and Opera. Support can be added to Safari (but not on iOS) with an add-on. The format is not supported in IE in any way.

Where can I find sounds for my projects?

Freesound – A collaborative database of Creative Commons-licensed sounds. You can browse, download and share sounds.

SoundJay – Library of free sounds you are allowed to use on your website, free of charge and royalty free in your projects.

Partners in Rhyme – Free and royalty free music and sound effects.

Pond 5 – One of the world’s largest collection of royalty free stock video and creative assets, sounds and music.

Premium Beat – High quality curated royalty free music library, organised by genre and mood.

Audio Jungle – Massive library of royalty free sounds and music from $1.

Sound Stripe – Membership model, for unlimited licenses for songs in large, curated music library.

The best sound libraries to try

Howler.JS
A solid, audio library for the modern web. A solid cross-platform solution, for managing multiple sound effects easily, with good documentation to get started quickly. Includes a wide range of file support, use of audio sprites, spatial audio for 3D experiences, automatic caching and has no dependencies.

Pizzicato.JS
Another well documented Web Audio JavaScript library. Pizzicato has a simple set of methods for creating cool effects, using sound inputs, and managing sound files. Compressors, stereo panning, tremolo, reverb, flanger and many more effects are included, to start creating cool effects out of the gate.

SoundJS
A JavaScript library that provides a simple API, and powerful features to make working with audio extremely simple. Designed with cross-browser support in mind, it gracefully falls back to HTML audio for wide support. Easily ties in audio file loading to PreloadJS. If you are already using CreateJS tools, this is a solid choice.

This article was originally published in creative web design magazine Web Designer. Buy issue 275 or subscribe.

Read more:

• 9 ways to smash UX on a small budget
• UX for emerging experiences
• Infographic: How people really use voice assistants

View the full article