Hello, this is Inho Kim for sound lecture
Sound effects
For a time, game sound in Korea
were out of our priorities
Lack of experience and knowledge on sound development of developers
and sound producers'
lack of efficiency in game sound production
made it hard to do smooth sound production
But as a result of not giving up and keep searching
now we have our own game sound
that does not lose in the world market
But still, a lot of developers
do not have adequate understanding or knowledge of
sound production process
They know it's important
but they don't know why and how
important they are
The reason turned out to be that there weren't enough opportunities
to access related knowledge or experience
So I, through this lecture
want to give you an understanding of sound and the production process
its specific roles and basic knowledges
Game sound is
an emotional medium between a game and the player
Here we have ET
ET's look is not ordinary
but using game sound, we can make him our friend
or a subject of horror
Of course, it can differ depending on the game production direction
but once that's established, sound exists
to decorate the chosen concept
Game sound can be divided into sound effect and background music
Let's start from sound effects
Sound effects mean sounds that correspond to actions
To give an easy explanation about the role of sound effect
let me give you the example of Pinocchio
Grandpa Geppetto wanted a child so he made Pinocchio
But one night, a fairy appeared
and gave Pinocchio life
enabling him to move like a child
You can think of sound effects as the breath of life
given by the fairy
Developers make lifeless objects
and sometimes what seems meaningless but with sound effects
they become a live creature
which helps the player get emotionally involved
This is the reason why we say that sound effects are important
Sound effects are divided into many types
First let's learn background, or ambiance
Ambiance is the sound effect that
depict the place and time using the
normal sounds of each place like the nature, city, or factories
Close your eyes and feel the sound only
People chatting
Broadcasting sound from far away
Train passing
These make us think of a metro station
Wave and seagulls
let us know that we're at a beach
Like this, players can have easier spatial knowledge and
situational information through ambiance
The reason why this is important is because it can show the overall mood
of the scenario and game concept at once
In a horror game for example
let's say that a player walks alone in an old building
Wind blows and from far away, wolves are howling
and you can hear someone sobbing from somewhere
This will multiply the horror
Healing game can be another example
Let's think of Animal Crossing
A bubbly character hops to go fishing
As it arrives at the sea
you can hear waves
and birds like seagulls
The ocean ambiance and bright music that plays
while waiting for a fish to bite on your bait
lets the player have an indirect experience of
fishing at the ocean
Like this, ambiance
is an effective element that lets players indirectly experience
the place inside a game
Ambiance is produced by recording each needed sound
and editing them using sound editing software according to situations
Using a cafe in Korea as an example
grinding coffee, extracting coffee, people chatting in Korean
and the receipt being printed
will be combined into one file
and if we change the location to America
Korean chatting will be replaced
with English chatting
Each independent sounds
can be recorded by sound designers in cafes
or it can be bought from the internet
and one can also buy an ambiance with all the sounds combined
Of course, there are many free sound effect websites
so you can also download it for free and use it
Next is foley
Foley is sound of objects like sounds of door, footsteps
guns, cannons, or cars
and people who make foley sounds are foley artists
Usually you can record them with real objects
but sometimes they are made by combining completely different sounds
This is a person recording foley sound for a movie
This is not much different from making game sound
Foley artists use knives or steel rods
to make the sound of knives crashing
and if they need walking sound on a wooden floor, they walk on wood
and if they need one walking on stone or pebbles
they will walk one stone roads or pebble and record it
I don't know it it still exists but
before, there were drama series aired on the radio
When voice actors acted with voice only
in front of them
there were foley artists expressing the sounds
Foley artists had various objects ready
and if the voice actor narrated that a door opened
they made a door opening sound
and if there's a horse riding scene, they would
make the horse shoe sound with tools or by themselves
The foley artists of that time
could make the sound of three to four horses running at the spot
But sound in real life isn't always expressed exactly the way we hear it
There might be no sound in real life
but there are many instances where fake sound are made up
For example, when hitting a person
try hitting a person next to you with a fist
Then, it doesn't make the thump sound that we know
but a sound closer to pat
In videos, there needs to be sounds corresponding to actions
to have a higher sense of realisticness and emotional involvement
so fake sounds are put in
Usually sound of hitting
are made with electric rice cooker or percussion instruments
and for a more realistic sound
one can record the sound of throwing fists on chucks of meat
Also, the sound of
stabbing people
isn't made by actually stabbing people for the sound
but made in other ways
First, make a hole on a watermelon
Then, throwing a fist into the hole
creates a sound with both 'thump' and 'squirt'
For the sound of
blood when stabbed
one can squish the watermelon inside the hole
to make the blood sound
that we hear on movies or games
Wave sound can be made by shaking a
sieve with beans in it
and the sound of thunder
can be made by shaking metal sheets
and sharp thunder sounds
can be made by ripping vinyl tape apart
and editing it with sound editing software
which makes it sound closer to ones that we hear in movies or games
The sound one game character makes
is much more than we think
Let's see what sounds we need
The sound one warrior makes
is about these
First there will be weapon sounds
There will be close combat weapon like hammer, or knife and ranged weapons like bow and arrow or crossbow
Usually, weapon sounds are called whipping sounds
These sounds are determined by the material, weight, and length of the weapon
Single gun sound is short
light
and big weapons that Guts of Berserk uses
will have a longer and heavier whipping sound
And depending on technology
these sounds will be edited in many different ways
Bow and arrow sound will be
selected among
pulling the bow, letting go of the grip,
or the arrow flying according to the situation
Of course, FPS and stealth games
can use all three sounds appropriately
Hit sounds differ for games' concept and situation
Games like Tekken where hitting is the main fun
have exaggerated hitting sound
and have a heavier feel
while games like stealth game where hitting isn't the main element
have a more realistic hitting sound
Of course, these are all based on the production direction and concept
so sometimes, a realistic character's magnificent vibe
might also be paired with exaggerated sounds
Since there are no rules as to what sound must be used for what situation
various vibes can be created with the right sound of the right situation
Next are sounds for magic, buff, nerf, gems, and level ups
These situations mostly don't exist in real life
so effecters like synthesizers are used to make sounds for these situations
Next are sound effects for magic
First is magic shooting sound
When fireball, lightning, or ice is shot out
the sound these make are called whoosh
and these refer to the sound of fireball being shot out
or spacecrafts, and airplanes passing by
These shooting sound can differ for each shooting scenes
If we say that fire drops from the sky and the character flips twice
and shoots it out
the sound must start from the moment that fire is shown on screen
and must make different sounds for when the character does a flip and when it's shot out
As the magic gets fancier the sound also gets fancier
and magic animation
is watched while the sound editing software follows it for each frame
Hitting sound is also different for each situations like
the explosion sound when it hits fire
or the sound of being stabbed when it hits ice
and based on the effecter video of the game
the sound work is done
Complex magic, buff, debuff sounds
are also made based on the image that the video is showing
As I mentioned before
usually sound is attached to graphic effects
and according to situations
they are adjusted
Footstep sounds are determined
based on who walks on what surface
Every object in the world has its own sound
When we say that we hit substance A with substance B
hitting it with a 1m long B and a 50cm B
and a 10cm B make different sounds
Man and woman walking on the same floor
will make different sounds as well
On the same track, a woman in heels and in sneakers
walking on a concrete ground will make different sounds
and if the material of the ground changes, the sound will as well
Walking and running sounds
since walking has longer times of each foot touching the ground
the sound wave is longer
while running kicks the floor
the wave length is short and attack is stronger
Usually, one file for the left foot and another for the right foot
another left foot sound and that of the right foot
A total of 4 sounds played randomly is recommended
Sometimes, the left and right foot sound are put for the length
of an animation as one file
but this way, for animations that are not predictable
it will fall out of sync
This is because for games, walking forward and backwards
and left and right all have different frames
In the video, the step of this guy running on ice is very irregular
He also slips on ice
For these complex scenes
sound middlewares like WWISE or Fmode is used
to create various footstep sounds
Next we'll see voice sound effects
Attacking and attacked voice are different
and it also differs for the subject of the sound
and of course the kind
If a monster has the shape of an elephant
the attacked and attacking sound
will sound similar to an elephant
If we say that the monster isn't an existing one
we can create virtual sounds using sound design
Monsters like Godzilla or KingKong
aren't existing creatures
so they use sounds that have sounds of similar animals mixed and edited
Story explanation and magic spell sounds
are created for the situations and times they are used
and when voice actors record them, simple script and character image with explanation is given
When recording voices
employees of developers can record their voices
but most of the time, voice actors are hired
to record in studios
It is good to prepare information about the character so that the voice actor can understand them
and make scripts of what the voice actor will record
and send them a few days before recording
Sometimes, voice actor directing is outsourced to professional directors
Imaginary animals and characters
are usually made with combinations of animal sounds
but people also make monster sounds
I have heard that there is a voice actor who does that professionally
When you search on the internet monster sound voice actor
you can easily find portfolio videos of
people recording monster sounds
The recorded monster sound is sent to computers
and edited with professional editing software and then used
Next is about sound design
Non-existing sounds like alien, laser, monster sounds
are made with synthesizer and wave files
by effecting them
You can see this complex images on the screen
which are the effecters of a synthesizer
These sound editing software using
synthesizer effecter are called DAW
and using this DAW, composers compose
engineer mix recordings
and sound designers create sounds
Protool is a widely used software
that is the practical standard of the industry
and Nuendo is also widely used
Nuendo officially provides the sound middleware WWISE
which makes a lot of game sound designers love it
On Star Wars, there is a small robot called R2D2
and it communicates with people making machine sounds
R2D2 sounds are real human recordings
put into synthesizers and edited to sound like a machine
and George Lucas says that he did so to show R2D2's more emotional side
on one of his interview
which shows that this style of sound design
has been being used for a long time
Next is what you should remember when doing sound effect works
First, in games often a lot of sounds play simultaneously
Each sound effects' sound balance
and the sound balance between sound effect and BGM must be checked
If hitting sound is excessively large just because it is a hitting game
it will be hard to harmonize it with other sounds
On the other hand, if the footstep sounds is too big in that game
that might affect the hitting sound
If you're playing Tekken and you can hear footsteps better than hitting sounds
the game won't feel very fun
So the balance between sound effects and BGM must be well established
for the player to not feel decreased fun because of the sound interfering the fun
Personally, I think the balance between sound effect and BGM
is sound effect 6
and BGM 4
The sound effect when hitting many characters
can be a different hitting sound from the original one
Warriors type games have one technique that attacks tens of characters
Assuming that each character make hitting sound
when 30 creatures are hit at the same time
the same 30 hitting sound will play at the same time
This causes the sound phenomenon paging
and makes the listener uncomfortable
so when a certain amount of characters are hit
it is set to no longer make sound
or attack a hitting sound like sound to attacking techniques
For example, for a technique of slamming a knife to the ground
on the effecter that happens the moment the knife touches the ground
an explosion sound can be put
so that even with the absence of other hitting sounds, the impact still exists
You also need to check the standard sound effect for each game
What's the important part of horror games?
It will be the footsteps in the quite ambiance
and some weird sound coming from far away
In a Japanese building falling down
it is quite but outside wind blows loudly
and a weird sound keeps coming from somewhere
and it's hard to see since it's dark
but you can only hear footsteps
This will intensify the horror the player feels
For games like Mario, picking coins up
is the main fun element of the game
Therefore, the bright and clear sound of picking coins up
will enhance the player's pleasant feeling
and for hardcore FPS games, using strong weapons
to crush the opponent is important
Therefore, focusing on the hitting sound
will make the game much more fun and exciting
Like this, sound effects have diverse types and expressions
and if the designer's imagination reaches
it can stretch endlessly
The sound that we usually didn't care
are actually created with this much process and effort
which makes us realize how
important of an element sound effect is in games
Background music
Now we'll talk about background music
Background music refer to everything that
musically expresses the video's background
ambiance, and implied theme
In games, musical expressions
due to the characteristic of games
are different from that of other media
but the process of creating music
is almost the same as other media
like movies or songs
The biggest job that background music can do
will be expressing the video's theme
Let's say there's a war scene
Normally
it will be aggressive and furious
so the music will have the same vibe
but if sad music is used
in furious war scenes
the madness of war
and warriors killing each other to live
in that madness and their sorrow will be expressed more
rather than the aggressiveness of wars
The power of background music is
that it can bring out the inner emotions that are
hidden behind the outer layers
The reason why music is always associated
to famous movies' highlights
is probably because music is very effective in
bringing out the theme of the movie
Game background music
also helps to bring out the inner
emotional layers well
so it is a very important game element
Next is game music production process
Game music production starts after
some game development is done
The first thing to do is planning
Based on the concept or theme of the game
what style and what music will be in what video
is determined by the sound team
Then, following the plan
music is composed, arranged, and sequenced
Sequencing refers to
making music with computers
Once sequencing is done, session recording takes place
Session recording is
recording instrument sounds
Now, with technological development
almost everything can be done with computers
but often they are worse than actual people playing
so each instrument's players
play the composed music
Once session recording is done, mixing is what's next
Mixing is gathering the recorded music sources
and make each instrument's sounds
more flattering
Like the same music sounding better with better speaker quality
after mixing
it is made much better to listen to
When mixing is done, mastering starts
Mastering is
adjusting volumes of sounds to make them consistent
and fix some parts that were lacking in the mixing process
to make it much better to listen to
In game music as well, adjusting music volume balance
and making better sounds are achieved by mastering
Then, the final output is converted into
the desired format
Now I'll talk about some of the most basic common sense about music production
There is something called MIDI
MIDI is short for Musical Instrument Digital Interface
which is the standard signal electronic instruments exchange
Computer and electronic instruments exchange signals
to play music with electronic instruments' sounds
and most pop music and BGM are made with MIDI
Before, electronic instruments were made of heavy hardware
but these days they have changed to software
so that one computer can do the job
Using many resources, it requires high specifications
The photo on the screen shows the connection between computer, synthesizer, and electronic instruments
The computer is connected to the keyboard with a USB cable
and the keyboard is connected to other instruments
If I play Do on the kehyboard
the computer senses that Do
and software
shows how long and when that Do was played
So when you play that on the computer
the identical Do will play
with the same length and timing as the one I played
This shows the forementioned DAW software
It is a software called Logic by Apple, that a lot of professionals use
When I play on the keyboard
this DAW will show that in bars and lengths
If I accidentally played a wrong note
or if I want to play a longer note
you can go to the window in the picture
and edit the length and placement of the bar
so play the fixed note
and not the one I played
Like this, each instruments can be composed and arranged the way I want
and this process is called sequencing
These days, using AI
there are tools that automatically fix parts that are to be played
and on the top of this photo you can see wave patterns
DAW these days sense the waves
and draw them as MIDI file or music sheets
so you can also edit a music
that someone has played
Music that are done with MIDI sequencing
are then recorded in these studios
for the instruments that are needed
On the left is what is called a control room
and the device in front of the chair is called a console
and that is what gathers all the sounds into one
Instrument sounds are recorded using those devices
If a large scale orchestra recording is needed
a large studio is rented to do the orchestra recording
These days, majority of game music don't use MIDI made orchestra
but actually played music
When the recording is done, using these devices
mixing and mastering is done
These days, composer and arranger usually do the mixing and mastering as well
but since there are expertise in each field
professional engineers doing the recording, mixing, and mastering
will be the best way to create the best sound
Conditions of good studios are
good devices, roomy recording booth, high ceilings
and good room acoustic
that can record any sound
These days, with the sound devices becoming software
studios with a lot of devices
are decreasing
Next we'll talk about the final output
When we record
we used tapes
Even when recording pop music
a big multitrack recorder recorded on tapes
and mixing and mastering
was done bu recording on DAT, digital audio tape
The final product
were released as LP, cassette tapes, or CD
but these days, most music's recording and playing
are done with computers or smartphones
Computer making sound
means analog is converted to digital
There is a bit of music theory
that we need to know to work with digital music
What you must remember is sample rate
When you convert to MP3 file
you might have seen 44.1kHz 16bit stereo in converting options
This is a format of wave file
and the 44.1kHz 16bit stereo written here
is the CD sound format
In more detail
it is the theory that when sampling to digitize
when it is sampled as twice the audio frequency which is 20 to 20,000Hz
it sounds similar to the actual sound
but to get into this theory, it is very difficult and out of our field
so just remember this number which is the CD sound sample rate
Sample rate is the frame number per second
and when this sample rate falls to 22kHz
the sound quality falls with it
Bit number is like the 16bit, 64bit that we are familiar with
and if 16bit is decreased to 8bit
the sound quality is damaged with a lot of noise
Stereo means the channel number of sound
Human have two ears, meaning that even the sound from the same source
will be received differently from the right and left ear due to the surrounding environment
Humans get a sense of space due to that difference
and the left photo
shows mono sound wave pattern which means sound being made from the middle
and stereo shows two wave patterns for left and right
If that becomes 5.1 channel
there will be 6 channels with 6 wave patterns
This sample rate is something that you must know if you're a developer
and you must remember it
In addition, the studio standard these days
is 48kHz 24bit stereo
It will also be good to remember that as the sample rate bit increases, the sound quality gets better
But what you must note is increasing sample rate that are low like 22kHz
back to 44 or 48kHz
one might think that this will increase the sound quality
but that in fact it doesn't
If the wave pattern is already lowered to 22kHz
it has lost a lot of sound data
so bringing it back to 44kHz does not bring the deleted data back
Now we'll learn about sound file format
As you can see in the table formats that are not compressed
are WAV and AIFF files
WAV is usually used in Windows
and AIFF is used in Apple's Mac
These two formats are basically the same
And MP3, OGG are both compressed files
FLAC is a format that is compressed but does not have sound quality loss
Usually WAV file with CD sound quality are about 10MB
When the sample rate is decreased to 44 to 22kHz
the size will fall to 5MB
And when 16bit falls to 8bit
it will again fall to 2.5MB
MP3 is about 3MB for 138kpbs
Other compressed files are the same
but MP3 have tags added when converted
so it has a short skips in the beginning when the music is on loop
which is why MP3 is usually not used for looping
When using MP3 industrially
there is a license
so you need to pay license fee to related organizations
OGG does not require that
but uses more resource compared to MP3
which is why it wasn't used much in the earlier days
but these days, with better specifications
OGG is also widely used as game format
Let's learn about other formats
First is Dolby Atmos
It is a 3D surround technology developed by Dolby
and it is the most used 3D sound format in the media market
It is provided in many console games like PlayStation
and games providing Dolby Atmos are increasing every year
and it is a format used in not only the theater but also for normal music
Apple music officially provides Dolby Atmos
It is provided from 7.1 channel or more so it needs more than 8 speakers
and there are headphones that provide Dolby Atmos
but virtually separating channels
has lower quality than
physically separating them
Next we have DTS:X
It is the format used most after Dolby
and it is hard to see in Korea
but in countries like the US where markets are big
Dolby and DTS are usually provided
Dolby Atmos requires accurate speaker placement set by Dolby
but DTS is relatively free
in speaker placement
Multi channel support format
is a great too that can maximize player’s
game experience
but it requires that much initial cost
so it is hard for normal players to use it
To test these two formats
you can easily find them in video streaming sites like Youtube
Music made this way
can be expressed as scripts
using tools provided by game engines
but using sound middleware
it can be done in a much easier way
Fmode and WWISE are two famous middlewares
They are the two most used middlewares
and they are free to use for normal users
And to use it industrially
you must pay license fee accordingly
Characteristic of sound middleware
is that it enhances the project audio quality
and provides direction for various situations
so a more detailed production is possible
Resource is used more efficiently
and the programming time is decreased
And 3D sound is provided
and the same audio standard as Dolby Atmos is also supported
which makes higher quality sound production possible
It can also make adaptive music or other various sound effect
Adaptive music means music changing according to the game situation
For example, when playing an FPS game
when a player is walking an empty forest
only ambiance effect plays without any music
but if an opponent shows up and a fight starts
the music switches to fighting music
When all the opponents are defeated the music fades out
and only ambiance sound is left again
For a more complex production
you can think of Silent Hill
On Silent Hill, if there’s a monster near you
radio noise starts playing
If your close, the sound is loud
and if you’re far it gets smaller
When you’re in a building alone and can’t see well
as your anxiety rises
and if the radio sound starts playing
the player can’t see
but will become more nervous thinking that there’s a monster near
And making a turn at a corner
and the radio noise intensifying makes them think that the monster is nearer
and gets them more nervous
Then end up facing the monster
and even if they fight off all the monsters they can see
the radio still doesn’t shut off
it means that there are still monsters near them
and the player can’t be relieved
This type of production is called adaptive music
It can also randomly play footsteps and gunshots
to eliminate boring or bothering elements
It can also make different footstep sounds for
wooden, metallic, stone floors
and according to the floor’s conditions
What’s important here is that
these production are done very easily
So the usage of sound middleware is increasing
About Fmode
first, the UI is similar to that of DAW software
and users who are familiar with DAW can easily use it
It is also possible to connect with a DAW called Reaper
so if you use Reaper
an easier Fmode project is possible
It is easy to use
so beginners can use it
and it supports many programmers
WWISE is more professional than Fmode
and can be connected to Nuendo which is a DAW that many people use
But it is quite difficult to use
and has less programmer support
but has great functions which is why it is one of the most used tools
What you’re seeing now is
an example script using Fmode in Unity
If you use a sound engine that Unity provides
there will be many lines of codes like you see in the before
but if you use Fmode
the same effect can be achieved with just a few lines
We talked briefly
about game sound
To conclude, game sound
plays a huge role in
making the game player get more emotionally involved in the game
Strong emotions are made stronger
and rests are made more peaceful
through game sounds
And game production
takes many processing steps by a lot of professionals
If poor pre production is done
a lot of difficulties and monetary loss will occur
Lastly for diverse expression
sound middleware can be used
to do more detailed and fancy sound production
I have worked in sound production for about 28 years
and have done lecture in educational organizations to
let game developers know of the trials and errors
that I went through
I hope my experience
helps those of you who are
just getting into game developing
I hope you achieve what you aim
Thank you
Sound effect Types of sound effect Background music: expresses location and time
Foley: expresses sounds of objects
Sound effects in games Weapon Sound: differs for weapon size Magic sound: shooting sound differs for elements
Footstep sounds : differs for gender, floor material Voice: if not of existing creature, sound can be created
What to consider when working on sound effect Balance between each sound effects and between sound effect and BGM When hitting many characters the sound effect can be different from the original hitting sound
The standard sound effect for each game must be noted
Background music Game music production process Planning -> composing, arranging and sequencing-> session recording-> mixing -> mastering -> converting
MIDI(Musical Instrument Digital Interface) Standard signal electronic instruments exchange
The End