Hello
My name is Hee-seok Jeong, and I will be giving a lecture on Motion Capture
I have been working on motion capture for 24 years since 1999
I've worked on projects in various fields such as
movies, games, music videos, and motion analysis
I currently run a motion capture studio for Westworld
Nowadays, movies like 'Avatar'
and 'Planet of the Apes' use motion capture
When I first started motion capture
There was no manual or barely any data on it
I had a hard time
Now you could learn them from
online or even Youtube
Still, I think it is very meaningful
to have this time
to have this opportunity where I can
share my knowledge of motion capture
My lesson on Motion Capture
will be divided in 3 lectures
Rather than talking about motion capture's history, theories, or manuals
I plan to give a lecture based on my experiences
for your overall understanding
Also the concept, types, and areas of use of motion capture
and how the field of motion capture has expanded
due to technological advancements
The first and second lectures
will be heavy on theory and explanation
they third one would be based on videos
Once you've been through all 3 lectures
I think you would be able to gain general understanding of
how motion capture works, the advancements of technology
how filming is done in Korea and abroad these days
and how data is produced
The first part is understanding motion capture
This time, I will explain and help you understand
the types of motion capture, the fields of use
and the technologies required for them
Lesson objective is to understand the concept of motion capture
by learning the basics
they types of motion capture, and its fields of use
We will learn application techniques and basic skills
required for each type of motion capture
Understanding of Motion Capture
Motion capture refers to recording all moving objects
including people, animals, tools, and cameras
in a computer-usable format
It is used when you want to apply the actual movement
of a person or object as is
or when we have to film a difficult movement
or a dangerous scene
Most motion capture involves attaching
markers or sensors to people
That is to apply it to 3D characters
We also attach sensors to
animals or tools that people hold
weapons, like knives or guns
to extract their data
Sometimes we attache them to actual cameras
to use as virtual cameras and extract camera angles
Motion capture was first created for medical reasons
and was advanced greatly in fields of
virtual reality, movies and games
Recently, most of the content using computers
is created as 3D images
The implementation of virtual reality and virtual characters have emerged
These things created an opportunity for us
to easily access them in real life through
the use of motion capture systems
Types of Motion Capture
Next we will look into the types of motion capture
Motion capture works by attaching markers or sensors
to various parts of the actor's body
determining the position values of them
then animating the movements of the 3D character, or digital character
Motion capture is usully distinguished
by the type of sensors
The number of sensors
determine the expression of
natural movements
It is largely divided in to 4 types of systems:
Acoustic, Mechanical, Magnetic and Optical
I will explain them one by one
First is the Acoustic system
It consists of multiple ultrasonic generators and three receivers
Ultrasound waves are generated from a device attached to each joint of the actor
and this system calculates the time and distance
compared to the speed of ultrasonic waves
The position of each device in 3 dimensional space
can be implemented by the triangulation principle
using values ​​calculated from each of the three receiving devices
The downside is that it is difficult to simultaneously measure
the positions of all ultrasonic generators
attached to the actor's body
and the size of the generator device is
big and has cables
which may result in unnatural and awkward movements of the actor
The second is the Mechanical system
This method attaches articulated steel equipment to actors
This steel device has variable resistance
attached to each joint
allowing the mechanical device to measure
how much a person bends their arm or leg
Simply put, the variable resistance is
a sensor, like an electromagnet
Installation and operation is easy and is not limited to space
so we could move far away to capture images
and has advantage of
very high sampling frequency
making it easy to use in real time
These are actually from 20 years ago
Now we have better equipment
and can collect real-time data
with higher qualities
The mechanical system is actually not very much used now
Mechanical equipment was used
in games liek Tekken because it was very difficult
to hold the keys
for such movements in the past
The downside is that the equipment
going on the actor's body is very uncomfortable
that we cannot produce natural movements
and has lower accuracy
This is the steel equipment used
This actually weighs about 10 to 15kg
Which makes it hard to do some movements
The acoustic and mechanical systems together
have lots of disadvantages
so it is not very much used now
But about 20 years ago
when we had to use motion capture
for extreme movements
Acoustic systems were almost never used because of their difficulties
So we had to use the mechanical system, although equipment weighed 10 kg
we had to make up for its short comings
and extracted data of the movements we needed
Third is the Magnetic system
This is a method where a magnetic field generator is placed
in a fixed location and a person attaches small magnetic sensors
to each joint to measure the position and angle of each joint
In this method, the magnetic sensor is
connected to the collector
and then from the collector to the computer
Magnetic system is easy to operate
There is no need for investment in studios other than equipment
real-time processing is convenient as the amount of calculation is not large
and there are no shielding issues
In Optical systems, the camera
may block the sensors
Gyro does not have those issueus
post processing is also not needed
One downside is that the extracted data may be a bit inaccurate
As you can see for Magnetic systems
the sensors are already mounted on the suits
so it is easier for actors to make movements
Gyro sensors are used mostly
and finger data could also be extracted by wearing gloves
This gloves have
small gyro sensors in every inch of the fingers
So this equipment can gather
detailed data of the finger joints' movements
I've pointed out that the data extraction may be
a bit inaccurate
Magnetic system sensors only extracts rotation data
For animation, there must be translation
or position value in the note part
but the magnetic system
does not provide position data
So, we have to forcefully make that with the rotation data
For example every time we move from
the pelvis to the knee, from the knee to the ankle, etc.
the error is taken as a square rather than a plus
Therefore the data is bound to be inaccurate
at the toes and fingertips
The more bones we have
In the case of feet, it is about stage 4 or 5
the hands are from root to the waist
The waist has about 3 joints
then the shoulders, elbows and wrists
This takes you through steps 6 and 7
In that case, situations like the feet flowing
or hands going out of sync
may occur
This is why additional processes
for secondary key work are required
This may be the difficult part
in animation after processing
The last one is the Optical system
The Optical system attaches highly reflected markers to actor's body
and captures the image with a camera that emits infrared light
You are able to move your body more freely compared to other systems
and it could almost perfectly capture movements
even the fast fighting movements
However, the equipment is very costly
and is sensitive to surrounding lighting
It is still widely used for its
accuracy in animation data
It has lots of space limitations
and does need extra time for post-processing
The post-processing
is not just a few hours step
It includes calibrations of the captures
and if the camera is
not installed correctly
It could take days or even months to fix
So the optical system requires
a huge amount of effort in the installation process
The optical cameras detects infrared rays
but the sunlight also has infrared lights so we need to over them
So in case of studio shooting, even if they are underground
we need to make sure to cover the windows
If not, they could generate lots of noise
that results in inaccurate data
and thus the post-processing clean up time will increase
In other words, the optical camera's principle
is to emit infrared rays
receive the infrared rays again
Sunlight contains small amount of infrared
and when this is exposed
The cameras will pick up both
the sunlight and markers and thus create noise
In the past the noise level was not twice or three times
but it was about 10 times the noise we have now
The clean up time was
significantly lengthy
Now we have more experiences with the equipment
and also lots of advancement in technology
that we have less noise
but still the clean up time is inevitable
Therefore, installation must be very sensitive
to things such as
exposure to sunlight
As you can see here the markers are attached on top of the suits
and we can also add the head mount
for facial capture
Recently, technology has emerged to extract
motion capture data using video cameras
Multiple cameras are installed
and these are not an optical, but cameras that
record everyday videos like a GoPro
about 4 to 8 cameras are set up in a circle
From here, we don't extract data from
gyro sensors or markers
but extract the person's movements
from the video to make a skeleton
Of course, this method
may lack accuracy compared to those that use sensors and markers
but this part can be made up by the AI technology
Securing deep data for deep learning is key
In the future, it is expected that these technologies
will continue to develop
so that motion data can be more easily obtained
Fields of Use - Motion Capture
Before motion capture was used in film, animation and games
they were widely used in medicine, sports
education and ergonomics
Walking, for example
We would film steps of a person with joint discomfort
from the front, side, from behind
and from above
and then analyzed while watching the video
However, it was lacking in detail
in terms of time and accuracy
By using marker data while using the optical system
you can measure rotation values ​​and position data for joints
and furthermore, speed and acceleration
Therefore, very detailed and highly accurate data
can be calculated
Rehabilitation doctors
or those who manage these data
can receive data on rotation or position values ​​in Excel
easily measure them
and calculate the data
And we developed from there
to film making and games
For movie production, motion capture is mainly used
in scenes where acting may be difficult
or where a large number of people is needed
in the film
Sports analysis is an important field
of motion capture use
Motion capture data has been carefully categorized
so that different elements of each movement can be studied
Sports and activities such as track, golf swings
swing analysis for tennis
and even swimming in water
all use motion data
to analyze movements and improve the performance of athletes
In medicine, it initially began to be used as a gait analyzer in orthopedics
and rehabilitation departments
Currently, it is used for spinal analysis, design of prosthetic hands and limbs
calculation of biomechanical data
and these data are widely used for physical therapy
and rehabilitation treatment of patients
In the industrial field
it is used in the field of ergonomics
necessary for robot development and automobile development
For example when making a car
the height, handle distance, etc.
The level of comfort for the passenger is
measured and developed
using motion capture equipment
Markers are also attached to the car's bonnet and doors
to measure how much the gap fat vibrates
to help with development
I actually tried measuring that
and I found that the vibrations last
5 to 10 seconds
These measurements help with the development of cars
I also had an experience of measuring a robot arm
When assembling a car
there are times when you need to measure a robot arm
that must always maintain the same motion
In the past they made camera measurements
by attaching small bristles
In this case, if we measure in one direction a day for one month
we would only end up with 30 direction data
But with motion capture, you can obtain 360 degrees
of data with a single measurement by attaching a small, 10cm marker
The accuracy is about 0.1mm
It's completely different from video extraction
Like this
Motion capture can replace
a year's worth of measurements
I've prepared two videos
In this video, motion capture technology
was used in developing hiking wear
Sensors are attached in 5cm distances on the body
then data is gathered by doing climbing motions
This is an example of how
the data on how much the arms and legs stretch
is quantified and used in climbing clothing
The data is made of
measurements on the bare body
and measurements after clothing
and the difference of those two
As a result, we can put into numbers
the stretches of the bare body
and then with clothing
and use the error between numbers to
make the developments
Let's watch the video
Developing Climb-wear Using Motion Capture Technology
In the next video, we motion captured some ski movements
after installing the optical system in the ski park
The purpose of this experiment was to measure
the ski turning movement of normal people
and that of disabled people
The difficult part here was
filming in optical system in day time
As said before, noise will be created in sunlight
and the snow accumulated on the ground has reflections
which results in more noise
So filming had to be done at night time
The next difficulty was the coldness
because it was even colder in the night there
It felt like negative 15-20 degrees Celcius
Shooting in that weather
attaching all the markers was very difficult to do
We had to keep the cameras running
during daytime so it would not freeze
We managed to finish them and gather good data
I will explain after watching the video
Measuring Ski Movements Using Motion Capture Technology
As you see, motion capture technology is being used in various fields
Simulations using VR are also being developed
using motion capture
Now I will talk about the technical parts of motion capture
Nowadays acoustic and mechanical systems are not being used
but Magnetic and Optical systems are
So this will be mainly based on those two
The first common factor is space
For magnetic, 30m or up to 50m of space
can be secured, depending on the specifications of the equipment
There are no major space restrictions, but as for height
you should be mindful that data does not come out well
for movements in the air
What this means is that
For the magnetic system, only rotation data is produced
by the gyro sensor and the position value cannot be secured
So movements like pull ups or
flying in air on karts
they have no position values
so the data becomes floating as well
The gyro system can sensor short jumps
that land soon after
but cannot gather data from movements
that stay in air for a long time
This is to take note of
So positions would need to be on the ground
For optical, the quantity of cameras is very important
it requires capturing the markers with cameras
More cameras mean more space
But that may be restricted due to
financial issues
Therefore, securing space with limited number of cameras is
one of the important skill
Next is marker settings
Depending on how you set the marker in the optical system
cleanup time can be greatly reduced during post-production
You may need to change the settings if some markers are covered
We also need installations on fingers and other structures
we need to be flexible and mindful of the placements
Even after the marker settings were done well
The data may be cut off or have errors
during the acting
We need a detailed skill in orderto
restore these lost data
This sometimes takes more than 50%
of Optical system processing
Some movements or postures
like crossing one's arms
or hugging each other
they cover the markers a lot
It is bound to take a lot of work process
To put it simply, simple movements
makes easy of a proccess
but for movements like squatting
which covers a lot of the markers
these will result in a lot of clean up processing
When the marker data is restored
we need to convert it to skeleton data made of bones
We need to set up not only human-form on two legs
but also in 4 legged animal form
and also in the form of inverted joins
When the skeleton data is made
we also need to convert it again
into various formats for animator to use
It must be converted to be used in
Maya, Max, and Unreal
If necessary, you may need to convert directly to the desired character
And lastly, incorporating a virtual studio
This is a real time technique
It requires basic knowledge of network technology
or interworking with motion capture equipment and
Motion Builder or Unreal to show
as the final output
The primary task is considered finished by
extracting the data using the application technology
required for motion capture
based on a general understanding
of the motion capture system
Furthermore, applying it to Motion Builder or Unreal in real time
to show directly on the character while shooting
will be the key skill
This will wrap up the first lecture on Motion capture
Understanding Motion Capture What is motion capture? A computer-readable record of all moving objects Used to create scenes where the application or execution of actual movements is dangerous
Types of motion capture Acoustic system A system that calculates the time and distance it takes compared to the speed for ultrasonic waves generated from a device attached to the actor's joints to be received
Mechanic system Method of capturing by attaching iron devices to the actor's joints
Magnetic system A method of measuring the position and angle of a human joint by attaching a small magnetic sensor to the joint
Optical system A method of attaching a highly reflective marker to a person's body and capturing it with a camera that emits infrared light
Areas of use for motion capture Entertainment: Movies, games, commercials, music videos, animation Movement analysis: rehabilitation, physical therapy, gait analysis, spine analysis, sports
Industrial: Ergonomics, automotive development, robot development Others: VR, simulation