I will talk about animation AI application cases
The fact that AI is eternally evolving the way
how stories are delivered is a well know fact
AI prompt literally changes how our brain processes information
and how it views the world
In this lecture, I will tell you how AI is changing creative storytelling
and how it is in real life right now
Storytelling using AI
The fact that AI will change how story is delivered enternally is not a secret
How exactly is AI changing film production?
AI helps all people become
curators of artworks
Before, everything had to be made from the beginning, all new
So not only the ability to create a work
but also the taste has so important
And now, everyone’s creativity has increased at least more than 50 times
Now is an era where even the people couldn’t produce movies
now can
And those who were film producers before can now use other methods
to create stories that were impossible before
Without relying on Hollywood greenlight
they can actually produce a project
I think that thanks to AI tools, more creative stories can be born
We can gather so much more information than any other time before
and I think that this isn’t something to fear for
but something that will help everyone and benefit us
So the most important question, will AI be able to replace your workplace?
I would say ‘no’
It is not AI replacing your work
but rather more of the possibility of your work being asked to be done in this way increasing
These days, most jobs require computers
Not long from now, AI will be everywhere in our everyday work flow
and it won’t be something as new or scary as it is now
Then a bigger question pops up
Will AI destroy Hollywood?
I’d say ‘no’ as well
But I do think that the entering barrier is lower than any other time
Things that couldn’t be said before is now able to be said by anyone
For example, when thinking about the phase of industrial revolution
I’ll take the example of sweaters
Let’s say we make 10 sweaters a day
Someone could make one sweater a day by knitting it
but after the industrial revolution, now, we can produce 5,000 sweaters a day
with the technology that we have
I think the creativity using AI is the same
A thought by one person is through AI
able to make hundreds of thousands of diversity
AI basically makes everyone a super hero
and I think there is no reason to fear this
Let’s take a look at some cases where it shows how AI can
actually benefit people’s creativity
Through AI, an artist can visualized
what their final work will look like
This technology can analyze and help the artist
even if they are not stopped by the wall of creativity
They can get feedback
without waiting for others to read the script
It can scan the creativity of the text and its originality
and can be used to check how true it is to the original work
It can also spot continuity error that’s hard to spot
It can also perform AI research scanning and summarizing millions of documents
doing the jobs of the following
Introverted people can also introduce a project visually
They do not have to become a extroverted, cool pitch machine
To editors, this tool
has the power to change their work immediately
A real life movie project can be made without actual hands on work
Camera color grading can easily
produce consistent colors
We can remove and replace objects from the background
According to the prompt, desired track can be
produced with one’s exact preferences
Also, by cloning actors’ voice and inputting desired lines
audio can be produced
With just one click, the background can be switched
Video resolution can be improved, increasing the quality
Frame speed can be changed post production
AI can be used in various tools from storyboard making
to complex processes
Next I will tell you the fields I expect to use AI a lot
I think that AI will grow in fields where
computer or mechanic analysis is possible
and where data gathering is convenient
Idea generation and script writing
Scene blocking, continuity error spotting, casting proposals
Realistic storyboard production, converting images into moving environments
Setting drone filming simulations, detail shots and more
The fields that were
done mechanically by programmers or
storyboard artists before
will be made possible to do
with AI, automatically
Among them, the most important ones in videography
will be removing backgrounds
or producing animation of characters using data
and drawing storyboards
I think these field of animation or video production
will use AI a lot
But as I told you before
this will help human
by minimizing effort of minimal people
without taking or intruding human jobs
Just like the industrial revolution, before AI
a human could make one animation work
in 1 or 2 years
while now, it is seen that tens and hundreds
of animations can be made
So managing these is very important
so I think that it is very important
for human to manage the AI and continue to play the role of directors
So although simple labor has decreased
I’m expecting roles of making creatuce decisions
will rather increase greatly
What AI has hard time doing include
improving creative scripts, showing empathy,
creating a video from the beginning, maintaining consistency, creating lively movements,
moving human facial expressions, finger rendering
These would be the examples
These are things that humans need to
decide in detail
so even though AI can make it roughly
deciding and choosing this
must be done by humans in my opinion
And considering the technology of the current time
ideally, looking into the fields of visual AI, video, movie projects
visual film pitch, concept trailer,
scene test and previsualization and storyboard, music video, experiment projects, ads,
concert visuals, science fictions, bio organic stories, shot and detail settings,
parody and comedy etc. are there
These, considering the realistic technologies
are the AI content fields that are being used
Realistically, even if they might not be able to complete a project,
it is being used for pitching or trailer or for research
for now
AI tools that can be used in animation production
Next, I will tell you about how these AI tools can be
used in animation production
focusing on pre-production
These AI tools are, as you may know,
aren't something that you can download or purchase the software
but instead you must subscribe in a website
or used is in remote
in the website server
That's how you'll use them
So when explaining AI
I think it will be good for you to see the software websites
that are out now and see what functions there are
and consider which is good and right for you
and use it
First, in Runway Gen2 you can see these softwares
and or subscribe to certain websites
You would not subscribe to all websites
but you could try the free trial first and then
see what would be useful to you in production
There is a reason to not rely on these too much
Not long ago, the release was announced
Sora, an image generation AI tool by OpenAI was introduced
When you look at the results of that
you'll be able to see much better quality results
compared to other AI products from Runway, Midjourney, or Dolly
Though it is not released yet, AI companies are
continuing to make these AI imaging tools
so in addition to tools you are using now
there will be more coming out
So when you're using them
consider that there are also these things
and once there comes out the perfect tool later
we will use that I expect
We will used them like Unreal Engine or Maya of now
but since we're in a transition period right now
you could approach it in a trying out many things sense
So when you go to Runway and sign up
you'll see this screen and if you click Start Generating
you'll be able to make videos with pictures
This Runway is a video creating tool
So if you have pictures or drawings of something
those are just still cuts that don't move
But using Runway we can adjust the camera here
or add simple animation movements
That's how it works
So if I add a photo by dragging it in
and then write how I want this image to move
in the option window, the prompt window down here
you'll be able to make a video
For example I say Cinematic
I put 'Cinematic shot of this picture with panning left' as the prompt
and click Generate
It takes a bit of time generating
so if you subscribe
it will take less time to make the video
So for now
if you use this photo to make this video
well it won't be able to move the photo but
in this platform, it will calculate the depth of the photo
and after calculating the depht
the camera, when I say camera
I mean that moving the photo to left and right
Moving it left and right,
due to the depth the things in the back moves slower
and the things in the front moves faster
Adjusting the speed
so that when we see the result
it seems like it is moving very 3 dimensionally
There is another video added
but you can see me moving in the video made with my photo
I am moving a little
but there aren't too big movements
with usually the water sparkling
or the hair flying
That's what happens most of the time
To create more diverse movements
it will need many kinds of data
about the movements that I have
Later there will be these things
but for now, this is the video that can be made with just this one photo
So when making it, in this option window
you can choose how you'll have the panning
If I say I'll have a lot of panning and regenerate it
Panning means moving the camera from left to right and up and down
We call this camera movement panning
So if I say that I'll have a lot of panning it will move a lot from left to right
So the terms appearing here,
if you get to know about the terms
used in movie cameras
it will be very convenient to use this
I once saw in the news
Jensen Huang of NVIDIA saying
'What you have to do now is
nothing about learning technology
You need to know a lot about humanities'
I heard that
So now, technological things
will all be done by AI or computers
and what humans must do is the prompt
According to how well the prompt is written
the video's quality will be high or low
so in a sense we'll need a lot of
humanities knowledge
Taking a look at this video
as I added more panning
you can see that there are much more movements here
Like this, making these shots one by one
and you extract shots from here
And editing what you extracted
you could make an ad video
or short animations for pitching
or things like that
Next is Pika Labs
When you go to Pika Labs website
you can see these example results
and these are also made using
one picture or a drawing
to make an animation
So putting these images
and you could also add lip syncing
So listening to the voices of the character you want
and write the line you want in the lip sync
and then if you do Generate Voice
it will automatically
make it in that voice, an audio
without a voice actor
So do 'attach and continue'
and if you click that you'll generate
a video is being generated now
All the AI tools released until now are in this format
so it is very easy to access
So you can decide the final resolution of this video
and the video length as well
So now we have a video
It became a very funny video
but since I used a photo where I have my mouth covered
the lip sync is a bit weird
So if you listen to the sound
the video is something like this
Simple lip sync
But as you can see it is a bit awkward
Even if it was a picture
where I did not have my mouth covered
the movements of the muscles when the mouth moves
won't be applied right
so it looks like only the mouth is opening and closing when lip syncing
I don't know about the case of 2D animations,
but it can be very awkward since this is real life photo
So if you want to make lip sync animation
with Pika or other tools
using 2D images will give you
relatively better results
We took a look at how to make videos from photos using
Runway and Pika, with prompts
These are these kinds of tools
Runway is used a lot
Runway has 'Prompt to video'
'Video to stylized video'
and 'Image to video'
and others as well
If you do 'Video to stylized video'
on Runway
you can upload a video here
For example if I put the video I prepared
Let's put Demo Assets here
This is called the original input
With the original input
you can adjust the settings
like style
weight, and seed
After adjusting them all and generate here
you can get results according to these settings
And these another very famous AI tool in Image to Video
It is Stable Diffusion
I think you would have heard of it a lot
In Instagram or Youtube
they are uploading a lot of content
This one as well,
if you put an image here
if generates a video
So in Advanced Options
if you just do Generate
But for example when you see
when generating, Stable Diffusion might
take a very long ime
Because this one, it doesn't use my computer's GPU
and uses Stable Diffusion's GPU instead
so if there are a lot of works loaded there
mine might be pushed back and take a long time to generate
but the generation itself is very fast
That is because it is using their computer
If my computer was a very good computer
it will be generated very quickly
But if my computer has a
very poor graphic card
then this Stable Diffusion might be very helpful
because it uses another
very good computer's network
So it does have the disadvantage of taking a long time due to the queue
For example, if you take a look at the examples here
like this, animation of eyes blinking
Let's see other ones as well
The left is the picture
and the right is the generated video
So in the back, the fire
or with the panning the girl in the front is also moving a little
It automatically adjusts the changed looks due to the
change in angle of the face
so there are some awkward parts when the girl turns her face
but the results by now
don't seem to be too bad
So some people say
'these videos made with AI now
are equivalent to the black and white movies
before color movies'
I believe this was said in the video
making Sora
Considering that speed
Considering how we came from black and white movies to VFX movies
like Avatar and such
movies made with AI will be possible very soon
Quicker than our expectations
It is thought to be possible
And then Dall-E
As you all know
this is a image generation tool from OpenAI
But if you go to Dall-E
it is out of service and it's waiting for the next version
I assume that that will be Sora
But it is currently out of service
So Runway, Pika, and Stable Diffusion
did write the prompts
but still generated videos
based on the picture, drawing, or video
This Dall-E
generates video only with prompts
or generates images
Stable Diffusion also has that function
If you visit the website
you can choose from many options
and generate one
Usually Stable Diffusion
or Midjourney
use Discord a lot
If you join Discord
and make your AI window
put a slash and click 'imagine'
then write a prompt
Then this 'imagine' means using the Midjourney bot
so writing 'imagine' and then the prompt
this gets automatically connected to Midjourney
to create an image in Discord
For example
chest shot of asian
chest shot of asian guy in seoul city
For example if I put the prompt like this
it will automatically connect with Midjourney
and generate one
Of course there are many AI tools made in Korea
but for now, tools from the US
have much better performance that the ones from Korea
and since they are from the US
the prompt also must be written in English
to get a good result
So if you write it in Korean
the result might
not be as good as you expected
And also how you write the prompt
really changes the result
So constantly checking while writing
and checking what the write prompt for the tool is
and turning that into a list
so that you can use that when
making other shots
I think that should be how it should be practiced
So normally when using Midjourney
It generates four images
They all have different vibes
So you can choose one among these
and take it again to Runway or Stable Diffusion
to create another shot
and then edit them to make a video
These processes are needed
Up until now we learned about image generation methods
centered around production in animation production
apart from these, as you can see in our materials
there are a lot of tools
and these will be the tools
used in the production when making animations
Then using these, when you produce a video
you'll be able to use many of them
as you can see in the examples
Prompt to Video Runway Gen 2 Pika Labs Image to Video Runway Gen 2 Pika Labs Stable Video Diffusion
Video to Stylized Video Runway Gen 1
Text to Image Tools Stable Diffusion Dall-E Friefly Midjourney Voice Over / Dubbing Elevenlabs Lovo
Ideas and Scriptwriting Bard ChatGPT Translation Heygen Elevenlabs
To used more diversely
would be a bit difficult for now
But still, I hope you can try producing
using the tools I introduced
Thank you
Animation storytelling using AI Characteristics of AI storytelling New stories Creative stories Diverse stories
2. AI tools that can be used in animation production Characteristics of AI tools It is recommended to try free trials and then subscribing to appropriate tools on websites since purchasing or downloading is not available
Prompt must be written in English to get good results Types of AI tools Prompt to Video: Runway Gen 2, Pika Labs
Video to Stylized Video: Runway Gen 1 Image to Video:Runway Gen 2, Pika Labs, Stable Video Diffusion
Text to Image Tools: Stable Diffusion, Dall-E, Friefly, Midjourney
Ideas and Scriptwriting: Bard, ChatGPT Voice Over / Dubbing : Elevenlabs, Lovo
Translation: Heygen, Elevenlabs