Let's explore how to use AI for animation production
Including sound, music and
Idea generation
AI Tools that Can be Used for Idea Generation
Currently, using AI tools
For idea generation
Such as the widely known ChatGPT is
A very popular method
And you can achieve very accurate results with it
When you launch ChatGPT, you'll see version 3.5
If you subscribe
You'll be able to access version 4.0
While I may not know all the technical details
It is stated that
ChatGPT 4.0 has significantly greater computational abilities
Capable of making many more judgments
Compared to version 3.5
Therefore
I believe subscribing to get more accurate results is a good idea
As you may have experienced
Using version 3.5 to research various basic things
Often yields inaccurate results
So, since we're not searching for specific information
Or anything like that right now
Using version 3.5 is fine
For example, to make sentences sound more natural
I think using 4.0
Would not be a bad idea for that purpose
Typically, when you come up with ideas
You write prompts in ChatGPT
Due to ChatGPT being widely used in Korean as well
You can obtain satisfactory results now that ChatGPT is being utilized in Korean
But it's a essential to know how to write prompts effectively
So, it would be beneficial
To observe how other experts write prompts
Also to see what I'm doing right now
then mix them well to create effective prompts
For instance, when I'm creating a story
About a knight and a dragon
And relevant information
And I hit enter
It will provide me brief plot summaries related to this
But if you also want details
Like shot breakdowns
Character descriptions or camera movements included
You can specify that
When you write prompts
In this manner
You can obtain more accurate results
For example
If you're arranging it like a framed composition
Or protagonist
Background
When you write prompts
When you create an idea
According to the 5W1H
Listing the necessary things
Will help create
A more specific story
For instance, if I were to go to this far
ChatGPT now arranges scenes
Into five segments as I desire
Then arrange them into framed compositions
And automatically generates descriptions of the characters
Sets the background
In Joseon Dynasty era
Here, when you use the expression
"in more detail"
ChatGPT provides a more detailed plot for the story
But if you read the story
generated by ChatGPT
You'll probably realize that
Using it as a script would be quite challenging
I also once conducted a class
Where we used AI to create animation videos
Most of the time, the resulting videos were quite typical
For certain shots
Students had to resort to drawing since there was no other option
I also tried using it once
It was quite challenging to get exactly what you need
So I think it's best to use AI
With the mindset of getting some ideas
Rather than relying on it completely
In a way, these AI tools can be seen as assistants to writers
Helping them with certain aspects of their work
And there's something called Bard
Which is an AI tool for generating ideas
Bard has now been replaced by Gemini
Which assets with interview preparation, household routines
Language learning plans, and place recommendations
So, typically
You might not use it for everyday idea generation
The basic presets are indeed for interview preparation and household routines
If you write prompts in a similar way here
You can expect to get similar results
This one might provide more details about each scene
And the characters involves, right?
It might give a more detailed description of the events
It might come out in a more concise, summarized format
Rather than in full sentences
It seems like a good tool
For generating ideas
Once you've generate the ideas
You can use AI tools like Midjourney or Runway
To visualize them into images
After visualizing them
You can edit the images and add music on top
If needed, you can also add voiceovers to the characters
Which can be quite essential
So, you can use AI for all these tasks
Of course, to create higher-quality videos
AI cannot accurately provide
The exact sound effects needed for a scene
Or the specific voice you envision for a character
While it can't provide exact details
It's still a useful tool for generating guidelines
Like "roughly this kind of feel"
Or if you consider this approach sufficient
For creating high-quality videos
Then that's okay too
So, for example, with the Voice Over Generative Voice AI
You choose your desired language
Then write a script
It's important to accept
That it may not be perfect
Then, you write a scrip like this and
You select the voices for the characters that match the script
Saying, "It's important to accept"
"That failure is inevitable in life"
Where you don't have to rely on voice actors for this
It's something commonly used
That many YouTubers are using these tools these days
They often use AI tools
To create virtual influencers
Who can then host live broadcasts
Thus, you can download these
And use them later for video editing
Of course, you can use them in other languages as well
If you want your content to be broadcasted in Russia
Or if you want Russians to understand it
You can choose Russian
Or you can choose German
Then you can use the same script in German
For the same content
There are tools like that
There's also a tool called Lovo
You can think of this as
A similar type of AI tool
So, if you're creating a test
You would start by creating a project
Then selecting "AI Voice and Video Start Project"
And then perhaps
"Short Voiceover" to begin
Then, the script would be generated
By default
You can choose the voice you want
It can also be accompanied by a user-friendly face
You can select the age
Whether it's a child's voice or a more mature one
As well as choose between female and male voices
The purpose for which it's used
Whether it's for audiobooks
Education
Marketing, etc., the tone will vary accordingly
So, you can choose the tone of voice
That's appropriate for your purpose
If you want to do it in Korean
You can change the language here to Korean
After changing it to Korean
You can then choose characters that match here
Then when you press "Generate"
Press "Generate" button
It will speak with a slightly awkward
Korean/English pronunciation
If I write it in Korean
"Hello, I'm a student"
"Nice to meet you," and if I generate it
Then I get the generated tones saying
"Hello, I'm a student. Nice to meet you"
When you listen to it
You'll understand the tone of this character and cans use it later for
When you need a character with this kind of tone
Write the script in that person's tone, and use it for voice-over
Where you can then incorporate it into your work
Working like this should suit your needs
If you want to translate
You can use the Generative Voice AI tool
From ElevenLabs, like we tried earlier
It just automatically translates here
If I write the script in Korean
Shall I give it a try first?
If I do it like this
"Hello, I'm a student. Nice to meet you"
Yes, like this
You need to sign up here
To enter
The AI tool automatically translates
The script you've created
So, if I go into the tool
I can skip and go to 'Voice Setting'
Where I can choose the language, right?
And then the person
But here, you can see the default tone
And other characteristics of this character
like hashtags, right?
Then you can also add your voice here
And automatically
if you've recorded your voice
It takes your voice from here
And automatically translates into other languages
Thus, if you try using this tool at home
You can get
Some really interesting results
Since I don't have the voiced file
I hope you all give it a try yourselves
Next is Heygen
Heygen is a tool that's been used lately
For things like news or for announcers
I've heard stories about
How there's a shortage of announcers in Jeju Island
So they're using AI to broadcast the news
Like such situation
Heygen is an AI tool that uses AI to act as announcers
Or to conduct question-and-answer sessions
Which is becoming increasingly popular for these purposes
Therefore, if you take a look at the demo here
You'll notice how natural it sounds
You can think of this as all being AI-generated content
Yet it sounds quite natural
Of course, the fundamentally AI-driven characters within it
or this man, based on real people, likely had photos taken
Collected data on things like closing eyes
or opening his mouth
So, this character was likely created using data collected from real people
When you see the generated video
It looks like it was shot in real-time, doesn't it?
So, once you get in
You can create an instant avatar
A photo avatar or a studio avatar
But with photo avatars, we can create them ourselves
To create instant avatars like the ones we saw in the demo
You typically need to subscribe
Which can be a drawback
So, for scenarios like
Creating realistic avatars for YouTube
or studio shoots, it could be quite useful and worth considering
Especially if you plan to use it frequently
When it comes to animation production
It's questionable whether such tools would be as necessary
There are various tools available
For generating music as well
When it comes to tools for generating music
There are tools for generating music
As well as tools where you can download
royalty-free music
But now there are many subscription-based services
That offer higher-quality music generation
When I enter the website
Called "Soundful"
You can choose various genres
and click on 'Create,' you'll need to mix here
Once you select another option
and create
You can also determine
The speed of the sound here
You can also choose whether the key is in C or D code
In this way, you can generate ambient music
or background music needed for videos
In SoundDraw, for example
You can choose genres like hip-hop
and create music
Generally, the prompts are predefined in this way
You can change the prompts here
For instance, you can change the mood to something happy
or set the theme to cinematic and the length to 10 seconds
or set the tempo to normal and choose to use all instruments
In this way, you can create music that you have composed
Of course, as I mentioned earlier
It might not be of the highest quality
But you will be able to use it well for instant purposes
Additionally, you can use royalty-free music, which means music that is free from copyright restrictions
There are numerous websites where you can download royalty-free music
Later, when you work on post-production after creating animation
Using these resources can be incredibly helpful
I've also directed about
Five short animations
I've found that after creating five short animations
No matter how well the video is made, if the music is subpar
The quality of the animation drop significantly
Therefore, I always tell my students that
Music is half of the work
So, music is just as important
Choosing the right music to complement your project is crucial
Thus
Even if you don't find the exact music you want here
You can also use these guidelines
To work with an actual music composer
To create the perfect track for your project
My student is also doing that right now
That approach has been yielding excellent results
When I produced animations
I usually recorded my vocalizations as guidance for the music composer
Providing them with an idea of what I'm aiming for
Whereas, with the advancement of AI nowadays
It seems that these processes have become much more convenient
AI Tools for Animation Production Stages
Next up is
When making a movie, for example
or when I want to overlay a 3D character on live-action background
or when I've shot live-action footage, but there are unwanted elements like
People walking in the background or unnecessary objects
Tools that remove such elements are also available
In the past, for instance
In VFX movies where everything had to be removed frame by frame
They would first connect the green screen
and then they would remove the shots with people in them
frame by frame
Nowadays, AI has advanced to the point where
You can simply erase these things with just one click of the mouse
Even in videos
Erasing things in photos isn't so difficult
But it's quite challenging when it comes to videos
But now, there are many tools available for such tasks, so
If you get a chance to see a demo
These days, even Photoshop has tools
That allow you to erase backgrounds
So you can remove unwanted elements from your images
For example, in Photoshop
You can right-click on the subject
and use the prompt like 'remove'
Then the subjects over here will be removed
Right now, because I've selected a blurry background
The selected objects are a bit unclear
If the object is clear, then
Should we try removing the face?
Using 'remove face' resulted in
A somewhat strange outcome
Anyway, if that's what came up
You can choose various options
From the Properties panel next to it
You can choose the options
Then edit as you want from here
So, by evaluating these results as
Good or Poor
After the AI learns from these evaluations
When you generate something else
It can produce results of higher quality
And when you've made the video
The most important thing in the end was
If I said I made a video
in HD resolution
If I later said that I'm going to screen this video in a movie theater
Then I would need to scale up the video's resolution
When scaling up for the theater platform
The resolution itself may degrade
Many AI tools have emerged
To upscale resolution without degrading it
Tools like Topaz Labs and BigJPG are good examples of such AI-powered upscaling solutions
and by in putting an image into these tools
They perform upscaling while preserving the quality
Making the final result look more vibrant and live
There's also Runway
Additionally, LeiaPix is a tool
That specializes in
Creating quick animations with efficiency
Well, the tools I've been introducing are
Primarily focused on
Those commonly used AI animation production
and there are many other tools available as well
There are many tools avilable
But I'm focusing on showing you the one
That are still widely used
As many have been phased out
Like this, when you run LeiaPix
After automatically calculating the depth values
It separates the background and me
Allowing for a tilt effect
As if the camera were moving like this
and it pans like this
As you can see
Hair is quite delicate
So it's challenging for computers to accurately capture the details of hair
Therefore as the hair stretches, it pans from side to side
There are some quality issues
But the zoom-in function works really well, doesn't it?
Zoom in, zoom out
You can also increase the amount of motion for better interaction
or you can use it in various ways
The AI tools I've introduced so far can be divided into
Cloud-based tools and local tools
In simple terms, cloud-based tools
Use the cloud network of the company that created the AI tools
Local tools, on the other hand, use own computer for their operations
So, both have their pros and cons
If the computer has high specifications
Using local tools is feasible without any issues
However, using local tools
Consumes the computer's graphic card memory
Which can slow down other tasks
Therefore, the best option is often
To use their cloud services
If the number of users utilizing the cloud service increases significantly
Without a properly built pipeline
Even that could become slow
Although there are pros and cons
It's becoming increasingly common for cloud-based tools to dominate as time goes on
If I were to create an animation
Using the tools I've shown you so far
I've briefly summarized some tools
That I recommended for each process
For ideas and scripts, of course
Your own thoughts are the foundation
Next, the most basic tool that would probably be used is
ChatGPT
And if you were to create an audio version, as I mentioned earlier
You would obviously use Premiere
You would use Premiere for editing
You would use ElevenLabs and Stock Library
For voiceovers and other audio elements
For music, you'd use Artlist and
Perhaps also Pixabay for additional resources
And when it comes to creating shots
or shot lists
Similarly, as you've seen in the examples
You would write prompts
Requesting specific scenes or shot compositions
That's why tools like ChatGPT would be
Heavily utilized for this purpose as well
And even in platforms like Notion
Which are widely used in companies these days
For explaining internal policies or pipelines
ChatGPT is also integrated
Therefore, using ChatGPT within Notion
Would likely yield similar results
And for image generation, Midjourney would be used
Image generation would be best done using Midjourney
For editing images, you could use Midjoruney
or Generative Fill
For image editing, Midjourney would likely
Be the most commonly used tool
And for combining individual slideshows
For assembling slideshows for your film
You could use Midjourney
For extracting and editing images
Similarly, you could use a video editing tool
To combine those images
Processed with Midjourney
And for animating images, you could use Runway
And you could continue to mix and use Pika Labs
and Stable Diffusion as needed
And for upscaling, as mentioned earlier
You could use tools like Topaz
To upscale while maintaining resolution
For effects and compositioning
Popular choices include After Effects
Runway, and Pika Labs
For editing, you could continue using the software you're already familiar with
Focusing on Premiere, or for those using Mac
They could use other editing tools specific to Mac
Later, for releasing your work
You could render
and publish it
On commonly used platforms
Like YouTube or Vimeo
With this, we've covered ideas and sound
For animation production using AI
Thank you
1. AI tools for Animation Ideas, Sound, and Music Utilization AI tools generally produce results at a standard level, so they should be used as guidelines to find direction
Since music plays a significant role in animation, it's worth exploring various uses of AI tools in this aspect Types of AI Tools Ideas and Scriptwriting: ChatGPT, Bard (Gemini)
Voice Over / Dubbing: Generative Voice AI, Lovo Translation: Heygen
Music Creation: Soundful, SoundDraw Video Background Removal: Photoshop Upscaling Images: Topaz Labs, BigJPG
Quick Animation: Leiapix
AI Tools for Different Stages of Animation Production 1. Idea: ChatGPT 2. Script: ChatGPT 3. Create the Audio Version: ElevenLabs for Voice Overs, Stock Library for Sound Effects
4. Music from Stock Library: Artlist Pixabay 5. Create a Shot List: ChatGPT, Notion Table
6. Pull Images: Midjourney
7. Edit the Images: Midjourney, Generative Fill 8. Put Together a 'Slide Show' Version of Your Film: Midjourney
9. Animate the Images: Runway, Pika Labs, Stable Video Diffusion
10. Upscaling & Polish Your Footage: Topaz 11. VFX and Composition: After Effects, Runway, Pika Labs
12. Find Tune the Edit: Using Familiar Tools, Premiere, Mac, etc. 13. Distribute: YouTube, Vimeo