Game Basics: Introduction to Game QA and Testing Strategies
Hello
I am Sukjoon Jin, working in QA at the Devsisters Technology Headquarters
Today, I would like to give you a brief introduction to the definition
and tasks of game QA
and also discuss the latest trends in the field
The lecture content is structured into four main parts
First, I would like to focus on the term QA
and provide a definition of what QA is
and what it involves
Next, I will explain why QA is necessary
in game development
I will then detail
the various types of tasks
we perform in game QA
Finally, I will discuss
how game QA is evolving with recent trends
particularly focusing on
how data is shaping these changes
Definition and Necessity of Game QA
As mentioned earlier
it's important to first interpret
and define what the term "QA" means
Let's start by examining what QA stands for
QA stands for
Quality Assurance
Most people commonly shorten it to QA
It's a combination of two words
Let’s examine the definition of
each word in "Quality Assurance"
First of all
'Quality' is generally understood as 'quality' in a social context
So, "quality" refers to the essential nature or foundation of products
produced in factories or services
provided in various industries
Renowned software engineers like Juran have defined quality
as the degree to
which a software is suitable for use
and meets its
intended purpose
Recently, engineers like Crosby
have defined quality as how well a product or service
conforms to its specified requirements
So, you can understand that
these are the basic definitions
of quality
To summarize in keywords
Quality should be measurable with numbers
and should focus on
the inherent value of the product
or service
We can define quality as
the measurable value
of a product
Even in the context of games
we can define
quality
similarly
The quality of a game is indeed a complex and often subjective topic
with no definitive answer
I’ve summarized
what I think defines the quality of a game
Game designers, programmers, and artists
begin by creating the game
They plan and design the game
then create it
resulting in user experiences that they deliver
These are metrics
that can quantitatively and qualitatively show
whether the user experience is effectively delivered
to the end user
Indeed, the concept can be complex
and varies for each individual
It's best to use these definitions
as a reference
to aid in your own considerations of quality
There’s no definitive answer
so it’s important for each person
to reflect on and define quality for themselves
So, as I just mentioned
I define the quality of a game
as measuring
how effectively it can deliver the user experience
When we say that a game has good quality
it doesn’t necessarily mean that
the game is immediately 100% enjoyable
Game enjoyment is
an important indicator of a game's quality
but saying that a game is fun
doesn’t automatically mean that its quality is high
An enjoyable game is one
where the developer's intentions are effectively communicated and realized
In that sense, from the perspective of quality as discussed
a game can be considered high-quality if it effectively conveys the developer's intentions
It’s an interesting question to consider
whether a game can be of high quality
even if it’s
not enjoyable
If the developer’s intentions are effectively communicated
and users can interact smoothly with the game
it can be considered high-quality
However, users might ultimately find it unenjoyable despite its high quality
So, I want to emphasize that
enjoyment and quality cannot be directly compared
from the same perspective
Next, I will discuss
the concept of "Assurance"
In a dictionary sense
"Assurance" often means insurance or guarantee
It involves proving that there is responsibility
and no mistakes regarding a certain object or matter
In a primary dictionary sense
QA can be seen as ensuring and guaranteeing
that the quality of a product or service
will meet a certain standard
So, when translating into Korean
it is often translated as "Quality Assurance"
QA is defined as the role
that ensures and guarantees
a product meets a certain level of quality
If we decide that
our product should be released with a quality score of around 90
QA guarantees that
all released products will meet or exceed
that 90-point quality standard
However, unlike products manufactured in traditional factories
software has many variables
making it practically challenging
to guarantee
a fixed quality level
In that case, it seems
we need to redefine the concept of Assurance
Traditionally, Assurance referred to
guarantees, insurance, and providing confidence
However, the concept of Assurance
has been evolving recently
It is increasingly evolving into a concept more akin to "Assistance"
The quality of a product is a shared responsibility
among everyone involved in creating it
Those working in QA cannot be solely responsible for the quality of the product
The quality of a product may already be determined by
how it was created by the development team
and QA professionals typically cannot change that quality
during their final testing phase
To create a high-quality product
each role—such as planners and developers
must perform their tasks effectively
QA helps by testing to ensure that
these tasks are carried out correctly
QA is not synonymous with testing
Testing is a crucial
and significant part of QA’s responsibilities
but it is not synonymous
with QA itself
Recently, one of the increasingly prominent roles of QA professionals
has been highlighted
Recently, QA professionals have increasingly taken on the role of fostering a quality
focused culture within studios or organizations
This includes
conducting seminars
and training to achieve this goal
To explore this in more detail
how do QA professionals collaborate and assist in creating and ensuring quality?
As mentioned earlier
one of the primary actions QA professionals take is testing
Testing involves
verifying whether products and services are developed
according to the specifications and requirements
Through testing, we also measure
whether the product is well-made, either quantitatively or qualitatively
So, if there is a target level
QA assists with all necessary actions
to achieve that goal
We consider these tasks
as part of the broader responsibilities of QA
To perform these tasks
we utilize various tools and processes for assistance
At a basic level
we use Bug Tracking Systems (BTS)
such as JIRA
which are representative of these tools
Recently, we use
a variety of additional tools
to help developers and planners
work more smoothly
I will mention this in more detail later
but we also use data to assess quality
and identify areas that need improvement
Recently, data has become a key trend
and QA professionals are increasingly using tools
that leverage this data
As mentioned earlier
establishing a culture
that facilitates these processes is also a significant part of
what QA professionals are
currently involved in
In summary
QA is focused on
accurately assessing the value of the products
and services
it evaluates
I'm about to share
a favorite quote from a poem
I think this quote symbolically represents
the role of QA
It’s a line from the poem “Flower” by Kim Chunsu
The line is: Before I called his name
he was just a gesture
When I called his name
he came to me and became a flower
In this context, calling out someone's name
and making them a flower
symbolizes discovering and recognizing
the inherent value within them
I believe that the true value of QA’s work is
in uncovering and revealing
the inherent value of the subject
we are testing
Now, let's explore
why these QA tasks are essential
and why game QA is necessary in game development
In December 2020, the Polish studio CDPR
released a highly famous game
CDPR, known for its highly acclaimed The Witcher series
released Cyberpunk 2077, in December 2020
This game, with its ambitious vision
aimed to make a significant impact in the sci-fi genre
However, as many who read the news
and remember will know
the game faced numerous bugs
and optimization issues after its release
As a result, just 8 days after its release
Sony's store issued a refund notice for the game
The notice stated that
full refunds would be given to everyone who purchased the game
Additionally, the title was removed from the PlayStation Store
The following day
the Microsoft Store also issued a refund notice
This reflected significant quality issues with the game
Within two weeks of the release, CDPR's stock price dropped by 40%
This is one of the most prominent examples of
how poor quality can lead to
significant financial and economic damage
When considering
how to prevent such issues
one might think that extensive testing before release
could cover these problems
However
gamers' expectations have significantly risen recently
Simply having a game function when buttons are pressed
isn’t enough for gamers to be satisfied
To meet
the high expectations of gamers
traditional testing methods
such as basic functional testing
and a single pre-release test
are no longer sufficient
Recently
the concept of Shift-left Testing has emerged
It means that
QA and testing stages need to move earlier in the development process
The concept of integrating testing from the very beginning of development
and planning has been emphasized more
especially due to the issues faced with CDPR's Cyberpunk 2077
The focus has also shifted to the development process
and the overall culture
CDPR had an internal QA team
and also utilized a significant amount of external outsourcing
In light of this, it’s essential for QA
to maintain internal quality
and prevent duplicated issues
to meet the minimum standards
To ensure these tasks are effectively managed
it is crucial to implement proper systems and processes
including configuration management
Building systems that align with recent trends
like Continuous Integration (CI) and Continuous Deployment (CD) is essential
Understanding these practices has become increasingly important
There are definitely engineers
who specialize in these areas
While QA professionals
may not reach the level of infrastructure or process engineers
having enough knowledge
to communicate effectively
with these specialists is important
for smooth QA operations
The issues with CDPR's Cyberpunk 2077 highlighted the importance of these aspects
in QA and development processes
For example, although we discussed Cyberpunk 2077
the need for QA
and the areas where QA needs to be effectively implemented
are expanding increasingly
Let's take a closer look at the types of game QA tasks
the latest trends
and the expanding areas in the field
We will examine in more detail
what game QA testers are doing, which aspects are being defined
and how these aspects are being solidified
First, let's take a look at the basic tasks
that game QA (Quality Assurance) testers perform
Game QA primarily focuses on ensuring the smooth launch of a game
by addressing necessary requirements
Additionally, after the launch
they work on
maintaining the quality levels
needed for smooth operation of the service
Broadly speaking
the field can be divided into Development QA and Publishing QA
Development QA
as the name suggests
contributes to the process of developing a game to the point where it can be launched
Publishing QA, on the other hand
involves the concept of launching and operating a game
Therefore, the organization performing Publishing QA carries out tasks
that align with these processes
Broadly, Development QA primarily focuses on
working closely with client and server developers
as well as platform-related developers
to ensure that
the features they create function correctly
This involves
performing extensive functional testing
Publishing QA professionals have a somewhat different perspective
Publishing QA professionals are responsible
for verifying whether the games that have been developed are ready
to be serviced to customers
Publishing QA focuses more on non-functional aspects
such as performance testing and compatibility testing
They assess whether the game
developed by client and server developers
performs smoothly and runs well
across the various devices users might have
Both areas, Development QA and Publishing QA
share some common aspects
Indeed, basic functionality and non-functionality are crucial
For instance, if a game doesn't start when launched
or if buttons don't respond when pressed
it renders the game essentially useless, regardless of any other features
Therefore
it's essential to verify these basic functionalities before performing
either functional or non-functional QA
These aspects are considered common areas across both functional and non-functional QA
These aspects are referred to as BVT (Build Verification Testing)
or BAT (Build Acceptance Testing)
We will explore these terms in more detail later
As mentioned earlier, those involved in functional QA
including game QA professionals, often focus heavily on performing functional testing
As mentioned earlier, functional QA
primarily involves verifying that the core client/server functionalities of the game
essentially the foundational elements
are implemented
according to the initial requirements
In some cases, testing is done in a build lab
Recently, however
editors like Unity and Unreal Engine are so well-developed
that testing is often performed directly within these editors
This distinguishes QA testing from developer testing
Developer testing and game QA differ in several ways
Developer testing is typically a light check
performed by the developers themselves
to quickly verify that
the specific areas they’ve just coded
are functioning correctly
and development QA goes deeper than that
it covers a broader and deeper range of testing
It checks whether the requirements are properly implemented
and verifies that desired functionalities work correctly
at different build levels and unit stages
However, Development QA must maintain close communication
with developers at all times
Development QA and functional QA are performed after developer testing is completed
so working closely with developers is highly beneficial
This proximity helps in promptly addressing issues
and ensuring thorough testing
From an ISTQB perspective
the most common types of testing performed are
unit testing
and integration testing
Unit testing involves
checking whether a single unit or component functions correctly after it has been developed
Following that, integration testing involves combining multiple units
In integration testing
you verify that the combined units perform
their intended functions correctly and meet the required objectives
Since Development QA
works closely with developers
the testing cycles and bug resolution periods
are relatively short
Testing can be done on an hourly basis
when developers request
immediate testing after coding changes
QA testers promptly conduct tests
and address any bugs that are found quickly
On the other hand, working closely with developers
often means using the same tools
and development environments they use
As mentioned earlier, QA often uses game engines
and other development tools directly
such as editors used by developers, for testing purposes
This is one of the advantages
for game developers
who also perform QA
Next, let’s look
at Publishing QA, as mentioned earlier
Publishing QA primarily focuses on non-functional QA tasks
Development QA focuses on the "What"
what the game software should do
In contrast
Publishing QA focuses
more on the "How"
Publishing QA focuses on how quickly
and smoothly the game performs
Publishing QA typically
tests the game at the build level
This is because users encounter the game in a specific build version
so Publishing QA tests the game in that environment
to ensure it meets quality standards for the end user
According to ISTQB
this level of testing corresponds to
system-level testing
After unit and integration tests are completed
these components come together
to form a larger system
the system in this context
can be considered as a single game build
Publishing QA involves
validating that the features and experiences
we aim to provide through the game build
are functioning smoothly and effectively
Thus, Publishing QA involves performing basic functional testing
Then, Publishing QA includes non-functional testing
Recently, major game platforms
like Google and Apple have become prominent
Publishing QA also involves
performing platform-specific validation tests
to ensure the game meets the requirements set by platforms
These tests typically have longer testing cycles
and bug resolution periods
compared to the shorter cycles seen in Development QA
While not always the case
the testing cycles and bug resolution periods in Publishing QA are generally longer, requiring a more extended approach
And as mentioned earlier, those involved in functional QA
Development QA professionals
use various tools
Recently, Publishing QA professionals are also increasingly using tools to support their testing processes
Publishing QA professionals often use data-related tools
which, while serving a slightly different purpose
are varied and essential
for their testing processes
Let’s take a closer look at
performance testing and compatibility testing
which are key aspects of non-functional testing
performed by Publishing QA
Performance testing can be broadly divided into two categories
client performance and server performance
We’re focusing on
general client performance testing at the moment
In client performance testing
you check how much resource consumption occurs on the device
or terminal while the game is running
In mobile games
client performance testing primarily involves monitoring CPU usage and chipset performance
You also check how much memory is used
and monitor any associated issues
like overheating due to resource consumption
Some testers
manually check these aspects
but increasingly, performance tools are being used to monitor
and assess these metrics more effectively
Through performance testing
you check for issues
such as excessive resource usage
that might cause lag on the game device
or unexpected crashes
Ultimately, the key is to ensure that
the game runs smoothly and comfortably
for users
on the tested devices
The core of client performance testing is to
verify that the game delivers smooth performance
on the client device
Compatibility testing does overlap with
performance testing to some extent
Ultimately, compatibility testing focuses on
ensuring that users can enjoy the game seamlessly
In compatibility testing
the focus is specifically on device compatibility
With the vast number of devices available
compatibility testing ensures that
your game performs well across various devices
when it is released
There are many different brands and types of devices
primarily categorized into Apple devices and Android devices
In compatibility testing
you prepare a diverse range of devices
including various brands, chipsets, and display sizes
You test the game
to ensure that it doesn't suffer from issues
like screen clipping
or crashes
on specific chipsets
When launching a game globally
different countries often use
varying types of devices
Compatibility testing ensures that
the game runs smoothly on commonly used devices in different regions
such as Southeast Asia or Europe
and on devices
with high market share
This involves
focusing on devices
that are popular in each specific mark
Among the publishing QAs that I mentioned earlier
the third one I talked about
is game platform testing
It is a test to ensure that
we can meet the standards
required by platform providers like Apple or Google
So, there are various platforms and stores
where the game can run, most notably the Apple Store, Google Play Store
as well as other game platforms
like Steam and PlayStation
These platform providers have basic requirements
for functionality and specifications
So, in order to release our game
we need to meet these standards
We verify whether our game
meets these specifications and functions smoothly
One of the crucial aspects of this test is that
if we fail to pass it
we won't be able to release the game on that store
Therefore, since we can't launch the game otherwise
we must always ensure that
these requirements are properly applied
Recently, these standards have become relatively more flexible
compared to before
but they still remain one of the important aspects
Each platform store considers these aspects
very important
so they provide their own quality guidelines
So, there are guidelines provided by each store
that specify what needs to be met in order to release the game
Based on these guidelines
you create
and execute test cases
Now, I've talked about the tests
performed by both development QA and publishing QA
Earlier, I mentioned that
both development QA and publishing QA perform tests on the most basic functions
as a common area
At that time, I used the terms BVT and BAT
In fact, these terms are often used interchangeably
and their dictionary definitions aren't always applied 100% accurately
Different companies and organizations might use these terms slightly differently
but generally
we've adopted specific meanings for them
in our organization
BVT stands for Build Verification Test
It's essentially about verifying
whether the build itself has the basic minimum functionalities
and whether it serves its purpose
as a build
So, before the development QA team
distributes the initial build
they perform basic functionality tests
such as checking operations, attacks
and stage transitions
to ensure that the build is ready for the next round of testing
The 'next tests' usually refer to
the subsequent level of testing
conducted by other teams
once the build is distributed
Next is BAT
which stands for Build Acceptance Test
So, when a build is received
the BAT is conducted by the receiving team
to confirm that the build is indeed testable
and can be used for further testing
Typically, this is performed by the publishing QA team
After the development QA team completes the BVT
and hands over the build
the publishing QA team typically performs a BAT
with almost the same content
So, as mentioned
this confirms that the basic functionalities are indeed working well
Based on that
we then proceed with performance and compatibility testing
using it as the starting point
Up to this point
I've provided a brief overview of the definition of game QA
and the tasks involved in game QA
Please keep in mind that
not everything I've mentioned applies 100% universally
as practices can vary slightly
from organization to organization
The most characteristic keyword would be 'data'
Recently, AI is indeed one of the hottest trends
but data can be seen as a larger category that encompasses AI and other trends
The game industry is evolving rapidly
Both gamers
and game developers are
likely feeling these changes very acutely
In 1972, there was a game called Pong
created by Atari, which caused a global sensation
It was a game similar to table tennis
where players hit a white ball back and forth, trying to make the opponent miss
It’s a game from about 50 years ago
If we consider that as the starting point for games
then take something
like League of Legends
which has already been out for over 10 years
It was released in 2009 and was quite an innovative game
League of Legends remains
a dominant force in PC gaming
Meanwhile, there are also rapidly evolving mobile games
like Cookie Run Kingdom
and new games
boasting impressive and vibrant resources are constantly emerging
The pace of change is accelerating
Game QA methods are also evolving
to keep pace with this speed of change
While games are changing rapidly
user environments are also becoming increasingly complex
The mobile phones and devices you use
are likely to become even more diverse
The replacement cycles are also getting shorter
and ensuring that mobile games run well
on all these devices
is one of the responsibilities of QA
Nowadays, people are spending significantly more money on games compared to the past
with more people making in-game purchases
these elements have become quite critical
With many people now spending real money on games
beyond just treating them as a hobby
in-game purchases
and payments have become
very important
for both players and developers
As a result
the work of QA becomes increasingly important
For example, when it comes to spending money
there’s a need for more thorough QA to ensure everything is functioning properly
In the past, the mission of QA was to perform
as many tests as possible
to simply reduce the number of defects
Back in the day, the trend was to conduct full testing
for two to three months before release
aiming to reduce defects to below ten
before launching the game
Those practices are now gradually changing
With faster launch cycles
and frequent updates
sometimes two or three times a month
conducting tests only two or three times
becomes less meaningful
Therefore, testing needs to be executed quickly
and during operations
live issues often arise that need to be addressed continuously
This means that issues need to be fixed and deployed quickly
leading to significant changes
in the role of game QA
Ultimately, QA teams are involved
in every aspect of quality
related to game development and service
In the past
the role of game QA was crucial before release
but now QA is important
throughout the entire lifecycle of the game
from launch to the end of its service
ensuring high-quality service
at all stages
Let’s take a closer look at
how the role of QA is evolving
and the underlying reasons
for these changes
You've probably heard of game engines
like Unity and Unreal
Recently, game development across platforms
whether mobile, PC, or others
is often based on game engines like Unity or Unreal
As a result, the basic functionalities provided by game engines
are often covered to a sufficient extent
reducing the need for deep testing by QA
Basic functions like movement and attacks
are covered by the engine
so QA no longer needs to spend
as much effort
on testing these fundamental features as they did in the past
Because these engines are now more accessible
developers previously found it
quite challenging to work with them
However, nowadays, since engines are easy to install
and use
QA teams are also frequently working with these editors
With the ability to develop and test
in the same environment
our work efficiency has greatly improved
Not only editors but also various development tools
have become widely used
We now have environments where we can perform simple coding
and to utilize data
tools like Kibana are necessary
In the past, accessing these tools
was not very easy for QA teams
Now, with increased needs
and established environments
we can easily access and use many tools
Earlier
I mentioned functional and non-functional areas
but from a different perspective
terms like 'data' and 'QAOps' have started to emerge
These areas have now expanded
to include such concepts
The term 'QAOps'
is becoming more trendy
In QA work
we often need various tools
such as editors
and separate environments
for testing
In the past
we relied heavily on engineers to handle those aspects
Engineers used to handle tasks beyond their regular duties
such as setting up environments
needed for QA testing
Recently, however
QA teams themselves have become capable of
managing these tasks directly
Tasks that QA teams can handle directly
are referred to as QAOps
That's why
our organization has set the goal of
implementing QAOps practices
We're making significant efforts to achieve this
And now the keyword "data" keeps coming up
but actually, the best example of this data is probably
game login information
In the past, I thought that leaving behind
such game logs was almost systemically impossible
Recently, as the cost of infrastructure and related resources
has decreased
and the quality of infrastructure environments has improved
it has become possible to effectively maintain
and utilize a large volume of logs
We now embed logs for every possible action
and, through log search tools
like Kibana
we can query the conditions we need
and search to see
what data has been recorded
In practice, QA teams now directly review such data
by performing log QA
They've been able to see it
Not only game logs
but also one of the tools
we heavily use in live QA
is Google Crashlytics, now part of Firebase
We use Crashlytics within Firebase
for quality monitoring
So, with this tool
we can immediately monitor
if crashes increase or if unusual symptoms are observed on specific devices
These tools also indicate
which part of the game is experiencing issues
allowing us to quickly communicate this to developers and enable rapid fixes
It has become possible to fix issues quickly
All of this has become possible because of the data
In a very short time
I briefly covered what game QA involves
what tasks it includes
and how recent trends
represented by data are evolving
As I've mentioned several times
these concepts cannot be defined with 100% uniform value
or meaning across all organizations
So, when listening to the lecture
it would be helpful for the audience to use it as a reference
for understanding what game QA is
developing their own perspectives
and organizing their tasks
They should consider these insights
as part of their broader understanding and application
Alright, we'll conclude the lecture here
Thank you
Definition and Importance of Game QA Definition of Game QA: A metric that quantitatively and qualitatively represents how well a developer's intentions are conveyed to the user
It is primarily evolving into a form that helps ensure high quality for products and services through testing
Importance of Game QA: The traditional concept of QA as merely testing before release is insufficient to meet the high expectations of today's game users
The development process and culture also significantly impact the quality of the product The areas where QA is needed and the scope of effective QA are increasingly expanding
Types of Game QA Tasks and Latest Trends Types of Game QA Tasks: Development QA verifies that client/server functions are implemented according to requirements in the development environment
Publishing QA: Conducts tests to ensure that the build meets the quality standards required for launch Performance Testing: Tests to determine how many resources are consumed when running the game on different devices
Compatibility Testing: Tests to determine the range of devices on which the game can run smoothly
BAT (Build Acceptance Testing): Conducted before deploying the build in development QA to ensure that basic functions are operational and that subsequent testing can proceed
BVT (Build Verification Testing): In publishing QA, verifies the testability of a build before starting the testing process after receiving the build
Trends in Game QA Providing Objective Data Through Analytics Developing specialized fields like QAOps that are technically advanced and distinct
The End