Game - Basics Game Test and Result Analysis
Hello
My name is Sukjoon Jeon, and I work in Devsisters’ technical department
In Previous lectures, I've talked about the definition of game QA
the types of work, and then the trends
For this lecture I want to talk about
how to write test documents effectively
First objective is understanding how to
write the test case and the structure of it
We will learn about some tips that are
actually being used in writing test cases
Then we will look at how to calculate data
when performing tests
through these test cases
and then reporting the results
Lastly, we will look at how to communicate the bugs
discovered during the test
and how to write a bug report about that
Structure and Writing Method of Test Cases
The phrase 'Test Case' is
actually frequently used in field
but is not easy to define it
I'll give you an real life example
The phrase was first used
in software engineering
Then, since the purpose of
these tests are to achieve a purpose
So, the documents needed to perform such a test
ere defined as test cases
As I said before
this is one of the most used documents in QA
We write the test case first
and then report the discovered bugs and errors
These two are the most used
Test cases are
generally talked about a lot
The biggest trait of it
is that a case should always be structural
It must be systematically organized
The writer of the case should always keep in mind that
the tests performed on theses cases
need to be done effectively and efficiently
Other than Test Case
another frequently used term is 'checklist'
The checklist relatively
serves a simpler purpose
Although there is no absolute way of
distinguishing between
which document is checklist and which is test case
They are simply used in their respective organizations
If the document is relatively simpler, it would be a checklist
if it is more complex, a test case
This is one way of differentiating them
In my workplace
The most biggest difference
between the two is
if it has specifically expected results written
or is there a space to write
the expected results
If that specified expectation is listed
we see it as a test case
If it is in a check-box style
to simply check what is working
we group them as checklists
I've said that test cases are more complex
and that they need to be
structurally organized
So let's go ahead and see
what they consist of
Generally, spreadsheets are used
Nowadays, different tools like Notion are used as well
to write a test case
but in all those tools, the first thing you see
is the case's unique number
It could be in all numbers
it could be a mix of letters and numbers
some places use tags
But the sole purpose of them all is
to identify a unique test case
Thus, the same number is never repeated
That is why every single case has a unique sequence
The reason for that is, for example
Let's say we have a problem within the case
We wouldn't say, for example
"Did you see in which environment it was tested?"
let's say the unique number of the test is 3
We can say, "did you check case number 3?"
which is a way more simple
yet effective communication method
It is also easier to communicate with developers
with this
That is the reason for the unique numbers
We can say that the traceability of test cases
can be improved with these
And after the case's unique number
you will see
the Category that indicates
which area of this product
this test case is testing
In case of games, generally
level 2 or level 3 is specified
It does not follow a set rule
It differs by organization
One could write level 2 or 3 depending on
what type of testing the organization is doing
If the case is very definite
the category may not be specified at all
Let's say we are
conducting a character test
the character will be level 1
and there will be separate classes within the characters
If we test warrior character, then level two warrior
The warrior must have various areas of action
if we test skills among them
Character, Warrior, Skill
It could be written in three levels like this
If we give a specified category like this
We can see right away
Which area of the game this case
is testing very easily and effectively
It is best if theses things
are structured the same as a proposal
If the proposal of the game is structured
in the category, character, warrior, skill
then the test case should also follow that structure
and specify the category the same way
This is the traceability I talked about earlier
It is easier for us to track or spot
what this test case is confirming
So it is very common to write the unique number
and then category at the front
Next we have the prerequisites
It is also called Precondition
and lists the conditions necessary before
performing the test case
This is not needed in all test cases
But only in specific cases that
require certain conditions
you would only write them then
For example if a certain skill
like an ultimate skill
that is not always available to use
In most games
you have to achieve a certain condition
in order to use the ultimate skill
So the Precondition clarifies that
the achievement is needed
So, if you need to achieve certain conditions
or meet the conditions to use the ultimate skill
these things must be specified in the preconditions
to actually perform the test case
This is needed only when it is present
but is not needed in
all test cases
The next most important
part that takes the most space in the test case
is the Test Step
This is a complete description of processes
performed by QA people and testers
Many people use bullets or
numbers to list
It doesn't have a set template
It just needs to be written at a level that is
comfortable to communicate within the organization
Sometimes these steps are
replaced by input values
If you need to test various input values ​​
in a box that allows different inputs
In some cases, only such input values ​​are specified
But it should be written on a level where
all of the testers of
this test case could immediately know
what they mean
So even if it only has input values
one should be able to tell
why the input value is there
in context
If the steps were numbered
and I were to follow those
it is best that no additional communication is needed
So when you follow the process in order
The second step falls naturally after the first
and after the second, the third step
should follow naturally
This type of test case
where things fall in place natrually
is the most effectively written test case
So the next thing to specify is the
expected results would be
after following these steps
This expected result is the part that I
mentioned earlier as being the biggest difference
between a checklist and a test case
An expected result generally
should be written in requirements or proposals
For example
When a specific skill is used
the expected result is as such
That should be specified there
But sometimes the tests cannot
have an expected result
Common sense tells us that when the forward button is pressed
the character moves forward and there is
no need to write down the
expected results in detail
In cases where these expected results are specified in detail
you must write the expected results so
that everyone can understand them
That is what determines the test case's worth
But writing phrases like "normal results are expected"
or "should be normal" just to fill in the
expected results part
should be avoided
If then, the results part should be better left blank
Only fill out these expectations part
when there would be
results we can look into in detail
After that there is space to write the actual results
This may differ from group by group
But a general rule of thumb is to write this result
when you wrote the expected result
Expected result is what you would expect
before writing this case
Forecasting that the
result might come out a certain way
The Actual result is where you write down
the active result after performing the test
So if the actual result is
different from the expected result
This specific case will be filed as a Fail
This is an issue of itself
And this issue or flaw
should be registered through the Bug Tracking System
Interpretation of whether the actual results match the expected results
and how they differ may be slightly different
It may also differ depending on
the tester's individual experience or
how the organization views this part
But strictly speaking
Even if the expected results and actual results are only slightly different
this should be filed as a Fail
Those things help us manage
quality a little more strictly
After all of these are done
after comparing the actual and expected results
Now you can record the test case's Final Result
Usually, we have about four main expectations
This might be different by groups
Some only write two
Some write three
But in general we use four
The first is to file it as a Pass
This is the case
where the expected and actual results align
The Pass means that
this case was performed well and the
purpose we wanted was verified
Next we have the Fail
Fail is when the result is different
We expected a certain outcome
but have a different one
Then, you would write the difference in detail
and must report it as an issue
and this case will be set as a Fail
The third is N/A which is
also known as Not Available
This is a case where the original plan
implemented 10 features in a build
but only 7 were actually perfomred
and numbers 8, 9, 10 were not impleneted
then 8, 9, 10 would be filed as N/A
It would be nice if all the
expected results were all delivered
when we see the build notes
but there are a lot of instances where this is not the case
those things should be processed as N/A in advance
so that errors do not occur later
when we calculate the Pass Rate
So, it is necessary to manage these things strictly too
Lastly there is Block
Block occurs when a specific prerequisite cannot be executed
Then all the test cases that cannot be executed
because of that should be also filed as Block
For example after you log in
You would start the test in categories
Like PvE or PvP
But if the log in is not complete
You won't be able to open those areas of the game
Even if the PvE or PvP was implented
because we couldn't log in
we wouldn't be able to perform the test
In this case, the part where we couldn't log in
would be processed as Fail
and the PvE and PvP that we couldn't check because of the login issue
would be processed as Block
We need to specify these things so that
we can understand what kind of situation each result
will be in this build after the test is finished
If it came out as expected
or if it has any issues
or if something is not developed in the build yet
or the test was not performed because of such issue
we can know things like this
So, it is very important for testers
to judge and specify these things
reasonably and objectively
These test cases don't seem that easy
You might think there are lots of conditions
and writing them may not be easy
There are a few tips in
writing these test cases
When you look at a test case for a first time
it will seem very complex
but there should be no issue in following them
So even if there is no relevant context or background knowledge
You should be able to get results when
testing by following steps 1, 2, 3, and 4
However, it is not that easy
to objectively write things
that anyone can recognize
Also, in order to perform these test cases
here are times when a lot of communication is involved
That is an example of a test case that is not right
You have to keep asking
the author of this test case
"What is this?"
"What does this word signify?"
Asking lots of questions like these
is definitely not ideal
If the test case is written
really objective and simply
performing should not be a problem
but to have to keep asking
the communication is inefficient
and cuts down our resources
You can say then
the test case was done incorrectly
To continue
Constructing a case well
means it cannot be wordy or complex
It needs to be simple
That is way we use numbers or bullets
Basically, it needs to have a good readability
I think things like letter spacing and
blank spaces are very important as well
these all take part in readability
Because there are not just a few test cases to read
In some instances there may be hundreds and thousands
In order for us to go through that without loosing our concentration
the document's readability is very important
So it is a tester's necessary skill to
write them efficiently
A concise and highly readable document
like this is needed to reduce the cost of communication
to understand the unclear words
as mentioned earlier
Therefore when choosing even one word
it is best to use a word that
all relevant departments within a department
can easily understand and agree on
You cannot think for yourself
So, it is very important to review the test case
so that you can write it objectively
As I said many times earlier
anyone should be able to understand it
The next thing I want to bring up
is something you might overlook easily
Positive results means Pass
and negative results should mean Fail
For example let's say we have a test case
that asks
Does the text contain typos?
If it is a Pass, then you might think there are no typos
When you ask 'if there are' and you answer 'yes'
you come to interpret it as 'it is included'
So since the test was a Pass
You would think there is a typo
In this case we need to change the
test case's actual steps
When asked 'Are there no typos in the text?'
and it says that is true (pass)
then you think Okay, that's positive, the expected results are good
You will be able to understand that the actual results came out that way
So generally for Pass
we use a lot of positive signals such as the color green
You need to handle it carefully so that
others don't interpret the those unnecessarily
You must not try to
verify multiple requirements in one case
There are many instances like this when you don't have time
or want to reduce the number of cases
If possible, one test case should be
able to verify only one requirement
in some cases, there are quite a few people who combine
two or three conditions into one case to verify them
Those cases, would not be a problem
if the case was a Pass
But when the case has a Fail
You have to go and analyze each one at that time to see
which of those three conditions was the cause
That would take up even more time
So, it is necessary to write it separately
so that one case can only verify
one requirement
even if it takes more time in the beginning
If there are x number of tests
then there obviously must be x number of test cases
Also it is good to additionally verify
what happens when you combine x conditions
Although it is a matter of principle
If possible, it would be better to consider this aspect
Data used to report results and how to write a bug report
After executing all the test cases written like this
you will get a result
Then, a report is usually made based on this result
One area is usually tested on one sheet
then the overall result is obtained
Then, based on the results of that test case
an overall analysis is done to determine how this test was
Usually, we report on two concepts:
test performance rate and test success rate
Test performance rate is literally Progress Rate
how much the test
we prepared has progressed
If it took a couple days
It will be a rate that shows
how much progress we have made
by testing till the day of the report
It shows the progress of the testing work
made to date as a percentage
The success rate is different
from the progress rate
The success rate is called Test Pass Rate
It shows how many test cases
actually passed compared to
all test cases performed
Of course, the higher this number the better
When this number reaches 90% or 100%
We can see that the function is truly performed
as we want and that it has been verified as desired
Many organizations set a target number
for test success rate in advance
So we can take this number
to use as quality standards
So, if we aim for a
pass rate of about 99.7%
Set a standard where you can say
we can only proceed forward
if this rate is achieved
When we discover bugs
while performing the test
A report must be written immediately upon discovering
This is a bit different from the
test case or report we've been talking about
However, since these two documents are written the most
Let’s take a quick look at how to best write a bug report
A bug literally means an insect
In software engineering, bugs are
often referred to as being alive
That is why it is important
to report a bug right away
When time passes
This bug may disappear or change
due to some unknown cause we couldn't figure out on time
It is important to report any bugs
we find right away at the right time
Because this is a document just like a test case
it should be delivered objectively
and ensured that it can be modified effectively
That is the justification of a document
Basically the same goes for test cases
and all documentation
you should be cautious of grammar and typos
it is important to write the document in a way
the context and intent are well conveyed
and the basics are grammar and typos
I don't think those should exist
You should always keep an eye
In the case of bug reports
one of the important keywords is recall rate
known as Reproducibility
How often does this bug reproduce the same symptoms?
Does this bug always appear when the same conditions are created?
Or does it appear only a few times even though the conditions are the same?
Test these multiple times
and write the rates in which they happen
One of the most difficult issues to solve is
is an issue that only happens once
When it happened only once and the symptoms were very serious
These things are also quite difficult for developers to track
So, one of the most important pieces of information
would be the recall rate
Rules should be set for this too, for example
you must retry at least 3-4 times
when a symptom is discovered
and the same symptom
reappeared several times
these things should be written in the bug report
And that would be the first step
for developers to track
Other important keywords in writing bug reports
are Importance and Priority
Importance indicates how serious the symptom is
Priority is how quickly this issue
needs to be fixed
Some organizations use a mix of
importance and priority
Or the concept of priority may be used
as importance and vice versa
These are applied differently in each organization
The important thing is to set one rule and apply it
so that these things do not get confusing
Usually, if the importance is high
the priority follows
Such as preventing a system crash
Or when an issue arises that makes it
impossible to proceed that doesn't have a woraround
payments not working
or the payment was made but
when the game money is not paid
These are issues of all high importance
These will be high both in
importance and priority
An organization must
judge these things and make a rule
that applies within
in order to
reduce confusion
So I think it would be a good idea to always keep that in mind
There are limitations in explaining
all of these in text
If a very severe issue appeared
and it was written in text
there is a risk of different people
interpreting them differently
So it is best to use images or videos
Areas that can be covered by a single image or video
are quite effective
When things are difficult
to put in words or written in text
simply attach images or videos
and maybe captions
The developers can usually utilize those
So take that into account
Especially when this issue is not just a momentary one
but when it
effects the flow of the game
It is best to attach a video
I've briefly talked about test cases
and how to report bugs
I've repeated multiple times but
these are not the ultimate rules
same with importance and priority
and applying these tips
Each organization should understand
the importance of these reports
and create a rule
Applying them across the group is the most important
In that process, I hope my lecture
may serve as reference material
This will conclude this lecture Thank you
Structure and writing method of test cases
Structure of test cases Unique number, category, prerequisites
Test steps, expected results, actual results, final results
How to write a test case
Must be easy for anyone to understand Be concise
A positive result must mean Pass, and a negative result must mean Fail
Avoid verifying multiple requirements in one case
Data used to report results and how to write a bug report Data used to report results
Test Progress Rate: Ratio of test cases performed compared to total test cases
Test Pass Rate: Compared to performed test cases Percentage of test cases passed
How to write a bug report
Be careful about grammar and typos Recall rate must be recorded
Importance and priority must be recorded Use images and videos
The End