WEBVTT 1 00:00:05.675 --> 00:00:09.825 Game - Basics Game Test and Result Analysis 2 00:00:23.725 --> 00:00:24.225 Hello 3 00:00:24.225 --> 00:00:27.675 My name is Sukjoon Jeon, and I work in Devsisters’ technical department 4 00:00:27.675 --> 00:00:30.175 In Previous lectures, I've talked about the definition of game QA 5 00:00:30.175 --> 00:00:31.874 the types of work, and then the trends 6 00:00:31.874 --> 00:00:34.974 For this lecture I want to talk about 7 00:00:34.974 --> 00:00:37.524 how to write test documents effectively 8 00:00:37.524 --> 00:00:40.224 First objective is understanding how to 9 00:00:40.224 --> 00:00:42.824 write the test case and the structure of it 10 00:00:42.824 --> 00:00:45.074 We will learn about some tips that are 11 00:00:45.074 --> 00:00:48.624 actually being used in writing test cases 12 00:00:48.624 --> 00:00:51.424 Then we will look at how to calculate data 13 00:00:51.424 --> 00:00:54.174 when performing tests 14 00:00:54.174 --> 00:00:56.074 through these test cases 15 00:00:56.074 --> 00:00:58.424 and then reporting the results 16 00:00:58.424 --> 00:01:01.374 Lastly, we will look at how to communicate the bugs 17 00:01:01.374 --> 00:01:03.924 discovered during the test 18 00:01:03.924 --> 00:01:07.574 and how to write a bug report about that 19 00:01:07.574 --> 00:01:11.574 Structure and Writing Method of Test Cases 20 00:01:12.274 --> 00:01:14.824 The phrase 'Test Case' is 21 00:01:14.824 --> 00:01:16.574 actually frequently used in field 22 00:01:16.574 --> 00:01:19.924 but is not easy to define it 23 00:01:19.924 --> 00:01:23.174 I'll give you an real life example 24 00:01:23.174 --> 00:01:24.574 The phrase was first used 25 00:01:24.574 --> 00:01:26.774 in software engineering 26 00:01:26.774 --> 00:01:29.524 Then, since the purpose of 27 00:01:29.524 --> 00:01:31.574 these tests are to achieve a purpose 28 00:01:31.574 --> 00:01:34.223 So, the documents needed to perform such a test 29 00:01:34.223 --> 00:01:37.773 ere defined as test cases 30 00:01:37.773 --> 00:01:38.873 As I said before 31 00:01:38.873 --> 00:01:41.273 this is one of the most used documents in QA 32 00:01:41.273 --> 00:01:42.473 We write the test case first 33 00:01:42.473 --> 00:01:45.923 and then report the discovered bugs and errors 34 00:01:45.923 --> 00:01:48.073 These two are the most used 35 00:01:48.073 --> 00:01:49.473 Test cases are 36 00:01:49.473 --> 00:01:52.473 generally talked about a lot 37 00:01:52.473 --> 00:01:54.073 The biggest trait of it 38 00:01:54.073 --> 00:01:56.223 is that a case should always be structural 39 00:01:56.223 --> 00:01:58.723 It must be systematically organized 40 00:01:58.723 --> 00:02:01.323 The writer of the case should always keep in mind that 41 00:02:01.323 --> 00:02:03.973 the tests performed on theses cases 42 00:02:03.973 --> 00:02:06.573 need to be done effectively and efficiently 43 00:02:06.573 --> 00:02:08.123 Other than Test Case 44 00:02:08.123 --> 00:02:11.173 another frequently used term is 'checklist' 45 00:02:11.173 --> 00:02:14.073 The checklist relatively 46 00:02:14.073 --> 00:02:16.023 serves a simpler purpose 47 00:02:16.023 --> 00:02:18.073 Although there is no absolute way of 48 00:02:18.073 --> 00:02:19.723 distinguishing between 49 00:02:19.723 --> 00:02:22.273 which document is checklist and which is test case 50 00:02:22.273 --> 00:02:26.523 They are simply used in their respective organizations 51 00:02:26.523 --> 00:02:29.073 If the document is relatively simpler, it would be a checklist 52 00:02:29.073 --> 00:02:30.523 if it is more complex, a test case 53 00:02:30.523 --> 00:02:32.523 This is one way of differentiating them 54 00:02:32.523 --> 00:02:34.023 In my workplace 55 00:02:34.023 --> 00:02:35.923 The most biggest difference 56 00:02:35.923 --> 00:02:38.322 between the two is 57 00:02:38.322 --> 00:02:41.622 if it has specifically expected results written 58 00:02:41.622 --> 00:02:44.772 or is there a space to write 59 00:02:44.772 --> 00:02:46.672 the expected results 60 00:02:46.672 --> 00:02:50.172 If that specified expectation is listed 61 00:02:50.172 --> 00:02:52.572 we see it as a test case 62 00:02:52.572 --> 00:02:54.222 If it is in a check-box style 63 00:02:54.222 --> 00:02:56.272 to simply check what is working 64 00:02:56.272 --> 00:02:57.872 we group them as checklists 65 00:02:57.872 --> 00:03:00.572 I've said that test cases are more complex 66 00:03:00.572 --> 00:03:02.322 and that they need to be 67 00:03:02.322 --> 00:03:04.322 structurally organized 68 00:03:04.322 --> 00:03:07.072 So let's go ahead and see 69 00:03:07.072 --> 00:03:08.722 what they consist of 70 00:03:08.722 --> 00:03:11.172 Generally, spreadsheets are used 71 00:03:11.172 --> 00:03:14.072 Nowadays, different tools like Notion are used as well 72 00:03:14.072 --> 00:03:15.722 to write a test case 73 00:03:15.722 --> 00:03:18.822 but in all those tools, the first thing you see 74 00:03:18.822 --> 00:03:20.572 is the case's unique number 75 00:03:20.572 --> 00:03:22.272 It could be in all numbers 76 00:03:22.272 --> 00:03:24.972 it could be a mix of letters and numbers 77 00:03:24.972 --> 00:03:27.222 some places use tags 78 00:03:27.222 --> 00:03:30.322 But the sole purpose of them all is 79 00:03:30.322 --> 00:03:33.172 to identify a unique test case 80 00:03:33.172 --> 00:03:36.322 Thus, the same number is never repeated 81 00:03:36.322 --> 00:03:40.521 That is why every single case has a unique sequence 82 00:03:40.521 --> 00:03:45.021 The reason for that is, for example 83 00:03:45.021 --> 00:03:47.171 Let's say we have a problem within the case 84 00:03:47.171 --> 00:03:49.871 We wouldn't say, for example 85 00:03:49.871 --> 00:03:51.871 "Did you see in which environment it was tested?" 86 00:03:51.871 --> 00:03:54.671 let's say the unique number of the test is 3 87 00:03:54.671 --> 00:03:56.421 We can say, "did you check case number 3?" 88 00:03:56.421 --> 00:03:58.221 which is a way more simple 89 00:03:58.221 --> 00:03:59.371 yet effective communication method 90 00:03:59.371 --> 00:04:01.521 It is also easier to communicate with developers 91 00:04:01.521 --> 00:04:02.671 with this 92 00:04:02.671 --> 00:04:04.421 That is the reason for the unique numbers 93 00:04:04.421 --> 00:04:07.221 We can say that the traceability of test cases 94 00:04:07.221 --> 00:04:09.621 can be improved with these 95 00:04:09.621 --> 00:04:12.321 And after the case's unique number 96 00:04:12.321 --> 00:04:13.571 you will see 97 00:04:13.571 --> 00:04:16.171 the Category that indicates 98 00:04:16.171 --> 00:04:17.671 which area of this product 99 00:04:17.671 --> 00:04:19.221 this test case is testing 100 00:04:19.221 --> 00:04:21.221 In case of games, generally 101 00:04:21.221 --> 00:04:23.471 level 2 or level 3 is specified 102 00:04:23.471 --> 00:04:24.871 It does not follow a set rule 103 00:04:24.871 --> 00:04:26.371 It differs by organization 104 00:04:26.371 --> 00:04:29.521 One could write level 2 or 3 depending on 105 00:04:29.521 --> 00:04:31.821 what type of testing the organization is doing 106 00:04:31.821 --> 00:04:33.471 If the case is very definite 107 00:04:33.471 --> 00:04:35.271 the category may not be specified at all 108 00:04:35.271 --> 00:04:37.271 Let's say we are 109 00:04:37.271 --> 00:04:39.621 conducting a character test 110 00:04:39.621 --> 00:04:41.670 the character will be level 1 111 00:04:41.670 --> 00:04:43.820 and there will be separate classes within the characters 112 00:04:43.820 --> 00:04:47.670 If we test warrior character, then level two warrior 113 00:04:47.670 --> 00:04:50.870 The warrior must have various areas of action 114 00:04:50.870 --> 00:04:54.320 if we test skills among them 115 00:04:54.320 --> 00:04:55.670 Character, Warrior, Skill 116 00:04:55.670 --> 00:04:57.670 It could be written in three levels like this 117 00:04:57.670 --> 00:05:00.970 If we give a specified category like this 118 00:05:00.970 --> 00:05:02.970 We can see right away 119 00:05:02.970 --> 00:05:05.570 Which area of the game this case 120 00:05:05.570 --> 00:05:07.820 is testing very easily and effectively 121 00:05:07.820 --> 00:05:10.370 It is best if theses things 122 00:05:10.370 --> 00:05:12.870 are structured the same as a proposal 123 00:05:12.870 --> 00:05:15.270 If the proposal of the game is structured 124 00:05:15.270 --> 00:05:17.470 in the category, character, warrior, skill 125 00:05:17.470 --> 00:05:20.013 then the test case should also follow that structure 126 00:05:20.013 --> 00:05:23.163 and specify the category the same way 127 00:05:23.163 --> 00:05:25.513 This is the traceability I talked about earlier 128 00:05:25.513 --> 00:05:28.614 It is easier for us to track or spot 129 00:05:28.614 --> 00:05:30.714 what this test case is confirming 130 00:05:30.714 --> 00:05:33.620 So it is very common to write the unique number 131 00:05:33.620 --> 00:05:36.470 and then category at the front 132 00:05:36.470 --> 00:05:39.370 Next we have the prerequisites 133 00:05:39.370 --> 00:05:41.320 It is also called Precondition 134 00:05:41.320 --> 00:05:43.670 and lists the conditions necessary before 135 00:05:43.670 --> 00:05:46.569 performing the test case 136 00:05:46.569 --> 00:05:48.869 This is not needed in all test cases 137 00:05:48.869 --> 00:05:51.869 But only in specific cases that 138 00:05:51.869 --> 00:05:53.769 require certain conditions 139 00:05:53.769 --> 00:05:56.619 you would only write them then 140 00:05:56.619 --> 00:05:59.819 For example if a certain skill 141 00:05:59.819 --> 00:06:01.769 like an ultimate skill 142 00:06:01.769 --> 00:06:03.169 that is not always available to use 143 00:06:03.169 --> 00:06:04.869 In most games 144 00:06:04.869 --> 00:06:07.869 you have to achieve a certain condition 145 00:06:07.869 --> 00:06:09.619 in order to use the ultimate skill 146 00:06:09.619 --> 00:06:12.669 So the Precondition clarifies that 147 00:06:12.669 --> 00:06:14.119 the achievement is needed 148 00:06:14.119 --> 00:06:16.769 So, if you need to achieve certain conditions 149 00:06:16.769 --> 00:06:19.419 or meet the conditions to use the ultimate skill 150 00:06:19.419 --> 00:06:21.719 these things must be specified in the preconditions 151 00:06:21.719 --> 00:06:23.669 to actually perform the test case 152 00:06:23.669 --> 00:06:26.269 This is needed only when it is present 153 00:06:26.269 --> 00:06:27.719 but is not needed in 154 00:06:27.719 --> 00:06:29.419 all test cases 155 00:06:29.419 --> 00:06:31.169 The next most important 156 00:06:31.169 --> 00:06:33.419 part that takes the most space in the test case 157 00:06:33.419 --> 00:06:34.969 is the Test Step 158 00:06:34.969 --> 00:06:37.969 This is a complete description of processes 159 00:06:37.969 --> 00:06:40.269 performed by QA people and testers 160 00:06:40.269 --> 00:06:42.669 Many people use bullets or 161 00:06:42.669 --> 00:06:43.969 numbers to list 162 00:06:43.969 --> 00:06:45.719 It doesn't have a set template 163 00:06:45.719 --> 00:06:48.568 It just needs to be written at a level that is 164 00:06:48.568 --> 00:06:50.318 comfortable to communicate within the organization 165 00:06:50.318 --> 00:06:53.118 Sometimes these steps are 166 00:06:53.118 --> 00:06:55.368 replaced by input values 167 00:06:55.368 --> 00:06:58.818 If you need to test various input values ​​ 168 00:06:58.818 --> 00:07:02.368 in a box that allows different inputs 169 00:07:02.368 --> 00:07:04.968 In some cases, only such input values ​​are specified 170 00:07:04.968 --> 00:07:10.968 But it should be written on a level where 171 00:07:10.968 --> 00:07:12.318 all of the testers of 172 00:07:12.318 --> 00:07:15.068 this test case could immediately know 173 00:07:15.068 --> 00:07:17.418 what they mean 174 00:07:17.418 --> 00:07:19.918 So even if it only has input values 175 00:07:19.918 --> 00:07:20.868 one should be able to tell 176 00:07:20.868 --> 00:07:23.518 why the input value is there 177 00:07:23.518 --> 00:07:24.718 in context 178 00:07:24.718 --> 00:07:27.418 If the steps were numbered 179 00:07:27.418 --> 00:07:29.718 and I were to follow those 180 00:07:29.718 --> 00:07:31.668 it is best that no additional communication is needed 181 00:07:31.668 --> 00:07:34.118 So when you follow the process in order 182 00:07:34.118 --> 00:07:36.968 The second step falls naturally after the first 183 00:07:36.968 --> 00:07:38.918 and after the second, the third step 184 00:07:38.918 --> 00:07:40.168 should follow naturally 185 00:07:40.168 --> 00:07:41.668 This type of test case 186 00:07:41.668 --> 00:07:43.368 where things fall in place natrually 187 00:07:43.368 --> 00:07:46.268 is the most effectively written test case 188 00:07:46.268 --> 00:07:50.518 So the next thing to specify is the 189 00:07:50.518 --> 00:07:52.568 expected results would be 190 00:07:52.568 --> 00:07:54.317 after following these steps 191 00:07:54.317 --> 00:07:56.517 This expected result is the part that I 192 00:07:56.517 --> 00:07:59.217 mentioned earlier as being the biggest difference 193 00:07:59.217 --> 00:08:00.817 between a checklist and a test case 194 00:08:00.817 --> 00:08:04.417 An expected result generally 195 00:08:04.417 --> 00:08:07.767 should be written in requirements or proposals 196 00:08:07.767 --> 00:08:09.167 For example 197 00:08:09.167 --> 00:08:11.467 When a specific skill is used 198 00:08:11.467 --> 00:08:13.667 the expected result is as such 199 00:08:13.667 --> 00:08:15.267 That should be specified there 200 00:08:15.267 --> 00:08:17.967 But sometimes the tests cannot 201 00:08:17.967 --> 00:08:20.267 have an expected result 202 00:08:20.267 --> 00:08:22.617 Common sense tells us that when the forward button is pressed 203 00:08:22.617 --> 00:08:24.767 the character moves forward and there is 204 00:08:24.767 --> 00:08:27.017 no need to write down the 205 00:08:27.017 --> 00:08:29.367 expected results in detail 206 00:08:29.367 --> 00:08:31.667 In cases where these expected results are specified in detail 207 00:08:31.667 --> 00:08:33.317 you must write the expected results so 208 00:08:33.317 --> 00:08:34.817 that everyone can understand them 209 00:08:34.817 --> 00:08:37.767 That is what determines the test case's worth 210 00:08:37.767 --> 00:08:42.967 But writing phrases like "normal results are expected" 211 00:08:42.967 --> 00:08:45.317 or "should be normal" just to fill in the 212 00:08:45.317 --> 00:08:47.567 expected results part 213 00:08:47.567 --> 00:08:49.067 should be avoided 214 00:08:49.067 --> 00:08:51.567 If then, the results part should be better left blank 215 00:08:51.567 --> 00:08:53.317 Only fill out these expectations part 216 00:08:53.317 --> 00:08:54.916 when there would be 217 00:08:54.916 --> 00:08:58.216 results we can look into in detail 218 00:08:58.216 --> 00:09:02.266 After that there is space to write the actual results 219 00:09:02.266 --> 00:09:04.316 This may differ from group by group 220 00:09:04.316 --> 00:09:07.366 But a general rule of thumb is to write this result 221 00:09:07.366 --> 00:09:09.316 when you wrote the expected result 222 00:09:09.316 --> 00:09:12.716 Expected result is what you would expect 223 00:09:12.716 --> 00:09:14.266 before writing this case 224 00:09:14.266 --> 00:09:16.116 Forecasting that the 225 00:09:16.116 --> 00:09:18.566 result might come out a certain way 226 00:09:18.566 --> 00:09:21.516 The Actual result is where you write down 227 00:09:21.516 --> 00:09:23.816 the active result after performing the test 228 00:09:23.816 --> 00:09:26.766 So if the actual result is 229 00:09:26.766 --> 00:09:29.266 different from the expected result 230 00:09:29.266 --> 00:09:31.566 This specific case will be filed as a Fail 231 00:09:31.566 --> 00:09:33.816 This is an issue of itself 232 00:09:33.816 --> 00:09:35.816 And this issue or flaw 233 00:09:35.816 --> 00:09:39.016 should be registered through the Bug Tracking System 234 00:09:39.016 --> 00:09:42.866 Interpretation of whether the actual results match the expected results 235 00:09:42.866 --> 00:09:47.366 and how they differ may be slightly different 236 00:09:47.366 --> 00:09:49.516 It may also differ depending on 237 00:09:49.516 --> 00:09:52.516 the tester's individual experience or 238 00:09:52.516 --> 00:09:53.966 how the organization views this part 239 00:09:53.966 --> 00:09:56.116 But strictly speaking 240 00:09:56.116 --> 00:09:58.115 Even if the expected results and actual results are only slightly different 241 00:09:58.115 --> 00:09:59.665 this should be filed as a Fail 242 00:09:59.665 --> 00:10:01.715 Those things help us manage 243 00:10:01.715 --> 00:10:05.015 quality a little more strictly 244 00:10:05.515 --> 00:10:08.065 After all of these are done 245 00:10:08.065 --> 00:10:10.865 after comparing the actual and expected results 246 00:10:10.865 --> 00:10:14.415 Now you can record the test case's Final Result 247 00:10:14.415 --> 00:10:18.465 Usually, we have about four main expectations 248 00:10:18.465 --> 00:10:19.665 This might be different by groups 249 00:10:19.665 --> 00:10:21.315 Some only write two 250 00:10:21.315 --> 00:10:22.765 Some write three 251 00:10:22.765 --> 00:10:26.565 But in general we use four 252 00:10:26.565 --> 00:10:29.853 The first is to file it as a Pass 253 00:10:29.853 --> 00:10:31.165 This is the case 254 00:10:31.165 --> 00:10:34.015 where the expected and actual results align 255 00:10:34.015 --> 00:10:36.465 The Pass means that 256 00:10:36.465 --> 00:10:38.865 this case was performed well and the 257 00:10:38.865 --> 00:10:41.965 purpose we wanted was verified 258 00:10:41.965 --> 00:10:43.765 Next we have the Fail 259 00:10:43.765 --> 00:10:47.015 Fail is when the result is different 260 00:10:47.015 --> 00:10:50.015 We expected a certain outcome 261 00:10:50.015 --> 00:10:51.865 but have a different one 262 00:10:51.865 --> 00:10:54.574 Then, you would write the difference in detail 263 00:10:54.574 --> 00:10:57.374 and must report it as an issue 264 00:10:57.374 --> 00:11:00.164 and this case will be set as a Fail 265 00:11:00.164 --> 00:11:03.664 The third is N/A which is 266 00:11:03.664 --> 00:11:06.264 also known as Not Available 267 00:11:06.264 --> 00:11:09.214 This is a case where the original plan 268 00:11:09.214 --> 00:11:11.164 implemented 10 features in a build 269 00:11:11.164 --> 00:11:12.714 but only 7 were actually perfomred 270 00:11:12.714 --> 00:11:16.614 and numbers 8, 9, 10 were not impleneted 271 00:11:16.614 --> 00:11:20.014 then 8, 9, 10 would be filed as N/A 272 00:11:20.014 --> 00:11:23.664 It would be nice if all the 273 00:11:23.664 --> 00:11:25.464 expected results were all delivered 274 00:11:25.464 --> 00:11:27.574 when we see the build notes 275 00:11:27.574 --> 00:11:29.764 but there are a lot of instances where this is not the case 276 00:11:29.764 --> 00:11:32.653 those things should be processed as N/A in advance 277 00:11:32.653 --> 00:11:35.214 so that errors do not occur later 278 00:11:35.214 --> 00:11:36.664 when we calculate the Pass Rate 279 00:11:36.664 --> 00:11:39.864 So, it is necessary to manage these things strictly too 280 00:11:39.864 --> 00:11:42.164 Lastly there is Block 281 00:11:42.164 --> 00:11:46.564 Block occurs when a specific prerequisite cannot be executed 282 00:11:46.564 --> 00:11:49.564 Then all the test cases that cannot be executed 283 00:11:49.564 --> 00:11:53.214 because of that should be also filed as Block 284 00:11:53.214 --> 00:11:55.364 For example after you log in 285 00:11:55.364 --> 00:11:57.864 You would start the test in categories 286 00:11:57.864 --> 00:11:59.414 Like PvE or PvP 287 00:11:59.414 --> 00:12:01.163 But if the log in is not complete 288 00:12:01.163 --> 00:12:04.763 You won't be able to open those areas of the game 289 00:12:04.763 --> 00:12:07.513 Even if the PvE or PvP was implented 290 00:12:07.513 --> 00:12:09.013 because we couldn't log in 291 00:12:09.013 --> 00:12:11.113 we wouldn't be able to perform the test 292 00:12:11.113 --> 00:12:13.063 In this case, the part where we couldn't log in 293 00:12:13.063 --> 00:12:14.413 would be processed as Fail 294 00:12:14.413 --> 00:12:16.563 and the PvE and PvP that we couldn't check because of the login issue 295 00:12:16.563 --> 00:12:19.013 would be processed as Block 296 00:12:19.013 --> 00:12:20.713 We need to specify these things so that 297 00:12:20.713 --> 00:12:22.713 we can understand what kind of situation each result 298 00:12:22.713 --> 00:12:26.563 will be in this build after the test is finished 299 00:12:26.563 --> 00:12:28.663 If it came out as expected 300 00:12:28.663 --> 00:12:30.113 or if it has any issues 301 00:12:30.113 --> 00:12:32.763 or if something is not developed in the build yet 302 00:12:32.763 --> 00:12:35.363 or the test was not performed because of such issue 303 00:12:35.363 --> 00:12:36.913 we can know things like this 304 00:12:36.913 --> 00:12:39.513 So, it is very important for testers 305 00:12:39.513 --> 00:12:41.653 to judge and specify these things 306 00:12:41.653 --> 00:12:43.713 reasonably and objectively 307 00:12:43.713 --> 00:12:47.113 These test cases don't seem that easy 308 00:12:47.113 --> 00:12:49.663 You might think there are lots of conditions 309 00:12:49.663 --> 00:12:52.013 and writing them may not be easy 310 00:12:52.013 --> 00:12:54.213 There are a few tips in 311 00:12:54.213 --> 00:12:55.913 writing these test cases 312 00:12:55.913 --> 00:12:58.363 When you look at a test case for a first time 313 00:12:58.363 --> 00:12:59.513 it will seem very complex 314 00:12:59.513 --> 00:13:01.963 but there should be no issue in following them 315 00:13:01.963 --> 00:13:04.512 So even if there is no relevant context or background knowledge 316 00:13:04.512 --> 00:13:07.112 You should be able to get results when 317 00:13:07.112 --> 00:13:08.912 testing by following steps 1, 2, 3, and 4 318 00:13:08.912 --> 00:13:10.762 However, it is not that easy 319 00:13:10.762 --> 00:13:12.712 to objectively write things 320 00:13:12.712 --> 00:13:15.712 that anyone can recognize 321 00:13:15.712 --> 00:13:17.512 Also, in order to perform these test cases 322 00:13:17.512 --> 00:13:20.012 here are times when a lot of communication is involved 323 00:13:20.012 --> 00:13:22.912 That is an example of a test case that is not right 324 00:13:22.912 --> 00:13:25.512 You have to keep asking 325 00:13:25.512 --> 00:13:26.662 the author of this test case 326 00:13:26.662 --> 00:13:27.562 "What is this?" 327 00:13:27.562 --> 00:13:29.212 "What does this word signify?" 328 00:13:29.212 --> 00:13:30.662 Asking lots of questions like these 329 00:13:30.662 --> 00:13:33.162 is definitely not ideal 330 00:13:33.162 --> 00:13:35.362 If the test case is written 331 00:13:35.362 --> 00:13:37.712 really objective and simply 332 00:13:37.712 --> 00:13:39.812 performing should not be a problem 333 00:13:39.812 --> 00:13:41.762 but to have to keep asking 334 00:13:41.762 --> 00:13:43.562 the communication is inefficient 335 00:13:43.562 --> 00:13:45.562 and cuts down our resources 336 00:13:45.562 --> 00:13:47.612 You can say then 337 00:13:47.612 --> 00:13:49.512 the test case was done incorrectly 338 00:13:49.512 --> 00:13:50.562 To continue 339 00:13:50.562 --> 00:13:51.762 Constructing a case well 340 00:13:51.762 --> 00:13:53.512 means it cannot be wordy or complex 341 00:13:53.512 --> 00:13:54.612 It needs to be simple 342 00:13:54.612 --> 00:13:59.612 That is way we use numbers or bullets 343 00:13:59.612 --> 00:14:02.112 Basically, it needs to have a good readability 344 00:14:02.112 --> 00:14:05.411 I think things like letter spacing and 345 00:14:05.411 --> 00:14:07.961 blank spaces are very important as well 346 00:14:07.961 --> 00:14:09.461 these all take part in readability 347 00:14:09.461 --> 00:14:12.461 Because there are not just a few test cases to read 348 00:14:12.461 --> 00:14:14.961 In some instances there may be hundreds and thousands 349 00:14:14.961 --> 00:14:19.211 In order for us to go through that without loosing our concentration 350 00:14:19.211 --> 00:14:21.211 the document's readability is very important 351 00:14:21.211 --> 00:14:24.561 So it is a tester's necessary skill to 352 00:14:24.561 --> 00:14:26.311 write them efficiently 353 00:14:26.311 --> 00:14:29.111 A concise and highly readable document 354 00:14:29.111 --> 00:14:31.361 like this is needed to reduce the cost of communication 355 00:14:31.361 --> 00:14:33.311 to understand the unclear words 356 00:14:33.311 --> 00:14:36.711 as mentioned earlier 357 00:14:36.711 --> 00:14:39.111 Therefore when choosing even one word 358 00:14:39.111 --> 00:14:41.261 it is best to use a word that 359 00:14:41.261 --> 00:14:44.061 all relevant departments within a department 360 00:14:44.061 --> 00:14:46.611 can easily understand and agree on 361 00:14:46.611 --> 00:14:48.261 You cannot think for yourself 362 00:14:48.261 --> 00:14:51.211 So, it is very important to review the test case 363 00:14:51.211 --> 00:14:54.061 so that you can write it objectively 364 00:14:54.061 --> 00:14:58.111 As I said many times earlier 365 00:14:58.111 --> 00:15:00.711 anyone should be able to understand it 366 00:15:00.711 --> 00:15:03.761 The next thing I want to bring up 367 00:15:03.761 --> 00:15:07.311 is something you might overlook easily 368 00:15:07.311 --> 00:15:09.361 Positive results means Pass 369 00:15:09.361 --> 00:15:11.810 and negative results should mean Fail 370 00:15:11.810 --> 00:15:16.210 For example let's say we have a test case 371 00:15:16.210 --> 00:15:18.460 that asks 372 00:15:18.460 --> 00:15:20.210 Does the text contain typos? 373 00:15:20.210 --> 00:15:23.560 If it is a Pass, then you might think there are no typos 374 00:15:23.560 --> 00:15:25.560 When you ask 'if there are' and you answer 'yes' 375 00:15:25.560 --> 00:15:28.810 you come to interpret it as 'it is included' 376 00:15:28.810 --> 00:15:31.360 So since the test was a Pass 377 00:15:31.360 --> 00:15:33.110 You would think there is a typo 378 00:15:33.110 --> 00:15:35.660 In this case we need to change the 379 00:15:35.660 --> 00:15:36.910 test case's actual steps 380 00:15:36.910 --> 00:15:39.960 When asked 'Are there no typos in the text?' 381 00:15:39.960 --> 00:15:42.460 and it says that is true (pass) 382 00:15:42.460 --> 00:15:45.860 then you think Okay, that's positive, the expected results are good 383 00:15:45.860 --> 00:15:48.710 You will be able to understand that the actual results came out that way 384 00:15:48.710 --> 00:15:51.110 So generally for Pass 385 00:15:51.110 --> 00:15:54.260 we use a lot of positive signals such as the color green 386 00:15:54.260 --> 00:15:56.910 You need to handle it carefully so that 387 00:15:56.910 --> 00:15:59.910 others don't interpret the those unnecessarily 388 00:15:59.910 --> 00:16:01.010 You must not try to 389 00:16:01.010 --> 00:16:02.610 verify multiple requirements in one case 390 00:16:02.610 --> 00:16:07.460 There are many instances like this when you don't have time 391 00:16:07.460 --> 00:16:09.160 or want to reduce the number of cases 392 00:16:09.160 --> 00:16:11.360 If possible, one test case should be 393 00:16:11.360 --> 00:16:13.559 able to verify only one requirement 394 00:16:13.559 --> 00:16:15.509 in some cases, there are quite a few people who combine 395 00:16:15.509 --> 00:16:18.609 two or three conditions into one case to verify them 396 00:16:18.609 --> 00:16:21.309 Those cases, would not be a problem 397 00:16:21.309 --> 00:16:24.259 if the case was a Pass 398 00:16:24.259 --> 00:16:26.359 But when the case has a Fail 399 00:16:26.359 --> 00:16:29.159 You have to go and analyze each one at that time to see 400 00:16:29.159 --> 00:16:32.259 which of those three conditions was the cause 401 00:16:32.259 --> 00:16:35.059 That would take up even more time 402 00:16:35.059 --> 00:16:36.859 So, it is necessary to write it separately 403 00:16:36.859 --> 00:16:38.509 so that one case can only verify 404 00:16:38.509 --> 00:16:40.409 one requirement 405 00:16:40.409 --> 00:16:44.559 even if it takes more time in the beginning 406 00:16:44.559 --> 00:16:45.709 If there are x number of tests 407 00:16:45.709 --> 00:16:47.809 then there obviously must be x number of test cases 408 00:16:47.809 --> 00:16:50.059 Also it is good to additionally verify 409 00:16:50.059 --> 00:16:53.209 what happens when you combine x conditions 410 00:16:53.209 --> 00:16:55.409 Although it is a matter of principle 411 00:16:55.409 --> 00:16:58.359 If possible, it would be better to consider this aspect 412 00:16:58.359 --> 00:17:02.359 Data used to report results and how to write a bug report 413 00:17:02.359 --> 00:17:05.559 After executing all the test cases written like this 414 00:17:05.559 --> 00:17:06.659 you will get a result 415 00:17:06.659 --> 00:17:10.109 Then, a report is usually made based on this result 416 00:17:10.109 --> 00:17:12.858 One area is usually tested on one sheet 417 00:17:12.858 --> 00:17:14.808 then the overall result is obtained 418 00:17:14.808 --> 00:17:17.458 Then, based on the results of that test case 419 00:17:17.458 --> 00:17:21.558 an overall analysis is done to determine how this test was 420 00:17:21.558 --> 00:17:24.508 Usually, we report on two concepts: 421 00:17:24.508 --> 00:17:27.958 test performance rate and test success rate 422 00:17:27.958 --> 00:17:30.958 Test performance rate is literally Progress Rate 423 00:17:30.958 --> 00:17:32.358 how much the test 424 00:17:32.358 --> 00:17:35.208 we prepared has progressed 425 00:17:35.208 --> 00:17:37.908 If it took a couple days 426 00:17:37.908 --> 00:17:39.758 It will be a rate that shows 427 00:17:39.758 --> 00:17:42.958 how much progress we have made 428 00:17:42.958 --> 00:17:44.658 by testing till the day of the report 429 00:17:44.658 --> 00:17:46.858 It shows the progress of the testing work 430 00:17:46.858 --> 00:17:48.508 made to date as a percentage 431 00:17:48.508 --> 00:17:51.908 The success rate is different 432 00:17:51.908 --> 00:17:53.508 from the progress rate 433 00:17:53.508 --> 00:17:55.358 The success rate is called Test Pass Rate 434 00:17:55.358 --> 00:17:58.108 It shows how many test cases 435 00:17:58.108 --> 00:17:59.808 actually passed compared to 436 00:17:59.808 --> 00:18:02.608 all test cases performed 437 00:18:02.608 --> 00:18:04.958 Of course, the higher this number the better 438 00:18:04.958 --> 00:18:07.358 When this number reaches 90% or 100% 439 00:18:07.358 --> 00:18:09.858 We can see that the function is truly performed 440 00:18:09.858 --> 00:18:13.508 as we want and that it has been verified as desired 441 00:18:13.508 --> 00:18:16.657 Many organizations set a target number 442 00:18:16.657 --> 00:18:18.107 for test success rate in advance 443 00:18:18.107 --> 00:18:19.657 So we can take this number 444 00:18:19.657 --> 00:18:21.107 to use as quality standards 445 00:18:21.107 --> 00:18:25.057 So, if we aim for a 446 00:18:25.057 --> 00:18:26.607 pass rate of about 99.7% 447 00:18:26.607 --> 00:18:29.457 Set a standard where you can say 448 00:18:29.457 --> 00:18:30.957 we can only proceed forward 449 00:18:30.957 --> 00:18:33.557 if this rate is achieved 450 00:18:33.557 --> 00:18:35.257 When we discover bugs 451 00:18:35.257 --> 00:18:37.307 while performing the test 452 00:18:37.307 --> 00:18:40.507 A report must be written immediately upon discovering 453 00:18:40.507 --> 00:18:42.957 This is a bit different from the 454 00:18:42.957 --> 00:18:45.207 test case or report we've been talking about 455 00:18:45.207 --> 00:18:48.457 However, since these two documents are written the most 456 00:18:48.457 --> 00:18:51.907 Let’s take a quick look at how to best write a bug report 457 00:18:51.907 --> 00:18:55.507 A bug literally means an insect 458 00:18:55.507 --> 00:18:58.157 In software engineering, bugs are 459 00:18:58.157 --> 00:19:00.307 often referred to as being alive 460 00:19:00.307 --> 00:19:02.857 That is why it is important 461 00:19:02.857 --> 00:19:04.507 to report a bug right away 462 00:19:04.507 --> 00:19:06.407 When time passes 463 00:19:06.407 --> 00:19:08.507 This bug may disappear or change 464 00:19:08.507 --> 00:19:11.057 due to some unknown cause we couldn't figure out on time 465 00:19:11.057 --> 00:19:14.857 It is important to report any bugs 466 00:19:14.857 --> 00:19:17.207 we find right away at the right time 467 00:19:17.207 --> 00:19:20.407 Because this is a document just like a test case 468 00:19:20.407 --> 00:19:21.856 it should be delivered objectively 469 00:19:21.856 --> 00:19:24.256 and ensured that it can be modified effectively 470 00:19:24.256 --> 00:19:26.156 That is the justification of a document 471 00:19:26.156 --> 00:19:28.506 Basically the same goes for test cases 472 00:19:28.506 --> 00:19:31.006 and all documentation 473 00:19:31.006 --> 00:19:33.706 you should be cautious of grammar and typos 474 00:19:33.706 --> 00:19:36.206 it is important to write the document in a way 475 00:19:36.206 --> 00:19:38.056 the context and intent are well conveyed 476 00:19:38.056 --> 00:19:40.206 and the basics are grammar and typos 477 00:19:40.206 --> 00:19:42.256 I don't think those should exist 478 00:19:42.256 --> 00:19:44.856 You should always keep an eye 479 00:19:44.856 --> 00:19:46.856 In the case of bug reports 480 00:19:46.856 --> 00:19:49.356 one of the important keywords is recall rate 481 00:19:49.356 --> 00:19:51.606 known as Reproducibility 482 00:19:51.606 --> 00:19:56.306 How often does this bug reproduce the same symptoms? 483 00:19:56.306 --> 00:19:59.606 Does this bug always appear when the same conditions are created? 484 00:19:59.606 --> 00:20:03.056 Or does it appear only a few times even though the conditions are the same? 485 00:20:03.056 --> 00:20:04.706 Test these multiple times 486 00:20:04.706 --> 00:20:06.506 and write the rates in which they happen 487 00:20:06.506 --> 00:20:09.956 One of the most difficult issues to solve is 488 00:20:09.956 --> 00:20:11.606 is an issue that only happens once 489 00:20:11.606 --> 00:20:14.456 When it happened only once and the symptoms were very serious 490 00:20:14.456 --> 00:20:17.856 These things are also quite difficult for developers to track 491 00:20:17.856 --> 00:20:21.006 So, one of the most important pieces of information 492 00:20:21.006 --> 00:20:24.106 would be the recall rate 493 00:20:24.106 --> 00:20:27.805 Rules should be set for this too, for example 494 00:20:27.805 --> 00:20:29.805 you must retry at least 3-4 times 495 00:20:29.805 --> 00:20:32.705 when a symptom is discovered 496 00:20:32.705 --> 00:20:34.505 and the same symptom 497 00:20:34.505 --> 00:20:36.405 reappeared several times 498 00:20:36.405 --> 00:20:39.355 these things should be written in the bug report 499 00:20:39.355 --> 00:20:42.155 And that would be the first step 500 00:20:42.155 --> 00:20:43.505 for developers to track 501 00:20:43.505 --> 00:20:46.805 Other important keywords in writing bug reports 502 00:20:46.805 --> 00:20:48.655 are Importance and Priority 503 00:20:48.655 --> 00:20:51.405 Importance indicates how serious the symptom is 504 00:20:51.405 --> 00:20:55.105 Priority is how quickly this issue 505 00:20:55.105 --> 00:20:56.755 needs to be fixed 506 00:20:56.755 --> 00:21:00.455 Some organizations use a mix of 507 00:21:00.455 --> 00:21:01.655 importance and priority 508 00:21:01.655 --> 00:21:03.505 Or the concept of priority may be used 509 00:21:03.505 --> 00:21:05.505 as importance and vice versa 510 00:21:05.505 --> 00:21:08.105 These are applied differently in each organization 511 00:21:08.105 --> 00:21:10.474 The important thing is to set one rule and apply it 512 00:21:10.474 --> 00:21:13.955 so that these things do not get confusing 513 00:21:13.955 --> 00:21:17.455 Usually, if the importance is high 514 00:21:17.455 --> 00:21:19.105 the priority follows 515 00:21:19.105 --> 00:21:21.005 Such as preventing a system crash 516 00:21:21.005 --> 00:21:23.805 Or when an issue arises that makes it 517 00:21:23.805 --> 00:21:25.705 impossible to proceed that doesn't have a woraround 518 00:21:25.705 --> 00:21:26.905 payments not working 519 00:21:26.905 --> 00:21:30.405 or the payment was made but 520 00:21:30.405 --> 00:21:32.455 when the game money is not paid 521 00:21:32.455 --> 00:21:34.454 These are issues of all high importance 522 00:21:34.454 --> 00:21:36.704 These will be high both in 523 00:21:36.704 --> 00:21:38.554 importance and priority 524 00:21:39.004 --> 00:21:41.204 An organization must 525 00:21:41.204 --> 00:21:42.854 judge these things and make a rule 526 00:21:42.854 --> 00:21:44.204 that applies within 527 00:21:44.204 --> 00:21:45.854 in order to 528 00:21:45.854 --> 00:21:47.304 reduce confusion 529 00:21:47.304 --> 00:21:49.754 So I think it would be a good idea to always keep that in mind 530 00:21:49.754 --> 00:21:52.354 There are limitations in explaining 531 00:21:52.354 --> 00:21:54.854 all of these in text 532 00:21:54.854 --> 00:21:58.554 If a very severe issue appeared 533 00:21:58.554 --> 00:21:59.954 and it was written in text 534 00:21:59.954 --> 00:22:03.654 there is a risk of different people 535 00:22:03.654 --> 00:22:05.704 interpreting them differently 536 00:22:05.704 --> 00:22:09.204 So it is best to use images or videos 537 00:22:09.204 --> 00:22:11.304 Areas that can be covered by a single image or video 538 00:22:11.304 --> 00:22:12.604 are quite effective 539 00:22:12.604 --> 00:22:14.804 When things are difficult 540 00:22:14.804 --> 00:22:17.504 to put in words or written in text 541 00:22:17.504 --> 00:22:19.604 simply attach images or videos 542 00:22:19.604 --> 00:22:21.554 and maybe captions 543 00:22:21.554 --> 00:22:25.004 The developers can usually utilize those 544 00:22:25.004 --> 00:22:27.903 So take that into account 545 00:22:27.903 --> 00:22:31.853 Especially when this issue is not just a momentary one 546 00:22:31.853 --> 00:22:33.053 but when it 547 00:22:33.053 --> 00:22:35.803 effects the flow of the game 548 00:22:35.803 --> 00:22:38.953 It is best to attach a video 549 00:22:38.953 --> 00:22:41.553 I've briefly talked about test cases 550 00:22:41.553 --> 00:22:46.953 and how to report bugs 551 00:22:46.953 --> 00:22:48.553 I've repeated multiple times but 552 00:22:48.553 --> 00:22:51.453 these are not the ultimate rules 553 00:22:51.453 --> 00:22:54.153 same with importance and priority 554 00:22:54.153 --> 00:22:56.803 and applying these tips 555 00:22:56.803 --> 00:22:59.803 Each organization should understand 556 00:22:59.803 --> 00:23:01.753 the importance of these reports 557 00:23:01.753 --> 00:23:03.353 and create a rule 558 00:23:03.353 --> 00:23:05.453 Applying them across the group is the most important 559 00:23:05.453 --> 00:23:07.253 In that process, I hope my lecture 560 00:23:07.253 --> 00:23:09.503 may serve as reference material 561 00:23:09.503 --> 00:23:12.303 This will conclude this lecture Thank you 562 00:23:12.303 --> 00:23:13.203 Structure and writing method of test cases 563 00:23:13.203 --> 00:23:14.153 Structure of test cases Unique number, category, prerequisites 564 00:23:14.153 --> 00:23:15.053 Test steps, expected results, actual results, final results 565 00:23:15.053 --> 00:23:16.153 How to write a test case 566 00:23:16.153 --> 00:23:17.203 Must be easy for anyone to understand Be concise 567 00:23:17.203 --> 00:23:18.303 A positive result must mean Pass, and a negative result must mean Fail 568 00:23:18.303 --> 00:23:19.303 Avoid verifying multiple requirements in one case 569 00:23:19.303 --> 00:23:20.203 Data used to report results and how to write a bug report Data used to report results 570 00:23:20.203 --> 00:23:21.603 Test Progress Rate: Ratio of test cases performed compared to total test cases 571 00:23:21.603 --> 00:23:22.953 Test Pass Rate: Compared to performed test cases Percentage of test cases passed 572 00:23:22.953 --> 00:23:24.103 How to write a bug report 573 00:23:24.103 --> 00:23:25.653 Be careful about grammar and typos Recall rate must be recorded 574 00:23:25.653 --> 00:23:27.203 Importance and priority must be recorded Use images and videos 575 00:23:27.203 --> 00:23:28.203 The End