Performance Test Tool ≠ Performance Testing

"What is this spaghetti? What am I meant to be looking for?"

Those were my first couple of questions to my mentor when I saw the response time graphs for the first time. Without any background in performance testing, the response time graphs just looked like colorful spaghetti (like the article photo). Beautiful colors & lines but yet made no sense to me at that point in time. And this was over 10 years ago. My first real experience of the performance testing world.

I went through a considerable amount of in-house training. During the training, less focus was on performance test tool (i.e. scripts) and more on other activities that encompass performance testing. Activities such as:

Designing an application simulation model (ASM). Learning concepts such as pacing & think time and how they influence the design. Deciding how many threads/virtual users are required to generate load.

Analyzing data. Slicing & dicing the result data to observe different patterns and making sense of them.

Learning about different types of systems, protocols & architectures.

Understand the differences between test and production environments. How to model tests if the test environment is not rightly sized as production. What if they have different architecture (i.e. clusters/load-balanced sets / single instances)?

Determine what kind of data to use in performance testing. There is a hypothetical scenario with questions at the end of the article. This is an example of a situation that a performance tester needs to be able to handle. Scripting or knowledge of tools will not help with the answers.

Learning about the different systems and what metrics to capture. Also, knowing what those metrics mean and how to interpret them (on their own as well as with others)?

Learning how to capture performance requirements and goals. What if the client has none. How do you go about collecting them? Can you use some kind of heuristics/oracles to help you with that?

Understanding key mathematical principles with focus on basic statistics, especially with respect to sample sizes.

Learning to consider appropriate duration for test runs. What should be good enough sample size?

How many deliberate failures should be incorporated in the test (e.g. failed logins or payments)? Do you know if your code is performance optimized to handle failures?

Learning about different perspectives on ‘Response Times’ that are to be measured (i.e. is the response time determined by particular data appearing on the screen, or the ability to interact with the screen)

Learning about "Three-Eyed Monster" and what matters to it. A term (/metaphor) I coined for the three major stakeholders of performance testing - End User, Business, and IT. As a performance tester, you will need to look through their eyes and understand what matters to them. End user/Customer will not care if the system is consuming 100% CPU. They are more likely to care how quickly the application responds or if you have the product they wish to buy. Whereas IT ops will be interested in knowing about system utilization during your test run.

...

Once I was ready for my first client engagement, I had a senior performance tester alongside to show me the ropes. They were with me for the first few engagements before I was on my own. They were still around if I did need help.

Over time, I have picked new skills such as analysis GC logs, thread & heap dump, and tuning different applications. But mostly as a performance tester, the major focus has never been on performance test tools or scripting.

Now let's fast forward to 2020. In the name of agility and moving to left, I see more emphasis being placed on the use of performance test tools rather than how to conduct performance testing (& what is involved). A trend that anyone who knows how to script can pickup a performance test tool and run a test. As if that is all there is to performance testing in their eyes. I also see minimal to no real training being provided to the person involved in performing the performance testing.

Knowing how to drive a car does not make someone a mechanic, similarly knowing a performance test tool does not mean it is performance testing. It is just the start of the journey. There is more to performance testing than just meets the eye. And it ain't just the performance test tools and scripting.

Take away points from this post are:

Knowing a performance test tool or how to script does not equate to performance testing.

The performance testing world is much more than just a performance test tool.

Have a mentor or group of people who can guide you in the field of performance testing/engineering. Read books/articles/blog posts to help you with your knowledge of performance testing.

Surround yourself with people with different technical & non technical knowledge and learn from them. If you want to learn about databases, find a DBA and learn from them or read articles around that.

Performance testing & engineering is an ever learning field. There is always something new to learn in this field. Whether it is learning how to tune MySQL DB or how to represent information that makes sense.

Here is the scenario mentioned earlier in the post. Test tool knowledge will not help answer the questions below.

Scenario:

Below is one of the graphs & information provided to you by Mr. X. The information is based on existing production data. The business is gearing up for Black Friday/Boxing day. The biggest sales day of the year for the business.

Mr. X wants you to use this data (& other) to conduct performance testing and report back with your findings.

The graph below compares # of Orders vs # of Items/cart from previous big sale and also from Week day & Weekend sales. Couple of comments around the sales:

On special occasions such as Black Friday/Boxing Day, the sales of Cosmetic products is higher than Children's product.

On week days the sale for Cosmetic & Children's product is equally distributed. Similar pattern is also observed over the weekend.

Average cart value on Black Friday/Boxing Day is around $300.