One of our clients, a betting company, had a strong need to use 5000 real users to perform web application performance testing, which means that each virtual user must perform their activity in a real browser. Sadly, popular reliability measures such as Jmeter, NeoLoader, etc. will not do the job in this situation.
Our main objective was to install the most important behavior that will be extensively used by real users. For example, users should do their tasks in a step-by-step manner: all users must sign in and wait for 5 minutes. Once all users are logged in successfully, bulk actions should be performed in the same period. We faced a lot of real-time update issues for bets during the testing. This prevented us from running performance tests, and to make our experiments effective we had to incorporate complex reasoning with various conditions.
When we performed our development test runs (this was our client’s main requirement), we had to add additional timeouts to mitigate security from DDOS assaults.
Another problem was performing all the experiments from one computer. As a consequence, both applications were in the queue, and the quality checking was not what we wanted to do.
Our customers were unable to prevent the introduction of new features, and as a result, they were patched once a week for UI or backend, which forced our application processes to be modified from time to time. Our output was slowed down by such practices.
We plan to use the Selenide automation system to create user behavior and to run all these huge numbers of tests with Zalenium. Zalenium is mainly a chain of selenium, but it is in Kubernetes. Kubernetes were picked for the potential to orchestrate and scale up selenium nodes rapidly. We needed a huge number of resources to create this volume of usage, and fortunately we were able to provide it to us by AWS. Our cluster of Kubernetes featured 200 c5d.18xlarge nodes with 72 cores and 144 GB of RAM each. We created a test suite that could produce test data based on the number of browsers in the cluster of Kubernetes. Test farm has been built to run our experiments from 7 machines in parallel. The frontend, backend, and GraphQl database analysis found big bottlenecks. The server required 5000 + users to sustain load, but the actual number of users without interruption was only 717 after the first test run.
1. Automation Testing
In addition to the 300 test cases from the customer’s in-house testing team, our engineers built more than 1000 test cases.
2. Performance Testing
3 Full-time automation QA Engineers have been involved in the project.
3. Web Testing
For each sprint, our QA Manager was active in the test cases formation phase.