Here at Trust IV, we’re always keen to practice our performance testing skills against different application platforms. Historically we’ve been called upon to test banking, retail, CRM and even video-streaming applications; consequently, when we practice our skills, train new users or demonstrate our testing capabilities to customers, we like to use a realistic application.
Since more than a third of the world’s E-Commerce websites run on Magento Software we thought that it would be useful to add Magento testing to our repertoire.
Our test approach
Our approach to this testing was straightforward. We took the most common online eCommerce platform (Magento Community Edition) which has 33.8% market share (Source: AheadWorks blog) and installed it on a Hyper-V server in our data centre in Trafford Park. We wanted to investigate the CPU and memory requirements for Magento because we were using this platform to train users in the use of LoadRunner for performance testing.
We chose a virtualised server platform so that we could; quickly and easily alter the system resources allocated to the server; and perform server snapshots and data roll-backs in between tests to ensure results consistency.
The application was configured with Magento sample data, and apart from tweaking the test data to ensure that our virtual "customers" didn’t run out of products to buy, the application was configured “out of the box”.
We recorded a test script using HPE LoadRunner 12.0 which simulated a user browsing the website and purchasing an item from the online store.
Each test was configured to ramp up to a load of 40 virtual users over a five-minute period. We then maintained a steady load for thirty minutes, during which time approximately 270 purchases were made simulating a load of 9 products purchased per minute.
We used standard performance monitoring techniques to monitor both the performance of our LoadRunner infrastructure (Windows Performance counters) as well as the target server (via UNIX RSTAT).
We produced reports on response times and infrastructure performance after conducting the same standard test three times:
- Same test data
- Same volume of users for each test
- Same workload per user
- Each test simulated approximately 2,500 page request in 30 minutes.
We found that the tests were remarkably consistent.
The next phase of testing - experimentation
Once we had established that our test application and our load generation capabilities were providing consistent results, we decided to do some experimentation. Often people who are developing applications and designing an application infrastructure want to know whether their application is “CPU hungry” or “Memory hungry”. This can help them to decide whether to scale up (to larger servers with more memory) or scale-out (to a larger number of individual load-balanced servers). Due to the fact that we were using virtual servers to support this application, it was simple for us to reconfigure the servers and run multiple tests to determine the optimum configuration.
We decided to run a series of tests with different CPU and memory configurations. The following configurations were chosen for comparison:
- Single CPU core, 4GB RAM (1C 4G)
- Single CPU core, 8GB RAM (1C 8G)
- Dual CPU cores, 4GB RAM (2C 4G)
- Dual CPU cores, 8GB RAM (2C 8G)
The Magento application was hosted on a single Ubuntu (64-bit) server. The same server was configured with Apache and MySQL. The only difference between each test was the resources made available to the virtual server from the underlying Hyper-V host.
Because we were testing over the Internet and using a virtualised application infrastructure we needed to be sure that applications beyond our control were not having an impact on our test results. Each test was repeated three times to prove consistency.
Average results were produced to make this report easier to read online. In each test, we measured a number of key performance counters including the average response time for the application home page, the number of hits per second and key performance counters such as CPU and memory utilisation for the Magento server.
We found these performance test results particularly interesting.
Both sets of tests where the Magento server was configured with only a single CPU core (1C 4G and 1C 8G) demonstrated poor application performance. Compared to the later tests where two CPU cores were configured, response times were significantly slower. The number of hits per second showed that the application handled less traffic, due to the fact that the server was “CPU-bound”. Under load, CPU utilisation reached 100 percent and processor queuing was taking place.
The average response time for the application home page, which is a key indicator of application performance as well as being crucial for a good user experience, was the lowest in the third set of tests (2C 4G) where the Magento server was configured with two CPU cores and 4 GB of RAM.
In the fourth set of tests (2C 8G) where the Magento server was configured with more memory the performance appeared to deteriorate as more memory was added to the server. Although the total number of passed transactions was similar in the “two-core” (2C 4G and 2C 8G tests), the average homepage response time was slower and CPU utilisation was higher. This indicates that the additional memory didn’t provide a significant performance improvement. In fact, it appeared to cause an increase in CPU utilisation, possibly due to the management of this additional, unused resource.
- Simply adding memory to a Magento application server, may not improve performance. Based on our simple test the application seems to be more “CPU-hungry” than “memory-hungry”.
- For “out of the box” Magento configurations, increasing the amount of resources available to an application may not always improve performance.
- As well as providing the appropriate application infrastructure, it is essential that performance testing and tuning is carried out to obtain optimum performance from a new application.
- Obviously testing four different ‘tin’ configurations isn’t an all-encompassing comparison. Further testing can and should be considered for anyone looking at rolling out Magento for a live e-Commerce web site, such as.
o Apache configuration
o PHP configuration
o Database configuration
o Multi-tier application infrastructures
o Load Balanced, multi-server configurations