Here at Trust IV, we’re always keen to practice our performance testing skills against different application platforms. Historically we’ve been called upon to test banking, retail, CRM and even video-streaming applications; consequently, when we practice our skills, train new users or demonstrate our testing capabilities to customers, we like to use a realistic application. Since more than a third of the world’s E-Commerce websites run on Magento Software we thought that it would be useful to add Magento testing to our repertoire.
Our test approach
Our approach to this testing was straightforward. We took the most common online eCommerce platform (Magento Community Edition) which has 33.8% market share (Source: AheadWorks blog) and installed it on a Hyper-V server in our data centre in Trafford Park. We wanted to investigate the CPU and memory requirements for Magento because we were using this platform to train users in the use of LoadRunner for performance testing.using HPE LoadRunner 12.0 which simulated a user browsing the website and purchasing an item from the online store.Each test was configured to ramp up to a load of 40 virtual users over a five-minute period. We then maintained a steady load for thirty minutes, during which time approximately 270 purchases were made simulating a load of 9 products purchased per minute.LoadRunner infrastructure (Windows Performance counters) as well as the target server (via UNIX RSTAT).We produced reports on response times and infrastructure performance after conducting the same standard test three times:
- Same test data
- Same volume of users for each test
- Same workload per user
- Each test simulated approximately 2,500 page request in 30 minutes.
We found that the tests were remarkably consistent.
The next phase of testing - experimentation
Once we had established that our test application and our load generation capabilities were providing consistent results, we decided to do some experimentation. Often people who are developing applications and designing an application infrastructure want to know whether their application is “CPU hungry” or “Memory hungry”. This can help them to decide whether to scale up (to larger servers with more memory) or scale-out (to a larger number of individual load-balanced servers). Due to the fact that we were using virtual servers to support this application, it was simple for us to reconfigure the servers and run multiple tests to determine the optimum configuration.We decided to run a series of tests with different CPU and memory configurations. The following configurations were chosen for comparison:
- Single CPU core, 4GB RAM (1C 4G)
- Single CPU core, 8GB RAM (1C 8G)
- Dual CPU cores, 4GB RAM (2C 4G)
- Dual CPU cores, 8GB RAM (2C 8G)
The Magento application was hosted on a single Ubuntu (64-bit) server. The same server was configured with Apache and MySQL. The only difference between each test was the resources made available to the virtual server from the underlying Hyper-V host.Because we were testing over the Internet and using a virtualised application infrastructure we needed to be sure that applications beyond our control were not having an impact on our test results. Each test was repeated three times to prove consistency.
Average results were produced to make this report easier to read online. In each test, we measured a number of key performance counters including the average response time for the application home page, the number of hits per second and key performance counters such as CPU and memory utilisation for the Magento server.
- Simply adding memory to a Magento application server, may not improve performance. Based on our simple test the application seems to be more “CPU-hungry” than “memory-hungry”.
- For “out of the box” Magento configurations, increasing the amount of resources available to an application may not always improve performance.
- As well as providing the appropriate application infrastructure, it is essential that performance testing and tuning is carried out to obtain optimum performance from a new application.
- Obviously testing four different ‘tin’ configurations isn’t an all-encompassing comparison. Further testing can and should be considered for anyone looking at rolling out Magento for a live e-Commerce web site, such as.
o Apache configurationo PHP configurationo Database configurationo Multi-tier application infrastructureso Load Balanced, multi-server configurations