Performance Testing...

Well, I've left my old job being a Performance Engineer, and into a new job with some Perf Eng responsibilities. I've used Loadrunner to do my performance testing and some of the terminology may be unique to Loadrunner.

Well, this got me reflecting, and thinking how best to implement a process here so that we are able to performance test well and rapidly implement the whole process from scratch.

Here's what I came up with.
0. Figure out WHY you are testing: Response time? Capacity Planning? Figure out WHAT you want to report (Current Performance?, Performance after changes?, Available headroom on hardware?) Now you can plan....

1. First off, identify what you want to test: Sure you know that you want to test your critical use cases, but what about after that? Sit down with some users and figure out what they use the most. Or you can use a really cool tool like Coradiant to do what you need. Using a tool like Coradiant will give you the most accessed pages, and you can get user sessions to see the most used paths. Guessing will get you only so far, but if you want to regularly test your application so that your users are always happy - tools like Coradiant are a very handy.

Test anything new thats going to be added to the application. Most Web applications are always in a constant state of development, and anything new you add, should also be tested, lest you end up with a bunch of very unhappy users.

2. Keep your tests small and repeatable: Testing 100 things at the same time will overload you with data and nothing more. You also will perhaps end up producing conditions that will never occur. That doesn't mean you should test every use case individually (which in my opinion may give you some data, but will probably never catch race conditions, deadlocks etc), it means keep a core number of tests.

When you add something new to the mix, be prepared for changes in numbers. If you are looking for exact same numbers, then you should be doing the exact same tests.

3. Length of Tests: Depending upon the application, you need to define the length of your tests. Too long tests mean that you'll have to wait a long time for results. Really short tests don't give you any reasonable data, because the users have not yet reached a stable state. I used to do 1 hr tests, but now I think that was over kill. 10 Min ramp up (100 VUsers) and 15 min tests would have given us just about the same amount of information as an hr long test.

4. Don't over complicate your testing: Remember your audience. Remember, that you want to prove / test that / whether the application will perform well under load. There are 2 different types of load:
1. Large data set
2. Large number of users on the system.

For regular testing, you need to find some sort of mid point and test. If you have written your performance tests right, you should be able to simulate 2 very easily. For 1, you will have to rely on your developers to provide an adequate data set.

Most important of all, always remember, you need to be providing useful information back to the team. Doing the same test 20 different ways will not give you useful results. Unless you define what & why you're testing, you're perhaps wasting your time.