Hack 79. Stress Test Web Applications
You'll never be able to
perform a decent test with one mouse and one keyboard. Use the
Application Center Test tool to simulate legions of users.
|
The focus of this hack is to use ACT from within Visual Studio. In
many cases, it's preferable to use the ACT tool in
its standalone version, so that will be discussed toward the end of
this hack.
|
|
Like all good web developers, we want to
know whether our applications perform well. Does our design and
implementation hold up to the expected load levels? Are we seeing the
right amount of throughput? How many concurrent users can we handle?
How many requests per second can the web application deliver? Will we
see memory leaks after a certain amount of time?
These are all valid questions when it comes to performance testing.
Of course, there are many other questions, but most of them come when
the performance metrics start to get collected and analyzed. In many
cases, performance testing occurs too late in the development
process. This is in large part due to time constraints; performance
testing is put off until toward the end of an application life cycle,
when it's too late to do anything about it.
To do some performance testing on web applications early in the
development cycle, developers can use the Microsoft
Application Center Test (ACT) tool. This tool
comes with Visual Studio Enterprise Edition and is a valuable
resource in any developer's toolbox; it can be used
from within Visual Studio or as a standalone tool. Although not a
replacement for something like Mercury's LoadRunner
suite, it is a powerful asset because it can provide developers a
clear indication of how their web application will perform in the
early stages of an application cycle, thus letting them iron out any
major performance issues long before the application is deployed to
production.
|
If you do not have the Enterprise Edition of Visual Studio, you can
still use a free utility called the Web Application Stress
Tool. This tool, while not as feature rich as ACT, will perform a lot
of the same actions. The Web Application Stress Tool is a predecessor
to ACT, so they have certain similarities as well. You can download
the tool from http://www.microsoft.com/downloads/details.aspx?FamilyID=e2c0585a-062a-439e-a67d-75a89aa36495&DisplayLang=en.
|
|
10.4.1. Stress Testing Configurations
The ACT tool generates load by using a
web browser. In all configurations, the web browser is the client.
And because the client is just a web browser, it can invoke HTTP
calls to any web server, local or remote. This creates a few
different configurations for setting up your stress tests. A typical
web application will consist of several layers that are (possibly)
deployed across multiple tiers. This means that your stress test
configuration will likely fall into one of the three configurations
shown in Figure 10-4.
Here is an explanation of Figure 10-4:
Configuration A shows that the client, the web server, and the
application logic all run on the same machine. This will likely be a
developer's workstation. Configuration B shows the client is now physically separated from the
web server and application logic, both of which are still deployed
together on the same box. This is the case in which the web
application is running on a separate server and the developer is
using her machine as the client. Configuration C shows that the client, web server, and application
logic are all running on separate machines.
Also note that products such as
Virtual PC or VMWare
could be used to emulate any of these configurations on a single
machine. In fact, it's highly suggested you use
these products to create a virtual
distributed environment to match the environment where your web
application will live. Doing so tends to provide more realistic
expectations, basic as they may be, during development.
10.4.2. Identifying Test Cases
Before creating your tests, you must first
identify your test cases. These could range from simple web page
requests to more complex tests that have user workflow with several
steps, tasks, and web pages. The key thing is to identify the start
and end points of the test. This will allow you to determine the
scope of your test and know when to stop recording your test, as
you'll see later.
10.4.3. Create an ACT Project
To use
ACT for performance testing a web
application, you'll first need to create an ASP.NET
application in Visual Studio and run it to make sure it does what you
want. You can then create an ACT Project by following these steps
(assuming you chose to install ACT during the installation of Visual
Studio .NET Enterprise Edition; it's selected by
default):
In the Visual Studio Solution Explorer, right-click the solution,
click Add, and select New Project. In the Add New Project dialog box, under Project Types, expand Other
Projects and click Application Center Test Projects. In the Templates box, select the lone choice of ACT Project. Type a name for the project, select the location, and click OK. You will now see that your ACT Project has been added to your
solution.
10.4.4. Creating a Test
Now
that you've identified
your test cases and created an ACT Project, you can create and add
individual tests to your ACT Project by performing the following
actions:
In the Solution Explorer, right-click your ACT Project, click Add,
and select Add New Item. Under Templates, select Browser Recorded Test. Type a meaningful name for the test and click OK. The Browser Record dialog will appear. Click the Start button. A new instance of Internet Explorer will open. Note that ACT uses IE
even if it is not your default browser. In the browser, type the URL where your web application is running.
Finish "clicking-through" your test
case and then close the browser. In the Browser Record dialog box, notice the Request Details text box
has a list of the requests that were made. Click the Stop button,
then click OK. You will now see that the test script file (e.g.,
TestName.vbs) has been opened up in the editor
and has been added under your ACT Project in the Solution Explorer.
|
One thing to understand before creating your tests: all ACT tests are
either VBScript or JScript, and you have the option of letting ACT
generate the scripts for you (recommended) or
writing the test scripts yourself. If you let ACT autogenerate the
test scripts, they will be created using VBScript.
|
|
10.4.5. Test Properties
Now
that you've created
your ACT Project and added some tests, you need to set the properties
of those tests. The Properties window is where you set your load
levels, test duration, and test iterations. To view these settings,
right-click on a test and select Properties. The Properties window
appears, as shown in Figure 10-5.
By default, a brand-new test is set up to run one browser connection
for five minutes (Connections = 1, RunTime = 300; RunTime is in
seconds). This default configuration is actually a good
"smoke test" to get started.
To gather a wide range of metrics, it's best to
perform tests with varying load based on
duration and iterations. A test based
on duration will tell you how many requests were served in the
specified time. A test based on iterations will tell you how long it
took to run the test the specified number of times.
10.4.5.1 Duration example
To configure a test to run 10 connections for one hour, enter 10 for
Connections, 3,600 for the RunTime, and blank out Iterations.
10.4.5.2 Iterations example
To set up a test to run 1,000 iterations using five connections,
enter 1,000 for Interations, 5 for Connections, and blank out
RunTime.
10.4.6. Running a Test
The
last thing to do at this point is run your test. To do this:
Right-click the test in the Solution Explorer and select Start Test. The Output window now appears and displays statistics for the test as
it is running. A sample is shown in Figure 10-6.
You now have two options: let the test run its course or stop it
manually, which can be done by right-clicking on the test in the
Solution Explorer and selecting Stop Test from the context menu that
appears.
10.4.7. Viewing Test Results
After
your test has finished running, you can
view the results by right-clicking on the test and selecting View
Results. A new window will appear that resembles the one shown in
Figure 10-7.
The test results window provides a quick view of the statistics for
the test run. You can see items such as total run time, total number
of iterations, and average requests per second. ACT autogenerates a
report name using the test name and a timestamp. Every time a test is
performed, ACT generates a report, even if the test was stopped
before it completed. And if you rerun the same test with the same
configuration over and over, ACT will still generate separate reports
for each test run.
To delete a test report, right-click on a report in the test results
window and select Delete.
If you do not like ACT's report naming convention,
you can rename a report by right-clicking on it, selecting Rename,
and typing in a new name. For instance, you might want to name a
report based on the test name and its configuration, such as
Report-GetOrdersDataSet-1000Iterations or
Report-GetOrdersDataSet-10Connections-1Hour.
10.4.8. Customizing Test Scripts
As mentioned earlier, browser
recorded test scripts are autogenerated using VBScript. This has its
advantages, because you can customize and/or modify the generated
VBScript to better suit the specific needs of your tests. In most
cases, the autogenerated script will be all you need, but
it's nice to know that if you need to change it, you
can very easily. You can see the generated VBScript by simply
double-clicking a test in your ACT Project. A sample is shown in
Figure 10-8.
The format of the script is simple and modular. For every single
request in the recorded test, a SendRequest() method is created, each one being
autonumbered starting at 1. It's important to
understand that a SendRequest() method does not
necessarily correlate to a page request. A single web page can
contain several images or references to JavaScript files and CSS
stylesheets. Each of these types of items is an individual request,
thus ACT will generate a SendRequest() method for
each one. Depending on how your web pages are designed, the generated
VBScript could become quite large.
The good thing is everything can be found in a given
SendRequest() method. Each SendRequest() method is a simple subroutine call that must be invoked
by the main program. To see this, look at the bottom of your
generated script. You will see something like this:
Sub Main( )
call SendRequest1( )
call SendRequest2( )
End Sub
The Main() subroutine is nothing more than a
series of calls to the SendRequest() methods. To
demonstrate how you can customize your script, let's
presume your test case is a single web page that contains 10 images,
and all you want to do is test the load time of that page. The ACT
tool will generate 11 SendRequest() methods:
SendRequest1() for the page request and
SendRequest2() through SendRequest11() for each image request. Your Main()
subroutine then looks like this:
Sub Main( )
call SendRequest1( )
call SendRequest2( )
call SendRequest3( )
call SendRequest4( )
call SendRequest5( )
call SendRequest6( )
call SendRequest7( )
call SendRequest8( )
call SendRequest9( )
call SendRequest10( )
call SendRequest11( )
End Sub
Simple enough, so you run your performance tests, analyze the data,
and realize the page is not loading within acceptable limits. You can
begin to streamline the performance by simply commenting out the
calls to some of the SendRequest() methods for
the images. Then rerun your tests and analyze the results. This type
of easy modification of the script allows you to make quick changes
to your test scripts without having to redesign your web page and
record a new test.
10.4.9. ACT Project Files
You've
seen how to create an ACT Project in Visual
Studio, add and configure test scripts, run tests, and view the
results. Another cool thing is that ACT creates and manages all its
project files in XML format. You can see a sample of these files in
Figure 10-9.
With the exception of the .vbs file, every file
shown in Figure 10-9 is an XML file, even the
.act and .actproj files. As
you configure and make changes to your ACT Project and tests, the ACT
tool is modifying these files behind the scenes as needed. A quick
look at the testProperties-GetOrdersDataSet.xml
file shows the following XML:
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE Settings>
<DefaultValues >
<TestProperties >
<ControllerReportTimeout type="long" value="5"/>
<Timeout type="long" value="120"/>
<Duration type="long" value="300"/>
<Warmup type="long" value="0"/>
<UseIterations type="bool" value="False"/>
<Iterations type="long" value="200"/>
<UseRateControl type="bool" value="False"/>
<TargetRPS type="long" value="40"/>
<PerfCollectionInterval type="long" value="10"/>
<TestType type="long" value="1"/>
<GenerateUsers type="bool" value="True"/>
<UsersToGenerate type="long" value="5000"/>
<CollectPerfCounters type="bool" value="False"/>
<CollectRuntimeHistory type="bool" value="true"/>
<Locale type="long" value="1033"/>
<CollectPerPageData type="bool" value="true"/>
</TestProperties>
<DynamicTest >
<Language type="string" value="VBScript"/>
<NumberOfThreads type="long" value="5"/>
<FollowRedirects type="bool" value="False"/>
<RedirectDepth type="long" value="15"/>
</DynamicTest>
<Network >
<Enable type="bool" value="False"/>
<NATRange type="long" value="0"/>
<Distribution.9.6K type="long" value="1"/>
<Distribution.14.4K type="long" value="4"/>
<Distribution.28.8K type="long" value="5"/>
<Distribution.56K type="long" value="20"/>
<Distribution.128K type="long" value="15"/>
<Distribution.512K type="long" value="15"/>
<Distribution.T1 type="long" value="15"/>
<Distribution.Unlimited type="long" value="15"/>
<PropagationDelay type="long" value="200"/>
<NATStartAddr type="string" value="10.10.1.1"/>
<NATStopAddr type="string" value="10.10.1.255"/>
<ClientPortStart type="long" value="1000"/>
<ClientPortEnd type="long" value="5000"/>
</Network>
<StaticTest >
<DefaultServer type="string" value="localhost"/>
<DefaultMethod type="string" value="GET"/>
<DefaultHTTPVer type="string" value="HTTP/1.1"/>
<DefaultPort type="long" value="80"/>
<SessionCount type="long" value="5"/>
<UseRandomDelay type="bool" value="True"/>
<MinDelay type="long" value="10"/>
<MaxDelay type="long" value="1500"/>
<FollowRedirects type="bool" value="True"/>
<RedirectDepth type="long" value="15"/>
</StaticTest>
</DefaultValues>
10.4.10. ACT: The Standalone Tool
Up
to this point, you've
seen how to use ACT from within Visual Studio. However, you cannot
get to all of ACT's functionality from within Visual
Studio itself. Other features available in the standalone tool are:
To see all that ACT has to offer, you must use the tool on its own.
This section will highlight ACT in that way.
10.4.10.1 Create an ACT Project
To use ACT for performance testing,
you'll first need to create and deploy your web
application in one of the configurations mentioned earlier. Once
you've done that, you can create an ACT Project by
following these steps:
Click Start All Programs Visual Studio .NET
2003 Visual Studio .NET Enterprise Features and select
Microsoft Application Center Test. Click File New Project (Ctrl-N). In the New Project dialog box, type a name for the project and select
a location (by default, the location is in the C:\Documents
and Settings\<username>\My Documents\ACT Projects
directory). Then click OK. You can save the project any time by clicking File Save
Project (Ctrl-S).
10.4.10.2 Creating new tests
You can create and add individual tests to your
ACT Project by performing the following actions:
In the left pane, right-click Tests and select New Test. A wizard
will start; just follow the steps. When prompted for a test source, select Record New Test. When prompted for browser record, click the Start Recording button.
This will open a new instance of Internet Explorer (note that ACT
uses IE even if it is not the default browser). In the browser, type the URL where your web application is running.
Finish "clicking-through" your test
case and then close the browser. In the wizard, click the Stop Recording button and then click Next. Give the test a name and click Next. Then click Finish. You'll now see the test you just recorded listed in
the right pane.
10.4.10.3 A sample ACT Project
To demonstrate the ACT tool,
I've created a sample web application named
Northwind that contains a single web page,
GetOrders.aspx. This page is used to retrieve
all the orders from the Northwind database on my local SQL Server. In
ACT, I've created a project named NorthwindStress
and recorded a test named GetOrdersDataSet, as shown in Figure 10-10.
The GetOrdersDataSet test is quite simple: open the web page, click
the Get Orders button, and display the results. Figure 10-11 shows the page when you first open it, and
Figure 10-12 shows the page with the results.
In fact, these two views are what was recorded when I created the
GetOrdersDataSet test in ACT. The end of the test case is the results
page.
10.4.10.4 Test properties
Each ACT test has a set of properties
where you set the load levels, test duration, test iterations, users,
and performance counters to capture. To view these settings,
right-click on a test and select Properties. The Properties window
appears and displays the General tab, as shown in Figure 10-13.
The General tab is where you set the
load level and duration. By default, a
brand-new test is set up to run one browser connection for five
minutes. This default configuration is actually a good
"smoke test" to get started. The
General tab is where you'll spend most of your time
when setting the properties of a test.
The Advanced button simply displays a checkbox to generate detailed
test results, which is on by default. It's best to
always leave this box checked.
The next tab is the Users tab, as shown in Figure 10-14.
For tests that do not need to store the HTTP cookies created and used
during the test run,
users can be automatically generated by
ACT during the test on an as-needed basis. Automatically generating
users will help to avoid potential problems that are caused when the
test's population of users is too small to match the
number required to create the specified load level for the test run.
For tests in which authentication is required or you want
to view or reuse the cookies that were created, users and user groups
must be created and then selected in the Users tab. You will be able
to set the name, domain, and password for each of these users.
The last tab of the Properties window is the Counters tab. This is
where you add any performance counters you wish to
capture, which could be for the test client and/or the web server.
For this sample, I've added a few performance
counters and changed the collection interval to two seconds, as shown
in Figure 10-15.
You can add performance counters in this tab by clicking the Add
button. A dialog then appears for you to make your selections, shown
in Figure 10-16. If you've used the
Performance Monitor tool at all, this dialog will look very familiar.
As you can see, using the ACT tool in standalone mode offers many
more test settings than when using ACT from within Visual Studio.
10.4.10.5 Defining users
Another advantage of using ACT outside
of Visual Studio is the ability to define users and groups of users
for your tests. Provided for you is a group of users named Default
Users Group, containing 200 users (User0-User199) with generic
passwords, as shown in Figure 10-17.
To create your own groups of users, you have three options:
type them in by hand, let ACT generate them for you, or import them
from a comma-delimited list. Also note that all users must belong to
a group.
First, to create your own users group, click on Users in the left
pane, right-click and select Add. A group with the name of New Users
Group will be added. If you'd like to rename it to
something more meaningful, right-click, select Rename, and type in
the name.
To manually add your own users, select the users group to place them
in, scroll to the bottom of that group's list of
users, and enter the username and password. Figure 10-18 shows an example.
|
To add users in a domain, use the
DOMAIN\Username format.
|
|
To let ACT generate a set of users for you, follow these steps:
Select the target users group. Click Actions Generate
Users. The Generate Users dialog appears. Enter the number of users, the
name prefix, and password (see Figure 10-19). Then
click OK. The list of generated users will now appear in the right pane.
The only thing of note with generating users is the start number. ACT
always generates users beginning with the number 1 (as you can see,
the ability to change the start number is disabled). In the sample
shown in the figure, the users created for me were
DOMAIN\NorthwindUser1 through
DOMAIN\NorthwindUser5.
The last way of creating users in ACT is by importing them from a
comma-delimited file; ACT will not import users if the file is in
other formats. The file should contain usernames and corresponding
passwords. To import users from a file, do the following:
Select the target users group. Click Actions Import Users. Browse for your file, select it, and click Open. The list of imported users will now appear in the right pane.
This shows the contents of a sample file (Figure 10-20 shows the list of users after being imported
from the file):
DOMAIN\NorthwindUserA,myPasswordA
DOMAIN\NorthwindUserB,myPasswordB
DOMAIN\NorthwindUserC,myPasswordC
DOMAIN\NorthwindUserD,myPasswordD
DOMAIN\NorthwindUserE,myPasswordE
10.4.10.6 Running a test
Running a test in ACT outside of Visual Studio
is much the same as running it within Visual Studio�right-click
the test to run and select Start Test from the context menu that
appears. Both methods display the statistics as the test is running,
but using ACT in standalone mode allows you to see a nice graphic
instead of a line of text. The graphic is shown in Figure 10-21.
10.4.10.7 Viewing the results
Once your tests have completed, you can
check out the results by clicking Results in the left pane and then
choosing the appropriate test report under Test Runs (top pane,
left). A snapshot is shown in Figure 10-22, with more
statistics shown in Figure 10-23.
Recall that I added a few performance counters to monitor for
the GetOrdersDataSet test. You can view the statistics for those
performance counters by selecting Performance Counters under Report
(top pane, right). Figure 10-24 shows the performance
counter statistics for the GetOrdersDataSet sample.
And that should about do it. You've seen how you can
easily create and manage performance tests for your web applications
using ACT, from both inside and outside the Visual Studio IDE. The
primary reason for using ACT within Visual Studio is for the tight
integration with your ASP.NET Projects, but you
don't get ACT in its full capacity. The main reason
for using ACT outside of Visual Studio is the better user experience,
plus you can take advantage of everything ACT has to offer, such as
graphical results, user groups, and performance counter monitoring.
�Dave Donaldson
| 
No comments:
Post a Comment