Telerik Test Studio Webinar: Performance and Load Testing with Test Studio Video Lecture | Telerik TestStudio:Automated Testing made Easy - Software Testing

the broadcast is now starting
all attendees are in listen-only mode
good afternoon thanks for joining
today's test Studio webinar this is Andy
Weiland I'll be conducting the
demonstration today all collars are
muted throughout the call so please feel
free to be as loud as you'd like and ask
any questions along the way in the chat
or questions panel we'll try to keep
track of those but you should be able to
see my screen at this point and hear my
voice any problems you might have please
feel free to chat or send something
across on the questions panel today's
focus is going to be around test Studios
performance and load testing we also
have our mobile testing which will
happen if this teams the same time slot
every other week so next week will be
mobile testing in the same timeslot and
then on Tuesdays we always have our
functional automation testing tuned in
to those if you'd like to see the other
areas of test studio but today's focus
will be on performance and load testing
and really where we start with
performance and load testing is where we
leave off with functional testing so
what would I have up here on the screen
is a basic functional test this is just
a simple login test at this point and to
begin with performance testing we need
to start with a functional test like
this you don't have to but I'm going to
go ahead and execute it just because I
haven't run this test yet today you're
just going to execute it real quick to
make sure everything's working correctly
and then we'll run it as a performance
test as well so we'll just pick some
pickup browser here to execute it in and
of course that's going to open on the
wrong monitor
so real quick and easy there it goes
through the execution of the functional
test everything's passing so it looks
like we're good to go so to run this as
a performance test all you need to do is
open up one of your functional tests
here as I have in this tab and then
click the performance tab if there are
any performance results if this test has
been run as a performance test in the
past it will load it will gather and
load those results for you if it hasn't
been run as a performance test before it
will prompt you to configure the
performance test settings which I'll
show you here momentarily up here you
can always go back into the
configuration or if it's the first time
you run it as a performance test this
will pop up for you you're gonna be able
to specify where you'd like those
results to be stored and then here on
the second page we can configure if we'd
like to do performance monitoring our
performance monitoring can take place
from the machine that we're executing
the test from which is this machine or
particularly we might want to actually
monitor performance of other machines -
that might be the Database Machine
that's hosting the database of the
application or the application server
the host of the application itself so we
want to be able to read into those
machines and understand how they're
handling the situation you'll see that
the same thing as applies to per to load
testing as well but to kind of give you
a perspective of how we look at
performance testing versus load testing
performance testing is really what we
call the micro vantage point it's going
to give us very detailed data it's going
to be one user running through the
scenario but giving us very detailed
data about every step of that particular
scenario that's being executed in this
case a login scenario now if we'd like
to we can monitor those metrics and
performance counters from those machines
you just click add computer in any of
our tests real licenses include a
profiler so you may have multiple
machines that are involved in executing
you may have as I mentioned the database
server or host app server and they can
be connected together
a test lab such as this and any and all
of these including these Ultimates and
functional licenses over here these
include the profiler as well as our
runtime licenses these are your
execution agents they also have the
monitoring of performance capability so
here in the case of the runtime number
six I'm actually not using it to
generate any load or to execute tests on
this application server directly instead
I'm only monitoring I'm only using the
profiler
capability in this case as you can see
below there there are some ports
involved with making sure that
communication is open and available but
when it is you basically can type in the
Machine name click connect and you
should get a list of available counters
that work for that particular machine
now this list of counters will be
different depending on what that machine
has as you can see there's some hyper-v
on this box I can monitor some stats
from that if it's a database server it
might be like sequel or Oracle
statistics things like that that we
might want to monitor and there's some
common things that we can start with too
to give you some assistance here let me
pull up a bit of our documentation and
you can get to this online user guide
directly from the help tab in tests
studio as well but in here we actually
have an article that's really useful
that it's called profiling your app and
this will go through how to really
understand the performance metrics and
monitoring the performance and of course
requirements involved with that and
there's some roll specific metrics here
to help you as well we call this a
starting point this is not the end-all
be-all for all applications are all
environments but it is a good place to
kind of start to get an idea of what you
might monitor if you're monitoring the
App server that's hosting the
application or the database server
that's hosting the database of the
application or if you're going to want
to monitor the performance of the
execution agent and again going back to
our visual here this is representing our
app server I don't have the database
server represented here but just imagine
there'd be another one over here that
might have a runtime on it as well and
then one through five runtimes rx
Dushan agents these are all tied
together with a scheduling server so
number one here is actually acting as
the scheduling server the storage server
as well as an execution agent and as
well as performance monitor so these are
very multi-purpose very cheap licenses
that we provide so that you can build
out your lab of machines and get all the
monitoring and all the automation out of
it that you need to so we tap into any
of those the profiler gives us access to
the statistics or to the performance
metrics and I'm going to go ahead and
drop the link to this document into the
chat window for you to to take with you
and then back over into the tests itself
what we're doing here is we're actually
giving we're checking check boxes to
build a template and as you can see I've
already got memory all checked here
there's all sorts of check boxes within
check boxes within check boxes every one
of these has just a myriad of different
things that you can look at and study
again the documentation will help you I
think give you some guidelines as to
where to start and you may also want to
work with your development team on maybe
other areas that you want to monitor as
well but once you've checked the check
boxes and created the appropriate
template of what you've got I've got
memory and processor and physical disk
in here server as well so lots of
different things that we can monitor
once we've got that that can actually be
saved as a template to use in the future
as you can see up here there is a Save
button and there's also a file button to
load a pre saved template so I have a
database template and an app server
template here if we were to select those
it automatically makes all the
selections for us so it's really
shortcuts the process once we're done
choosing our check boxes and what we
want to monitor and setting up our
machines we'll click done editing again
we can repeat this for multiple machines
that might be involved in the test and
once we have that we'll go ahead and
click Save so those are the
configuration options for setting up
your functional tests to run as a
performance test now this particular
test has been run as a performance
in the past so there's some history to
it you can see here a trend line at the
bottom giving us a view of how it's
trending as well as the individual
results that were derived from execution
of this as a performance test the
overall test time as well we see that
one years is quite large compared to the
others now I want to point out that lots
of times we'll have environments that
are separated from the public right the
qat environments the staging
environments and in those cases you may
not have any load on those environments
so doing a performance test gives you
maybe some inadequate information you
definitely want to run performance tests
in a no load environment to get a
baseline but then also run your
performance tests in a peak load
environment and then average load
environment so you get a good feeling
for where that trend is where it should
be with regards to the load involved now
if you do have an outlier like this you
could delete that from the result data
if you'd like as well we'll go ahead and
run this we'll run this in Firefox
because I only have one of those we'll
just click on Firefox here and execute
that this will now execute the same test
the one we saw before it will look
exactly the same - it may take a little
longer in this set up and in the
teardown of the tests since it is
turning on and opening up the ports and
communicating with the machines and at
the end it's capturing all of the data
that's being collected and bringing it
back but as far as the execution of the
test itself it should not be affected in
any way once it does kick off here it
should run just like normal now in this
type of test it's not concerned with
functional right it will let us know if
there's a step that passes or a step
that fails but it's not focused on
failure here like it would be in a
functional test and a functional test if
there's a failure will diagnose a
failure we'll get lots of failure
details and we'll go in triage add in
and fix that issue in performance
testing it's not about failure we're
gonna take the test as it is and provide
as much statistical data back as
possible but if there was a failure you
could you could then go back to the
functional version of that test and
correct the failure there
so here's our result we see everything
did pass we got green checks all the way
down but again if it failed it wouldn't
really matter it would just give me an
indicator for us to head back over and
maybe troubleshoot the functional
aspects of the test but this is the data
were provided in the overview of a
single run a single execution of this
test that's now looking at more of a
step level so we could see the server
timing and client-side timing and the
server the total time as well as well
size and any errors that may have
occurred too so this really gives us a
first look at out of these steps which
one is really maybe least efficient
efficient or costing us the most time
now keep in mind I do have a wait step
in here for two and a half seconds so
obviously that's a lot of time just
sitting on the client-side and a little
bit more than that looks like it was 2.6
total as a transition between steps and
that gives us a you know view of where
things stand out so so here we're really
looking at from a performance test
standpoint we're trying to locate the
bottlenecks in a particular application
and in a particular scenario or user
story within that application in this
case it's logging in now outside of my
wait here everything else looks pretty
clean right we do see here on the step 5
there's lots of time spent on the client
very little on the server in that
particular case but it's a nice pretty
easy split and these are pretty low
numbers at this point now if I run this
in a peak load or average load
environment I might expect these to be
increased and I might expect certain
things certain issues to be exacerbated
and that's the whole idea of performance
testing it's really to feel out where
there may be bottlenecks
and that points to where you may be able
to focus some development efforts in
order to make things more efficient now
the real I think
detailed information is here on the
details tab you can get to that by
clicking a step or by clicking the
details tab itself and this is really
that micro vantage point that we talked
about from a performance test so it was
one user in this case going through a
login process and here we're seeing
each step broken down to the method
level and the order of operations and if
we float over these little nodes some of
them are super super tiny because
they're just very my new amounts of time
that are spent but it gives us the the
waiting the sending the receiving all
all these different time events that
occur that make up that particular
method and we can correlate that to the
particular method in question over here
and really that's where we can look at
to focus our attention so this
particular JavaScript event here is one
that took the most amount of time now
it's pretty much pretty small amount of
time and the total of things but again
this is something that if we'd see in a
peak load environment we might expect us
to take two and a half seconds just for
this one method versus this quarter
second that we're that we're looking at
here we also see status codes that come
back and of course any errors to that
might have occurred along the way too
okay so this is just looking inside of
one step we can click through and see
step two here and again a lot of these
are very small we can change actually
the the scope up here to kind of
increase the size of these but we're
just going to click through a few of
them real quickly and here you can kind
of just get a feel for where you may be
able to look at some improvements and
where those bottlenecks might be
occurring in those particular steps okay
now on top of this literally we can
overlay all of those performance
counters that we've enabled so if I
expand my machine list over here here
are all those perf counters we could
take a look at you know let's say page
rights per second or reads per second
we'll get down into processor and look
at like % processor time and we can look
at where the spikes occur and we can see
maybe how things line up right we'll get
down here to further and - let's see
we've got server sessions forced out off
or or air it out for example might be
something new
sessions timed out from the server this
would be if I'm monitoring the app
server itself and further on down here
there are tons and tons of these and you
can certainly turn them all on just to
show you how beautiful that is you could
certainly turn them all on it is better
to be a little bit more pinpoint and
precise and of course this is something
that you can go through and and pick
away at and kind of see which of those
particular performance counters are most
important for you but this really allows
us to you know again an overlay the
important data and really kind of scroll
through all these methods and see you
know what was it that led up to this
spike occurring alright and of course we
can overlay any of those performance
counters from the other machines that we
might have been monitoring for this as
well so we can have multiple machine
performance counter graphing data over
the top of this method level data and
the timing and everything else it's a
great amount of information to really
let you get that introspective on where
you can get the most efficiency gain or
where the biggest problems are in your
particular application where the the
biggest bottlenecks might be in your
application so that's really performance
testing in a nutshell now that was just
the history of the overview in the
details you can also compare these runs
to each other and here we could see two
of these compared to each other in the
last time it looks like I ran this was
free Christmas and it was as you can see
here the B run versus the a run which is
the one that we just ran so we can
really take a look and see where the
differences are side by side step by
step and we can also see these red X's
when we have some Delta's that occur now
these deltas are tiny right a lot of
these are within an allowable threshold
of time and that could be modified up
here so that it can really eliminate the
noise so we'll add a three-second total
time we'll go with two seconds on the
server or one second on the client and I
look back down and all
those red X's are now green they're
within the threshold and we'll give it a
data size variation too and now we've
got all green checkmarks except for one
red X down here on this step six log in
button step where it was oh sorry that's
the yep that's step six and basically
the difference in size there that
occurred just in today's run versus the
last run from here we can go back into
that run we can go back into step six
specifically we can overlay performance
counters looks like this one also has
some from another machine to in here and
we can basically look for you know what
was it that they cost us that lag in
time you also can see we had some 304 s
in here it's a little bit different
status on that return looks like the
rest of those are two hundreds so
everything looks pretty good so that
comparison really kind of helps you
evaluate especially when you want to
compare a baseline no-load environment
to a an average load or peak load you
know put those side-by-side and really
look at how big those deltas are and if
they fall outside of the threshold or
not and that's really all there is to it
so we're taking a test we've already
built a same test that we used for
functional regression testing we've run
that same test now as a performance test
now we've got a lot of method level
micro vantage point type of data back so
that we can identify bottlenecks now
performance tests can be run in test
lists as well when you create a test
list you'll give it a name and you'll
assign it a type a default is just the
functional automated kind of tests so
we'll go ahead and give this a name and
we'll grab some tests in here we're just
basically grabbing any of these tests
that we've that we want to include to
build out our test list and if we were
to switch this from automated to
performance you'll see that it gives us
some configure options here now if this
tests has previously been configured
it's the same wizard we saw from the
performance tab but if this has already
been configured once before you don't
have to go back through that again
you certainly can from here but it will
it will carry over any of the settings
that you previously had it looks like
it's trying to load some stuff for me
here so give it a minute but see here we
have all of those perf counters already
selected we don't need to modify this
unless we really want to change
something around so if you've previously
used it as a performance test that
configure option isn't necessary unless
you want to change something if it's the
first time you've run it as a
performance test you can do that from
here or you can go back to the
performance to have it do that for each
individual test as well so this gives us
a performance test list okay and that
can then be scheduled to run or run
remotely across multiple machines of
course that opens up so many
possibilities and even running it from a
CLI or a build server process that might
call that performance test list as part
of a build build process or continuous
integration process okay so that's
really the again the nutshell of
performance testing again not too much
to it just because we do a lot of the
work and just creating a functional test
and it's that reuse that tests studio
really provides it allows you to just
get even more mileage out of the same
test so the same test is telling us if
it's functionally working correctly and
now where the bottlenecks are in the
process so moving on we're gonna jump
into load testing now load testing is a
different type of test if we were to in
our project say add new tests we'll get
the choice of the web test the load test
and then of course WPF in manual again
for performance tests it is a web test
just run from the performance tab for a
load test it is a distinct type of test
and test studio but again you do have
the concept of reuse we like to reuse
things as much as we can in test studio
because it cuts down
your maintenance time and you get more
reliability more more mileage out of
your tests as well so from a low testing
perspective let me get back here to the
design tab
the load test has a number of different
components to it and configuration
options to it as well but primarily what
we want to do is capture traffic that's
generated by certain user scenarios
they're also called user profiles and
essentially build together a
conglomerate of what the typical traffic
would look like so let's take for
instance a Monday morning when
everybody's logging on at the same time
we may have a system that just struggles
a bit with log on with that initial
Monday morning burst of everybody
logging on for instance you know that
that might be one scenario and then
there may be a small other scenarios
where people are updating records or
running reports things like that yeah
you really want to just have a
combination of traffic of different
scenarios being conducted by different
users just like you would you know if
you think about a shopping cart
application like an Amazon you've got
some people that are just browsing some
people that are actually logging into
their account you've got some people may
be doing returns an RMA requests you've
got others that might be actually
checking out and buying things actively
so those are the different scenarios we
would want to represent in that type of
application and hopefully those
scenarios are already represented in
your functional web tests if they are
you can simply capture the traffic this
is the little capture button here you
could capture the traffic from your
existing web test so again that reuse
really just giving you more mileage
again from from your test so let me go
ahead and do a quick one here we'll just
do this language test I'll pick a
browser here and it's going to now
execute that what this part does look
just like we saw previously just like we
see it in the functional test just like
we saw in the performance test however
once it's done
with this it will bring back the traffic
for us now it looks like the site may
have changed since this test was last
run so it might give us partial result
there might have a failure in it again
this type of test is not focused on
failure so if there is an issue it will
finish out what it can for the from what
its captured but it's again it's not
really going to be focused on fixing the
failure for that I'd need to go back to
my functional test to do that but here
in a moment we should get some data back
hopefully at least a little bit from
that test before it error it out we'll
try that with a different test as well
uh bear with me here it looks like it
may have been stuck a little bit from
our GoToMeeting we'll try to get that
screen Reshard here momentarily open one
thing I needed to do is actually I'm
going to make an adjustment to my
scheduling server you of course can have
that environment I showed previously
where you've got multiple machines
connected to each other I actually have
my machines sometimes occupied so I'm
actually going to just connect this from
a local low test I'm not gonna I don't
want to kill our session by having a low
testing running and keep in mind that
these low tests can really do some
damage so you definitely want to be
careful I always like to tell people
don't forget you know if you if you've
scheduled a load test don't forget
you've scheduled a load test but we've
set that to local now so that should be
just running locally and shouldn't
affect our GoToWebinar okay so back into
this design area and we'll try a test
that hopefully succeeds this time we'll
do capture from existing web tests and
let's see we'll do a super simple login
here and we'll have it go execute that
again the execution portion should look
much the same as what we are used to
with functional tests and now we see the
traffic that's been captured and the
first thing that comes back are the
dynamic targets now dynamic targets are
the session IDs that were detected by
test studio now keep in mind test you do
doesn't specifically target all session
IDs there are some out there that are
more advanced there are some that
certainly if you if you don't see them
in your list here you can contact us and
we're always trying to expand our
support for different session ID types
and things like that so definitely feel
free to you know if you don't see it
that doesn't mean we can't do it just
let us know what's missing if something
is missing that we can certainly try to
get it prioritized but
ideally here what we're seeing are the
session IDs in this case there's an
authentication token and a commit that
were generated as the user went through
the process and the the concept of this
is that we want to be able to make these
dynamic we want them typically to be
dynamic because that's how they usually
work when each of user each of the users
go to the page they should get a unique
authentication token and session ID that
help to shake hands with the application
that let the application know throughout
the process who they are and that
they're allowed to be there now when we
capture the load test scenario and get
the traffic like this it's captured the
value of these session IDs at the point
of execution so over here this value
this particular key here is what was
captured at the point of execution and
this might not be something that is
valid for other users or might expire
what we typically want to do is we want
to enable these this this list really
gives us the option to view them and
turn them on and off making them dynamic
or making them just disabled and the
test itself so typically we're going to
just check the checkbox to make them all
dynamic because that's generally how
they work in real life and we'll click
done you get back into the dynamic
targets up here I'll show you another
scenario where there's a ton of dynamic
targets and sometimes there certainly
can be some noise that you don't
necessarily need in your load test but
this is a pretty clean test here you can
see all the steps of going to the page
we also do have think time and actually
let me go ahead and click on the first
one here where you can see the request
as well as the response over here
there's a set cookie step here as well
and you know stop by step we can see all
of the different data points that were
captured from that and as I mentioned
there's thing time now think time is
something that's generated to give you
variability this means we put time into
the test to make it so that each of the
virtual users that when you execute this
with let's say a hundred users or
thousand users we want to have a little
bit of variation that's more like the
real world and think time helps you do
that so robotic
every virtual user will go through these
first four steps and then when they get
to think time they'll sit and wait and
they'll sit and wait for a duration of
10 seconds plus or minus 3 in this case
this is something I can completely
control and customize I can also get rid
of think time and I can also add
additional think time into the test the
idea here is that now each virtual user
will vary a little bit and that will be
a little bit more similar to what the
real world is like ok not everybody
operates like a robot so we make our
tests so they don't act like robots they
act more like people so you can add that
think time you can modify that thing
time and so forth here's our post
operation and then the post operation
this is where in fact if we go back and
look at our dynamic targets you can see
where the dynamic targets are generated
and captured from step 0 as soon as we
hit that page and where they're used in
this case
step 7 and that's the post operation
here you can see the actual
authentication token again and it's set
to from step 0
the reason is set to from scepter 0 is
because we've turned it on in the
dynamic target section already but here
I can change that if I needed to
it basically be undoing what I just did
but the idea is that it exists in both
areas so I could actually go back to the
value that was captured at runtime or I
could go to the dynamic value which is
captured for each virtual user
separately at their execution time or I
could even day to drive it so there's
actually a data source on this test
already and that's why we see these
other columns of data here this is also
how you can take a scenario and data
drive it so here's the username and
password that were passed by the
functional test that we just captured
and if we have a set of data maybe from
a spreadsheet or a database that we want
to use we could actually set that up in
the performance during the load test
here so that each virtual user could log
in as potentially different real user
with a different
user name and different password from
each other so that way we've got all
these different users logging in quite
slightly different timing involved so
it's a little bit more real-world and we
can even have them logging in with
different sets of credentials that are
coming from a data source okay
now that data source is something that
you can bring in at the the project
level I can show you where that is as
well but once it's attached to the load
test you'll see there's a little data
icon here indicating there is data on
that load test or there's a functional
test down here that has data on it too
now keep in mind that if I data if I
captured this simple login test down
here which is data driven that's
different that's a functional test it's
data driven so that means these fourteen
steps would be captured multiple times
and be repeated in this list and that's
typically not what we want because that
means when we execute the load test each
virtual user would be attempting to log
in more than once per user usually we
want them to log in one per user so that
means you typically want to capture one
single iteration just like we did of
that login process and then add data to
it in the load test so this is different
than data driving a functional test and
typically we're going to capture just
one iteration add the data in and here
we can just choose our username and our
password columns and we now have a data
driven scenario within our load test all
right so next we'll go ahead and jump
out of this oh I should mention -
there's some filtering that you can do
some cookies you can add in as well and
even some simple find/replace so maybe
we migrate this to another server we can
do a find/replace on that URL pretty
easily so that was capturing traffic
from an existing web test you can also
capture new sessions so here you don't
have to use what you've already made you
can go crazy and do your own thing to do
this we'll just click one of the
browsers here it will pop up that
browser for so I'm going to try to share
my screen
or move this around a bit and here we
could see that in the background it's
going to show us there's already one
request that's been captured and now
whatever we do in that browser should be
captured as well so if we jump over to
the test studio webpage you can see it's
jumped up to 56 to 60 requests captured
now I maple going to discover more here
we'll go look at the visual studio
plug-in we're up to over 100 requests
that have been captured now we can drill
down in and look at all these different
areas and maybe we'll click into another
page here we'll do a comparison table
we're up to 137 136 now I maybe will go
to the checkout here and look at some
pricing we'll browse some products and
we're up into the 270 range now 272 and
so forth okay well whenever we're done
with that we can close the browser we
can click stop recording and that
traffic that we just captured will come
back for us
so 287 requests that last test that we
did was 14 so quite a bit more that was
captured from this particular test and
we really didn't go that far into the
into the web page but you will see that
in this case there are a lot of dynamic
targets a ton right now not all of these
are valid not all of them are necessary
some of these are just noise that we
don't need to use in our load test again
the rule of thumb is to just turn them
on because actually in this next step
I'll show you how we can eliminate those
you can always turn them off from here
too and sometimes they're not well
labeled so you might actually have to
talk to the development team to
understand which of these are important
and which are not so forth
all right so that's the dynamic targets
in this case now as I mentioned there
may be some extra noise in this so
here's Telecom slash test studio
and there's a lot another telluric there
and then there's cloud front Optimizely
and Google tag manager and demand base
and all this other stuff now typically
we don't want exercise
somebody else's traffic here's an ad
from Twitter it looks like we don't
necessarily want to exercise Twitter or
ellicott or Facebook in this test those
happen to be there because they're
integrated to our website so if you're
on our website you see something cool
and you want to share it to Facebook you
click a little Facebook button and it
does the magic for you
well there's dynamic targets associated
with that and there are steps of that
traffic that are associated with that as
you can see this connects step and
really we don't necessarily need this
and this is where this filter really
helps and we can delete things kind of
line by line down here if you want to
but here you can do this at a domain
level so we'll just uncheck the ones
that we don't need and it will in turn
unselect any of those below that
correspond to it so we'll turn off our
Twitter and our let me get this back up
to where we want it we'll stretch this
out a little bit there we go yeah
there's some Bing and some cloud front
Google looks like Facebook Linkedin all
the social sharing kind of icons if you
can picture that you might have you
might even see some YouTube in here as
well so I'm gonna leave behind only that
telecom traffic that I was looking to
exercise actually oh well you will leave
the progress stuff as well up there's a
Celt teller there that will keep that
one I guess and that one there so now
I've disabled all of those and in turn
it is unchecked any of the corresponding
steps which also will now disable or
really just not use the corresponding
dynamic targets okay so even though we
have all these checked on here they're
not going to be used if the steps are
disabled right so some of those that
might pertain to Facebook could can live
in that list they can be set to dynamic
but since they're not even
exercise then it won't it won't be
necessary okay so we've filtered that
traffic out again you can go through and
really delete things out that you don't
want or turn them on and off kind of at
the step level here too so that's
capturing traffic from a new session so
we've captured traffic from existing web
tests we've captured traffic from a new
website web to session as well you can
use the different browsers if you'd like
and then also we can capture traffic
from any local source and this is really
good for capturing API traffic and then
we can capture traffic also from any
mobile source and of course that's good
for capturing traffic from a mobile
device now if you think about something
like Amazon again as an example I used
earlier there's probably a ton of people
these days that are using that on their
mobile device so we have to factor that
into our load test we may want to
simulate a certain amount of traffic
from a desktop browser but we also may
want to capture traffic that's generated
directly from a mobile app there's some
proxy settings that go along with this
and as you can see throughout our our
screens you might see these blue links
that'll take you right into our
documentation on how to set that up so
that you can capture that traffic from
the device itself now the device is not
needed for execution of the load test
just the capturing of the of the traffic
once it's captured we'll just exercise
that headless Lee from within test
studio ok so there's four different
options of capturing traffic right here
and the fifth option here that's letting
us import user profiles or user
scenarios from fiddler so if you're
familiar with fiddler four user fiddler
or if you even haven't heard of fiddler
check it out it's a very popular tool
for debugging and capturing web traffic
there's some greater filter capabilities
and you can actually see rather than
kind of cleaning the traffic up after
the fact like we did in studio you can
actively kind of monitor it and filter
things out during the process as well
so a little bit different it's
definitely one that I think our
developer classes or developer teams
prefer over our built-in but either one
works fine and of course fiddler is our
own product you're actually using part
of fiddler when you're using test studio
it's built in but that's another means
of capturing these scenarios so here we
have the different scenarios and this is
where we may have different workloads
for each of those scenarios so the
workloads are the distribution of load
for each of my scenarios so if we were
doing a test for a to simulate at 9:00
a.m. Monday morning we may have a
heavier amount of people that are just
doing the login portion on that
application and a smaller amount of
people they're using cascading combo
boxes and things like that so we can
basically put the slider where we'd like
to divvy up the percentage of total load
and that comes from our settings over
here so in this case I only have it up
to 30 you know what we'll push it up a
little bit we'll go up to 60 today I've
I just I keep this actually rather low
for demo purposes because the
GoToWebinar and things that we're doing
on the web can be degraded when I'm
running a low test and that's to be
expected right we're actually generating
load and we really in many cases want to
find out where things do degrade and
where things break so so I do keep this
pretty simple from her demo standpoint
but this is where you would put your
10,000 or 100,000 we've got some
customers that are in the hundreds of
thousands of virtual users to use to do
their load test so you could certainly
do that the test studio includes a
hundred virtual users with each of our
ultimate licenses so if your team buys
two licenses you'll have 200 to use and
they can easily be allocated that's
actually done up here I have six
ultimate licenses so I have six hundred
total and I can allocate those between
my different machines so and then this
is not something that's necessary it's
only if you want to
split your pool across different
scheduling servers so let me refer back
to my image real quick that there's
usually a central scheduling server that
talks to many machines so I'm not
telling tests to do to give this machine
200 and this one 100 and so forth all
I'm doing is saying hey here's 500
available to the scheduling server and
then during the load test it will choose
how to balance those across the agents
for me so all that's handled
automatically but all we're doing when
we're looking at that split here is if
we had more than one scheduling server
so if you can picture maybe we have two
of these big test labs and maybe there's
500 virtual users on each of our test
labs but one day we want to have all
thousand on one machine so that we can
do a little bit bigger load well then we
can easily change that up so again the
scheduling server in this case is the
one here in the middle typically you're
going to have one of those that connects
to many but you may also have more than
one scheduling server when you have
multiple teams multiple labs typically
and of course answer the question that
usually comes up after that all tests to
do licenses include the storage and
scheduling server capabilities so really
if we look at this diagram one more time
I've got six run times and three full
licenses here so I have nine different
machines they could have that could be
scheduling servers that all have
performance profiling built in that all
can execute web tests execute functional
tests can execute load tests as well and
then of course the ones that build and
maintain the tests are the full licenses
okay so that's kind of the difference
between a run time and a full license
but here all we're doing is we're saying
okay I've got more than one scheduling
server how do I want to split up my
total pool of virtual users across my
different scheduling servers so at the
test level this is where we're going to
specify what load ultimately we want to
achieve we may start at 59 and go to 60
we
start at zero go to 60 it just depends
on on what you want to do you you're not
required to to build like this but I do
like to see how things kind of progress
during sort of that growth phase and
then I sort of have a plateau here and
that actually come actually comes from
our ramp so we could actually start
right in with no ramp at all or we could
do more of your classical load test
stress test here where it's going to
continuously ramp but I like to have a
little bit of a ramp and then Plateau so
I can see kind of during that growth
phase how things transpire and then sort
of that maintained plateau phase how
things continue to to work I have this
set just to run for three minutes but
you can set that to whatever and
actually before I go further let me jump
back real quick to our documentation
because there's one more option I want
to give you or one more link here I want
to provide you which is called load
strategies and this talks about the
different kind of approaches to load
testing stress testing is sort of that
general kind of one to ten thousand
pedal-to-the-metal we're just gonna add
virtual users constantly throughout that
test and then as we scroll down you'll
see some of the data stuff there and
then here's more of the sope test the
endurance type test this actually talks
about running this test for multiple
hours days or even weeks you really want
to know how does that application
perform under strain for a long period
of time right so it's a little bit more
of a flat line at a constant level load
typically in those scenarios okay this
gets into a little bit more on
geographic diversity testing this is
actually where you might want to have
run times in different countries so you
can actually distribute your load from
different machines that maybe represent
different areas of the globe so there's
lots of cool things you can do there as
well all right so we've got scenarios
we've got work load distribution and
again for the purposes of this test I'm
not going to hit public sites I'm going
to hit my own test site because I don't
want to kill our GoToWebinar so we're
gonna add one percent of that 60
concurrent users going through the login
process that's being data-driven if
you'll remember there is also the
ability to manage windows authentication
identities and so you can add the
identity and that's actually instead of
having a data-driven load test where
each user uses a different set of
credentials this is really kind of the
opposite this is saying I want to
provide one common set of credentials
for all users all virtual users that are
going to go through the process and this
is something that's very useful when
you're in a Windows authentication
environment we're really just want to
get past the authentication point so we
can test the application itself but
you're gonna create those identities and
then here you would actually select the
identity to use for this scenario now on
the bottom here so we talked about sort
of the strategy stuff over here the only
other thing we have left is this and
this is something we've already covered
monitoring performance the same thing we
saw before now it's here in in this
environment too
so again same thing if we just click
into this we'll see the windows
performance counters we'll see all the
data points that we can look at and
evaluate and include in our test results
in our load load test results here oops
I do need to click on it I suppose and
you can see I've previously set this up
this also carries over the same
templates so whatever you've done in
performance testing you should see here
as well and you can use those same
templates now there's one other aspect
of these performance counters that is
different in load testing than the
performance testing we'll see that here
in a moment but now my load test is all
kind of set it's ready to go my scenario
is my workload my performance monitoring
and again keep in mind that we may be
monitoring machines that are not
executing right we're just monitoring
them we just want to see how they're
handling especially in a load test when
you have your app server you want to
know well ok I know I want to understand
the user perspective that's the
execution agent
but I uh but I really also want to know
how is the app server handling things
right because the user perspective is a
small segment of the overall load test
we're going to be generating multiple
virtual users per per machine but it's
the ultimate app server that usually
tells us how it's doing how its handling
things if that goes down then nobody has
access to the application right the user
experiences null let's flip over here to
the run tap and here on the run tab with
the load test this is really I think a
big differentiator for test Studios so
because here you can actually give it
the criteria to make a successful load
test and in order to pass a load test
you have to pass all of your goals
now these goals are usually with the
load test people just think about one
thing they think about how many users
can we handle right I call that the Big
Boss number the big boss says we got to
be able to handle 10,000 users
simultaneously and that's keyword two
simultaneously or concurrent you know
usually people get a little confused
about that there's total visits and then
there's concurrent visits right if we
look at our google statistics you know
we could probably get down to a minute
maybe I'm trying to remember how
specific it does get but we don't
necessarily see at a specific point in
time how many total users there are I
think we can usually get down to what it
looks like for a particular hour and
then sort of assimilate from there what
type of load to use but but typically
again that big boss number is you know
we need to be able to handle X number of
users simultaneously right now that's
not the whole story and that's that is
in here that's current current virtual
users for overall test so we want to try
to hit we put this up to 60 so I'm going
to hit 59 we're gonna pass this goal if
this goes over 59 and if it's a critical
goal we could say stop the test if this
goal is met something like an errors per
second we may want to say stop the test
of the
goal is met especially if we're about to
run a six hour soak test and it starts
airing in the first five minutes we
might want to stop and look at that
before we have it run for another five
hours and 55 minutes so you could
certainly turn that on but really the
rest of these goals spell out what I
call the edge of the cliff all right
because current virtual users how many
users can we get on a website is is
great that's a great concept of okay we
need 10,000 users but I've seen failures
like that in the past and usually what
happens is people think that about that
big boss number they're focused on that
big boss number and they're not
necessarily thinking about the user
experience so what I mean by that is we
may be able to fit 10,000 users on this
website without the app server smoking
and going down but it may be a 15-second
response time we may have a fraction of
those users seeing errors so they're on
our website but they're seeing a 404 or
something like that so it's not just
about how many users we can jam on there
it's about what's the average response
time that those users are seeing and of
course everybody likes to get to that
sub three-second response time so we
have a goal for that and of course
errors per second so I want to stay at
zero errors per second responses
received per second will maybe jack this
up to like 25 we'll see if we can get up
to that point and then here's our Big
Boss number and the goal for that and
I've also added another one and you can
actually add custom goals the custom
goals let you spell it out specifically
what you want for your test to be
considered a pass or fail so we're going
to create our own goal we're gonna pass
this goal or fail this goal if if either
the overall test which means all the
machines involved the execution agents
the servers that we're monitoring all
that stuff if the overall test or a
specific machine involved and when we
turn one of these on now we've got all
those performance counters that we've
enabled so we can actually go build very
specific goals around these performance
counters we could do like a processor
privilege time must stay under certain
threshold for example or transactions
per second needs to reach a certain
level now this not might not make a ton
of sense when you first see it you might
not know what your goals are to begin
with but the first key is to turn on the
monitoring because when when these are
turned on we're gonna get data back for
every single one of these after we run
the test the first time and once we
could get that data back then we can
know maybe what values to start working
on so so adding goals to your load tests
is something that you're going to kind
of hone in on over time okay so what we
don't expect you to know that right away
or or you know be ready to do that first
thing it's typically something that
you'll want to sort of hone in on but
that really allows you to spell out what
combination of things and values for
those is going to equal success for your
load test so I have to pass all these
goals in order to make a passing load
test alright now if my agents are ready
I'll hit run this test and it should get
things started for us it should talk to
the scheduler it should flip us over
here to the analyze tab here in a moment
and start bringing some data back for us
momentarily so here it goes the analyze
tab there are a couple of notes up there
you can enable and you can see there's
some previous results in here too and as
soon as we start getting new data back
it'll start giving us the new results
now you don't have to be here for a load
test it is something that's kind of fun
to watch and and kind of graph things
but you can always schedule these to run
as well I always mentioned you know
don't forget you scheduled it to run oh
so I had something oh there goes now
it's starting it's just getting started
up there so you could see it is running
I have a three minutes for 60 users
here's from current values and I have a
sampling rate I didn't point this out
but I have a sampling rate of every five
seconds so we're gonna get data back
every five seconds here currently are
the goals that were passing and failing
and here are page specific metrics and
any
that may have been encountered and their
status codes and so forth now the real
fun part of this I think is the graphing
where we can see as it's carrying out
the test how many errors per second how
many requests received how many sent
responses receive all those different
values and this is for overall test so
we do have all of the specific machine
counters here too so we can graph and
map those and watch those as well so
just a ton you know I've got a ton of
these turned on right now
more than I probably need but we could
graph those we can watch those and we
can see and to see here it's coming back
every every five seconds with data
points for us now what you'll be looking
for we have average response time that's
staying relatively low what you want to
avoid is a a ver adjouris Ponce time
that's super low because sometimes we
may have a test a site that's failing
and it's responding really quickly
saying hey it's failed it's failed right
so really really low response time is is
an indicator that there may be an issue
especially when it's coupled with a
spike in errors per second which is zero
down here right now the other thing we
want to avoid of course is a high
average response time so typically when
you see a server crash happening you'll
see the average response time spike and
then right after that average response
time spikes it drops drastically low and
at the same time the errors per second
starts to climb and climb and climb so
this particular graph looks like
everything's pretty clean
the responses request and receive have
been pretty much neck-and-neck the whole
time so that's very clean looks like and
again we can drill down into these
different machines here we can filter
down to a particular machine and we may
look at memory or these other ones that
we've got here oh there he goes from
right per second so you could see that
spike occurring for the memory you know
just a couple of those spikes now
sometimes you'll see spikes that they're
not always bad
you know you might see more of a
heartbeat where it's kind of flat and
the spike and then flat and then spike
that's typical you know when you see
that type of pattern that's that's
typically okay it's really where we look
for those anomalies and this ones is
this one suspect but it didn't didn't
seem to cause any errors so we're okay
but it was in that kind of growth phase
and as you can see there's a plateau
phase of my test now I think it's done
yep my test failed we had four goals out
of five pass so let's look at our goals
we get we got the big boss number we
made it to sixty concurrent users we had
zeros zero errors per seconds we didn't
crash the server our average response
time was was very low but not too low
and then a response to received we got
where we wanted on that and then this
goal was really unaffected actually I
don't even have this machine I don't
think on right now so it's not not
grabbing any data for us but that's our
perform our load testing result data one
more thing I know we're running short on
time here but one more thing I'm going
to show you is how we could take this
data so I have average response time
errors per second and current virtual
users sorry I have that set to overall
test so these three and I'm gonna jump
back and look at let's say the one in
the middle of December here we actually
failed on our average response time and
our responses received in that case but
what I can do is I could take that
and map it over this current result too
so now I actually have two results sets
overlaid on top of each other now both
of these ran for three minutes this one
says three minutes and nine seconds and
the reason for that if there are virtual
users still in the middle of doing
something they'll finish out so in my
load test if I have a 5 minute long
scenario and I run a 3 minute load test
well guess what that's gonna last least
5 minutes before the load test ends
because it will make sure each of those
virtual users finish their job before
they're released ok so it's natural to
see it go over a little bit but here you
can see that spike in average response
time looks like it didn't have in any
errors but it was potentially close to
erroring and this was actually done at a
lower level this was a 30 users so twice
the number of users and we still had
very good performance in fact this spike
looks to be somewhat of an anomaly but
this is what I'm gonna continue to watch
as I continue to increase this up to 90
120 or whatever values I want to set it
to I want to see at what threshold do we
you know start to see things kind of
come apart and again keep in mind
there's the idea of how close to the
edge of the cliff are we and then
there's the cliff the cliff itself does
how many virtual users can we handle
concurrently or current virtual users as
it's noted here that's the big boss
number of can we handle X number of
users that's like an on/off switch you
either can or you can't
at that particular for that particular
goal but it's the other ones the other
goals that tell you more about is
something that users are going to stick
around and use you know if it's if it's
10,000 users and this the server still
up but it's 15-second response time
you're not gonna keep many users around
too long right right on so it looks like
we're right at the right past the hour
I'm going to stick around a little bit
answer any additional questions that may
come up please feel free to hit the
question panel or the chat log with
anything you might have I want to thank
everybody for attending today and
hopefully this has been informative for
you definitely check out test
for your own self and let us know if you
need any assistance I'll also point out
before we go where you can access the
user guide up here in the help menu you
get to the user guide online you can get
to videos and forums and here's our
feedback portal where you can ask us for
new features and vote for things to help
us prioritize our roadmap and even
submit ticket you do have access to our
support team throughout your trial this
is a supported product so we ask you to
go ahead and trial our support with our
with our application as well all right
so that's it for me again thanks for
watching one more thing I'll note I did
talk about the micro vantage point of
the performance test the load test as
you may gather is a macro vantage point
and I will let you know that it's good
to run them together run a load test and
a lot of that load test is running run
performance tests as I mentioned before
with performance testing you want to see
how performance tests work at different
levels of load so use your load test to
generate the load and then hit it with
performance tests while you're doing
that all right thanks everybody have a
great day thanks for checking out tests
to do and happy testing