In my current team, we happen to have Azure Test Plan included in our enterprise subscription. This naturally leads down an investigation to see how this compares to the many other tools and platforms out there. It’s not in the scope for this post or for our needs but there are also loads of other hosted services available these days. You can also check out Blazemeter’s (another testing company) article on which tools you can use.
The interesting thing is, none of these articles even mention Azure Test Plan. So what if you’re on Azure cloud and is already paying for a subscription. Wouldn’t you want to use that? Reuse your stack and your C#/Microsoft skillset?
That is what we are going to find out.
Why do load testing?
Let’s consider the goal before we talk about the how to do it.
Load testing is what we talk about when we generate lots of traffic to our service to ensure that the infrastructure and code can handle peak traffic. Imagine for example that we are expecting a four-doubled amount of traffic during a sale it is imperative we test the service beforehand to ensure that we will not experience downtime. We may also generate load during a long period of time to ensure for example that the service does not run out of memory after hours of use.
We may also talk about performance testing, as in looking into our service to see if it is performing as well as it is supposed to be during extended periods of use or simply as in how fast our current service is responding. The main difference here is that we don’t need concurrency or load to measure this, it is enough to track the response time of a given endpoint over time to ensure that it responds fast enough and that new changes do not impact the performance. An example of manual performance testing is loading the web page and using chrome developer tools to check how fast certain resources are loading.
Who uses azure test plans?
In my personal experience it is mostly large enterprises or public sector that moves toward the Microsoft stack. This oftentimes means there is little sharing of practices and implementations. So maybe it’s not strange that the first 10 hits on googling for azure test plan comes up with only Microsoft hosted articles.
There are also some disconcerning notes around parts of it becoming discontinued, such as this article Cloud-based load testing service end of life stating that it will become end of life in March 2019.
There are still these two offerings from Microsoft though:
URL-based load tests
Some shortcomings of these URL based tests are:
- You cannot populate dynamic data, for example generate a list of users or go through a list of data entries
- You cannot act on returned data, for example reuse a returned login token
For these reasons I would argue that the URL based testing might be worthwhile for “read only” type of pages, blogs or CMS systems where the user flow does not involve logging in or posting data.
JMeter tests
The JMeter tests do exactly what it says on the box. It allows you to upload a JMeter test and run it. Which makes sense. Although if you’ve come as far as to start building JMeter tests you may want to consider managing these in source control.
JMeter tests are stored as xml files. JMeter is very flexible and despite not being scripted it can achieve most of what you might want to do with a script. You can use 3rd party scripts to convert HAR files to the JMeter .jmx format and open these in JMeter for execution or modification. There are functions to pull out data to verify or reuse in later and perform logic. It supports a wide range of formats (xml, json, html, urls) as well as protocols (http, ftp and more).
JMeter can give you all you would want to test. It is up to you to decide if JMeter and GUI based construction is what you prefer over something like Gatling which is scripted. My understanding is that JMeter does a lot and is well established whilst Gatling and similar frameworks are younger, targeted and more lightweight. I would compare it that JMeter is the fully fledged enterprise option whilst Gatling is minimal complexity DIY option.
There are a few good comparison articles out there from Baeldung (spring creators), Octoperf and DZone.
Triggering the tests
We may want to run performance testing continuously on code check-in to ensure our endpoints do not respond to slow but load tests can impact your network and take a long time to run, so doing it manually or scheduled might be preferable.
Though I haven’t found a way to trigger the Test Plan tests through scheduling, you can add an MS official step to your release pipeline:
Though there are some, now outdated, guides on how to schedule your load execution, I haven’t found an easy built-in way to do this with the Azure Load Test.
Testing in C#
Reuse your stack and Microsoft/C# skills.
This might be your initial reason to want to explore Azure Load Testing. However, looking around for testing tools you’ll quickly notice that no DotNet tools make the top ten. These tools are more often than not either relying on open source frameworks such as JMeter or they execute tests build visually inside the tool.
Googling load testing in c# shows equally bleak prospects.
It’s not that there aren’t any tools out there, NBomber for example, seems sleek but, comparing community support with the other giants like JMeter or Gatling it can’t compare.
There were load testing tools inside Visual Studio but these got deprecated early 2019. Their motivation was that most teams rely on internal toolsets for the load testing anyway so I suspect that the community need/want for load testing simply wasn’t big enough which might explain the lack for C# frameworks out there as well. Maybe we might start seeing something coming up with the increased c# open source community within .Net Core (like BenchmarkDotNet).
So, unless you’re very intent on staying with C# it looks like the best option is to add another language or tool to your CV.
How it compares
The alternatives usually come in two flavours, hosted or open source that you run locally. The one is usually paid and the other you pay for in sweat and tears.
That being said, in my experience, most of the effort in load testing is not the actual writing of the tests but rather to figure out what to test and then how to interpret the results. How many users doing what actions concurrently do you need to test? How many services should you hit concurrently?
Using Azure DevOps to run the tests does completely remove the need to think about where to run the tests.
It’ll all run within your test plan on Microsoft cloud infrastructure. That can be a big win since you don’t want to overload your build servers during load testing.
As mentioned previously there are a few issues around lacking support and features getting deprecated and seeing posts about how deleting tests is not possible is of course not very encouraging.
The conclusion is that if you are already paying for Azure, reusing that infrastructure to execute your test may very well be a good idea. Given that you are willing to get into JMeter or can manage with URL based testing.
Have you got other experiences, amendments to what I’ve written here? Please reach out and help settle the question!
Update 2019-06-25: On not being able to delete tests
I may be a bit outdated in this topic. But I created url based tests with MS load tests a couple of years ago before they migrated them from VSTS to devops. I wrote an extension for dynamic users and one to extract the access token from the callback from our login page and added it in the header on all requests.
I did go for the url based then but maybe something has changed?
Cheers! I had another look at it, noticed the picture was broken and fixed that but alas, doesn’t look like you can put any conditionals into the URL based. As you say, maybe it’s possible to upload a visual studio test suite though. And as with the article you provided (albeit a bit dated now) use that to run chromium or something headless to generate proper load using Selenium tests:
https://devblogs.microsoft.com/devops/using-selenium-with-cloud-based-load-testing/