| title | titleSuffix | description | services | ms.service | ms.topic | ms.author | author | ms.date | adobe-target |
|---|---|---|---|---|---|---|---|---|---|
Load test configuration YAML | Azure Load Testing | Learn how to configure a load test by using a YAML file. The YAML configuration is used for setting up automated load testing in a CI/CD pipeline. | load-testing | azure-load-testing | reference | ninallam | ninallam | 12/06/2023 | true |
Learn how to configure your load test in Azure Load Testing by using YAML. You use the test configuration YAML file to create and run load tests from your continuous integration and continuous delivery (CI/CD) workflow.
A load test configuration uses the following keys:
| Key | Type | Required | Default value | Description |
|---|---|---|---|---|
version | string | Y | Load test specification version. The only supported value is v0.1. | |
testId | string | Y | Unique identifier of the load test. The value must be between 2 and 50 characters ([a-z0-9_-]). For an existing test, you can get the testId from the test details page in the Azure portal. | |
testName | string | N | Deprecated. Unique identifier of the load test. This setting is replaced by testId. You can still run existing tests with the testName field. | |
displayName | string | N | Display name of the test. This value is shown in the list of tests in the Azure portal. If not provided, testId is used as the display name. | |
description | string | N | Short description of the test. The value has a maximum length of 100 characters. | |
testType | string | Y | Test type. Possible values:
| |
testPlan | string | Y | Reference to the test plan file.
| |
engineInstances | integer | Y | Number of parallel test engine instances for running the test plan. Learn more about configuring high-scale load. | |
configurationFiles | array of string | N | List of external files, required by the test script. For example, JMX fragment files, CSV data files, images, or any other data file. Azure Load Testing uploads all files in the same folder as the test script. In the JMeter script or the Locust script, only refer to external files using the file name, and remove any file path information. | |
failureCriteria | object | N | L oad test fail criteria. See failureCriteria for more details. | |
autoStop | string or object | N | Automatically stop the load test when the error percentage exceeds a value. Possible values: - disable: don't stop a load test automatically.- object: see autostop configuration for more details. | |
properties | object | N |
| |
zipArtifacts | array of string | N | Specifies the list of zip artifact files. For files other than the main test script and user properties for JMeter-based tests and Locust script and configuration files for Locust-based tests, if the file size exceeds 50 MB, compress them into a ZIP file. Ensure that the ZIP file remains below 50 MB in size. Only 5 ZIP artifacts are allowed with a maximum of 1000 files in each and uncompressed size of 1 GB. Only applies for testType: JMX and testType: Locust. | |
splitAllCSVs | boolean | N | False | Split the input CSV files evenly across all test engine instances. For more information, see Read a CSV file in load tests. |
secrets | object | N | List of secrets that the Apache JMeter or Locust script references. See secrets for more details. | |
env | object | N | List of environment variables that the Apache JMeter script or Locust references. See environment variables for more details. | |
certificates | object | N | List of client certificates for authenticating with application endpoints in the JMeter or Locust script. See certificates for more details. | |
appComponents | object | N | List of server -side resources to monitor during the load test. See appComponents for more details. | |
subnetId | string | N | Resource ID of the virtual network subnet for testing privately hosted endpoints. This subnet hosts the injected test engine VMs. For more information, see how to load test privately hosted endpoints. | |
publicIPDisabled | boolean | N | Disable the deployment of a public IP address, load balancer, and network security group while testing a private endpoint. For more information, see how to load test privately hosted endpoints. | |
regionalLoadTestConfig | object | N | Distribute load across regions to simulate user traffic from multiple regions. For more information, See regional load test configuration for more details. | |
referenceIdentities | object | N | List of managed identities used in the test for accessing the secrets from your Azure Key Vault, metrics for server-side failure criteria and authentication of your endpoints. See referenceIentities for more details. |
The following YAML snippet contains an example load test configuration.
version: v0.1testId: SampleTestdisplayName: Sample Testdescription: Load test website home pagetestPlan: SampleTest.jmxtestType: JMXengineInstances: 1subnetId: /subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/sample-rg/providers/Microsoft.Network/virtualNetworks/load-testing-vnet/subnets/load-testingconfigurationFiles: - 'sampledata.csv' - ‘testfragment.jmx’zipArtifacts: - bigdata.zipsplitAllCSVs: TruefailureCriteria: - avg(response_time_ms) > 300 - percentage(error) > 50 - GetCustomerDetails: avg(latency) >200autoStop: errorPercentage: 80timeWindow: 60secrets: - name: my-secretvalue: https://akv-contoso.vault.azure.net/secrets/MySecret/abc1234567890def12345keyVaultReferenceIdentity: /subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/sample-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/sample-identityTest fail criteria enable you to define conditions to determine if a load test run was successful or not. If one or more fail criteria are met, the test gets a failed test result. Learn more about using load test fail criteria. You can define failure criteria on client metrics, such as response time, latency, etc., and on server-side metricc for your server-side app components.
You can define fail criteria that apply to the entire load test, or that apply to a specific request. Fail criteria have the following structure:
- Test criteria at the load test level:
Aggregate_function (client_metric) condition threshold. - Test criteria applied to specific requests:
Request: Aggregate_function (client_metric) condition threshold.
Azure Load Testing supports the following client metrics:
| Metric | Aggregate function | Threshold | Condition | Description |
|---|---|---|---|---|
response_time_ms | avg (average)min (minimum)max (maximum)pxx (percentile), xx can be 50, 75, 90, 95, 96, 97, 98, 99, 999 and 9999 | Integer value, representing number of milliseconds (ms). | > (greater than)< (less than) | Response time or elapsed time, in milliseconds. Learn more about elapsed time in the Apache JMeter documentation. |
latency | avg (average)min (minimum)max (maximum)pxx (percentile), xx can be 50, 90, 95, 99 | Integer value, representing number of milliseconds (ms). | > (greater than)< (less than) | Latency, in milliseconds. Learn more about latency in the Apache JMeter documentation. |
error | percentage | Numerical value in the range 0-100, representing a percentage. | > (greater than) | Percentage of failed requests. |
requests_per_sec | avg (average) | Numerical value with up to two decimal places. | > (greater than) < (less than) | Number of requests per second. |
requests | count | Integer value. | > (greater than) < (less than) | Total number of requests. |
You can define fail criteria on server-side metrics for your server-side app components.
The following table describes the different fields in the serverMetrics: configuration:
| Parameter | Description |
|---|---|
resourceId | Required. The resource ID of app component on which the criteria should be applied |
metricNamespace | Required. The server-side metric namespace. |
metricName | Required. The server-side metrics on which the criteria should be applied. |
aggregation | Required. The aggregation to apply on the server-side metrics. |
condition | Optional. The comparison operator, such as greater than, or less than. |
value | Required. The numeric value to compare with the metric. |
The following code snippet shows a load test configuration, which has three load test fail criteria.
version: v0.1testId: SampleTestdisplayName: Sample Testdescription: Load test website home pagetestPlan: SampleTest.jmxtestType: JMXengineInstances: 1failureCriteria: clientMetrics: - avg(responseTimeMs) > 300 - percentage(error) > 50 - getCustomerDetails: avg(latency) > 200serverMetrics: - resourceId: /subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/sample-rg/providers/Microsoft.Compute/virtualMachines/sample-vmmetricNamespace: Microsoft.Compute/virtualMachinesmetricName: Percentage CPUaggregation: Averagecondition: GreaterThanvalue: 80 - resourceId: /subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/sample-rg/providers/Microsoft.Compute/virtualMachines/sample-vmmetricNamespace: Microsoft.Compute/virtualMachinesmetricName: Available Memoryaggregation: Averagecondition: LessThanvalue: 20You can monitor server-side resources during the load test. Learn more about monitoring server-side resources. When you run a load test for an Azure-hosted application, Azure Load Testing collects resource metrics for your application components and presents them in the load testing dashboard.
The following table describes the different fields in the appComponents: configuration:
| Parameter | Description |
|---|---|
resourceId | Required. The resource ID of app component on which the criteria should be applied. |
resourceName | Optional. The name of the resource to be monitored. |
kind | Optional. The kind of the resource to be monitired. |
metrics | Required. The list of metrics to be monitored for the app component. This contains the name, namespace, and aggregation for the metric. |
The following code snippet shows a load test configuration, which has two app components.
version: v0.1testId: SampleTestdisplayName: Sample Testdescription: Load test website home pagetestPlan: SampleTest.jmxtestType: JMXengineInstances: 1appComponents: - resourceId: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/samplerg/providers/microsoft.insights/components/appComponentResource"resourceName: appComponentResource #Optionalkind: web # Optionalmetrics: - name: "requests/duration"namespace: microsoft.insights/components aggregation: "Average" - name: "requests/count"aggregation: "Total"namespace: microsoft.insights/components - resourceId: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/samplerg/providers/microsoft.insights/components/appComponentResource"resourceName: appComponentResource #Optionalkind: web # Optionalmetrics: - name: "requests/duration"aggregation: "Average"namespace: microsoft.insights/components - name: "requests/count"aggregation: "Total"namespace: microsoft.insights/componentsThe load test autostop functionality enables you to automatically stop a load test when the error percentage exceeds a specific threshold during a given time window. Learn more about the load test autostop functionality.
| Key | Type | Default value | Description |
|---|---|---|---|
errorPercentage | integer | 90 | Threshold for the error percentage, during the timeWindow. If the error percentage exceeds this percentage during any given time window, the test run stops automatically. |
timeWindow | integer | 60 | Time window in seconds for calculating the errorPercentage. |
The following code snippet shows a load test configuration, which has three load test fail criteria.
version: v0.1testId: SampleTestdisplayName: Sample Testdescription: Load test website home pagetestPlan: SampleTest.jmxtestType: JMXengineInstances: 1autoStop: errorPercentage: 80timeWindow: 60You can specify a JMeter user properties file for your load test. The user properties file is uploaded alongside the test plan and other files. Learn more about using JMeter user properties in Azure Load Testing.
| Key | Type | Default value | Description |
|---|---|---|---|
userPropertyFile | string | File to use as an Apache JMeter user properties file or a Locust configuration file. For Locust, files with extensions .conf, .ini and .toml are supported as a configuration file. The file is uploaded to the Azure Load Testing resource alongside the test script and other configuration files. If the file is in a subfolder on your local machine, use a path relative to the location of the test script. |
The following code snippet shows a load test configuration, which specifies a user properties file.
version: v0.1testId: SampleTestdisplayName: Sample Testdescription: Load test website home pagetestPlan: SampleTest.jmxtestType: JMXengineInstances: 1properties: userPropertyFile: 'user.properties'The following code snippet shows a load test configuration, which specifies a Locust configuration file.
version: v0.1testId: SampleTestdisplayName: Sample Testdescription: Load test website home pagetestPlan: SampleTest.pytestType: LocustengineInstances: 1properties: userPropertyFile: 'locust.conf'You can store secret values in Azure Key Vault and reference them in your test plan. Learn more about using secrets with Azure Load Testing.
| Key | Type | Default value | Description |
|---|---|---|---|
name | string | Name of the secret. This name should match the secret name that you use in the test plan requests. | |
value | string | URI (secret identifier) for the Azure Key Vault secret. |
The following code snippet shows a load test configuration, which references a secret my-secret in Azure Key Vault.
version: v0.1testId: SampleTestdisplayName: Sample Testdescription: Load test website home pagetestPlan: SampleTest.jmxtestType: JMXengineInstances: 1secrets: - name: my-secretvalue: https://akv-contoso.vault.azure.net/secrets/MySecret/abc1234567890def12345You can specify environment variables and reference them in your test plan. Learn more about using environment variables with Azure Load Testing.
| Key | Type | Default value | Description |
|---|---|---|---|
name | string | Name of the environment variable. This name should match the variable name that you use in the test plan requests. | |
value | string | Value of the environment variable. |
The following code snippet shows a load test configuration, which specifies an environment variable my-variable and value my-value.
version: v0.1testId: SampleTestdisplayName: Sample Testdescription: Load test website home pagetestPlan: SampleTest.jmxtestType: JMXengineInstances: 1env: - name: my-variablevalue: my-valueYou can pass client certificates to your load test. The certificate is stored in Azure Key Vault. Learn more about using client certificates with Azure Load Testing.
| Key | Type | Default value | Description |
|---|---|---|---|
name | string | Name of the certificate. | |
value | string | URI (secret identifier) for the certificate in Azure Key Vault. |
The following code snippet shows a load test configuration, which references a client certificate in Azure Key Vault.
version: v0.1testId: SampleTestdisplayName: Sample Testdescription: Load test website home pagetestPlan: SampleTest.jmxtestType: JMXengineInstances: 1certificates: - name: my-certificatevalue: https://akv-contoso.vault.azure.net/certificates/MyCertificate/abc1234567890def12345You can use managed identities for various scenarios in your load test. Managed identities can be used in the test for accessing the secrets or certificates from your Azure Key Vault, fetching metrics for server-side failure criteria and authentication of your endpoints.
The following table describes the different fields in the referenceIdentities: configuration:
| Parameter | Description |
|---|---|
kind | Required. This defines the scenario for which the managed identity is being used. This can be one of the following KeyVault, Metricsor Engine. There can be multiple items for the kind Engine. |
type | Required. The type of identity. This can be UserAssigned or SystemAssigned. |
value | Required. The resource ID of the managed identity. This need not be provided if the type is SystemAssigned. |
The following code snippet shows a load test configuration, for multiple identities.
version: v0.1testId: SampleTestdisplayName: Sample Testdescription: Load test website home pagetestPlan: SampleTest.jmxtestType: JMXengineInstances: 1secrets: - name: my-secretvalue: https://akv-contoso.vault.azure.net/secrets/MySecret/abc1234567890def12345failureCriteria: serverMetrics: - resourceId: /subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/sample-rg/providers/Microsoft.Compute/virtualMachines/sample-vmmetricNamespace: Microsoft.Compute/virtualMachinesmetricName: Percentage CPUaggregation: Averagecondition: GreaterThanvalue: 80referenceIdentities: - kind: KeyVaulttype: UserAssignedvalue: /subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/sample-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/sample-identity - kind: Metricstype: SystemAssigned - kind: Enginetype: UserAssignedvalue: /subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/sample-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/sample-identity - kind: Enginetype: UserAssignedvalue: /subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/sample-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/sample-identity1If you use a URL-based test, you can specify the HTTP requests in a JSON file instead of using a test script. Make sure to set the testType to URL in the test configuration YAML file and reference the requests JSON file.
The requests JSON file uses the following properties for defining requests in the requests property:
| Property | Type | Description |
|---|---|---|
requestName | string | Unique request name. You can reference the request name when you configure test fail criteria. |
responseVariables | array | List of response variables. Use response variables to extract a value from the request and reference it in a subsequent request. Learn more about response variables. |
responseVariables.extractorType | string | Mechanism to extract a value from the response output. Supported values are XPathExtractor, JSONExtractor, and RegularExpression. |
responseVariables.expression | string | Expression to retrieve the response output. The expression depends on the extractor type value. |
responseVariables.variableName | string | Unique response variable name. You can reference this variable in a subsequent request by using the {$variable-name} syntax. |
queryParameters | array | List of query string parameters to pass to the endpoint. |
queryParameters.key | string | Query string parameter name. |
queryParameters.value | string | Query string parameter value. |
requestType | string | Type of request. Supported values are: URL or CURL. |
endpoint | string | URL of the application endpoint to test. |
headers | array | List of HTTP headers to pass to the application endpoint. Specify a key-value pair for each header. |
body | string | Body text for the HTTP request. You can use the requestBodyFormat to specify the format of the body content. |
requestBodyFormat | string | Format of the body content. Supported values are: Text, JSON, JavaScript, HTML, and XML. |
method | string | HTTP method to invoke the endpoint. Supported values are: GET, POST, PUT, DELETE, PATCH, HEAD, and OPTIONS. |
curlCommand | string | cURL command to run. Requires that the requestType is CURL. |
The following JSON snippet contains an example requests JSON file:
{ "version": "1.0", "scenarios": { "requestGroup1": { "requests": [ { "requestName": "add", "responseVariables": [], "queryParameters": [ { "key": "param1", "value": "value1" } ], "requestType": "URL", "endpoint": "https://www.contoso.com/orders", "headers": { "api-token": "my-token" }, "body": "{\r\n\"customer\": \"Contoso\",\r\n\"items\": {\r\n\t\"product_id\": 321,\r\n\t\"count\": 50,\r\n\t\"amount\": 245.95\r\n }\r\n}", "method": "POST", "requestBodyFormat": "JSON" }, { "requestName": "get", "responseVariables": [], "requestType": "CURL", "curlCommand": "curl --request GET 'https://www.contoso.com/orders'" }, ], "csvDataSetConfigList": [] } }, "testSetup": [ { "virtualUsersPerEngine": 1, "durationInSeconds": 600, "loadType": "Linear", "scenario": "requestGroup1", "rampUpTimeInSeconds": 30 } ] }The requests JSON file uses the following properties for defining the load configuration in the testSetup property:
| Property | Type | Load type | Description |
|---|---|---|---|
loadType | string | Load pattern type. Supported values are: linear, step, and spike. | |
scenario | string | Reference to the request group, specified in the scenarios property. | |
virtualUsersPerEngine | integer | All | Number of virtual users per test engine instance. |
durationInSeconds | integer | All | Total duration of the load test in seconds. |
rampUpTimeInSeconds | integer | Linear, Step | Duration in seconds to ramp up to the target number of virtual users. |
rampUpSteps | integer | Step | The number of steps to reach the target number of virtual users. |
spikeMultiplier | integer | Spike | The factor to multiply the number of target users with during the spike duration. |
spikeHoldTimeInSeconds | integer | Spike | Total duration in seconds to maintain the spike load. |
You can distribute load across regions to better simulate real life traffic patterns. You can specify the regions that you want to generate the load from and the amount of load that you want to simulate from each region. You can do that by specifying the region name and the number of engine instances that you want in that region. Learn more about generating load from multiple regions.
| Key | Type | Default value | Description |
|---|---|---|---|
region | string | Name of the Azure region. | |
engineInstances | integer | Number of engine instances for that Azure region. |
The following code snippet shows a load test configuration, which specifies two Azure regions eastus and eastasia and the number of engine instances for each region.
displayName: Sample TesttestPlan: sampleScript.jmxdescription: 'Load test website home page'engineInstances: 4testId: SampleTesttestType: LocustsplitAllCSVs: FalseregionalLoadTestConfig: - region: eastusengineInstances: 2 - region: eastasiaengineInstances: 2failureCriteria: - p90(response_time_ms) > 10000autoStop: errorPercentage: 90timeWindow: 60- Learn how to build automated regression testing in your CI/CD workflow.
- Learn how to parameterize load tests with secrets and environment variables.
- Learn how to load test secured endpoints.
