Test Description
Introduction
The test description in Skyramp simplifies the process of writing and running tests for complex distributed applications. It consists of three primary components:
-
Test Configuration: This file, residing in the
tests
folder, defines the overall behavior of the test. It enables you to configure test patterns and load testing. -
Scenario Configuration: Found in the
scenarios
folder, these files define request behavior for specific methods or chains of requests and asserts in scenarios. They also allow you to configure payloads, dynamic requests, and set overrides and parameters. -
Endpoint Configuration: Located in the
endpoints
folder, these files specify details related to the service’s networking aspects, supporting gRPC, REST, JSON-RPC WebSocket, and JSON-RPC HTTP endpoints.
To get started, follow the steps outlined in the How to Test Services page. This guide will teach you how to dynamically generate a test description by providing service-level information. Alternatively, if you prefer to create a test definition from scratch, you can create .yaml
files in the tests
, scenarios
, and endpoints
directories of your project (e.g., my-test.yaml
, my-scenario.yaml
, and my-endpoint.yaml
) and configure the necessary information by following the guidelines below.
Test Configuration
The test configuration serves as the central component of the test definition and defines the overall test behavior.
Example Test Configuration:
version: v1
test:
name: routeguide
testPattern:
- startAt: 1
scenarioName: scenario1
atOnce: 2
targetRPS: 3000
duration: 10
rampUp:
duration: 3
interval: 1
override:
mock:
- endpointName: helloworld
methodName: SayHello
blob: |
{
"message": "myTest"
}
In this example:
name
: Specifies the name of your test.testPattern
: Enables the definition of various test scenarios.override
: Allows you to override mock behavior.
This example showcases advanced test capabilities, including:
- Overrides: Customizing endpoint behaviors by specifying mocks.
- Load Testing: Simulating heavy user traffic with features like target RPS and ramp-up controls.
Overrides
The override
attribute in the test configuration allows you to customize specific endpoints within your tests, providing flexibility in your testing scenarios. By setting the override
attribute, you can specify a mock
to modify an endpoint
defined elsewhere in the project folder. This customization enables you to simulate various scenarios and test your application’s robustness effectively.
Example Test Configuration:
version: v1
test:
name: routeguide
testPattern:
- startAt: 1
scenarioName: scenario1
override:
mock:
- endpointName: routeguide_jMBp
methodName: GetFeature
blob: |-
{
"name": "fake",
"location": {
"latitude": 400,
"longitude": 600
}
}
The override
attribute is used to customize endpoints. The mock
section in the example specifies the details of the override:
endpointName
: Identifies the endpoint you want to override, in this case,routeguide_jMBp
.methodName
: Specifies the specific method of the endpoint you wish to modify, here,GetFeature
.blob
: Contains the customized data or response that will replace the original response from the endpoint.
By leveraging the override
attribute, you can seamlessly adapt endpoint behaviors within your tests, allowing you to create various testing scenarios and evaluate your application’s performance under different conditions.
Load Testing
Load testing allows you to simulate heavy user traffic on your application by transforming functional tests into load tests. This can be achieved by incorporating specific load profile keywords into your tests.
Example Test Configuration:
version: v1
test:
name: routeguide
testPattern:
- startAt: 1
scenarioName: scenario1
atOnce: 2
targetRPS: 3000
duration: 10
rampUp:
duration: 3
interval: 1
The atOnce
attribute in the testPattern
signifies the concurrency of the scenario1
. With atOnce
set to 2
, everything defined in scenario1
will run with a concurrency of 2, happening in parallel.
To control the load, you can utilize the following parameters:
targetRPS
: Specifies the target Requests Per Second (RPS) your application should handle.duration
: Indicates the total duration of the load test in seconds.rampUp
: Allows you to gradually increase the load on an endpoint.duration
: Specifies how long it takes for the traffic ramp-up to occur.interval
: Signifies the rate at which traffic increases.
In the provided example, the load increases over 3 seconds, with increments every second until it reaches the target RPS of 3000. Modifying these values enables you to model traffic behavior for your application and test how it responds to varying levels of load. If targetRPS
is not specified, Tester will attempt to send as many requests as possible within the system’s context.
Scenario Configuration
The scenario configuration file defines request behaviors for specific methods and creates scenarios as chains of defined requests and asserts.
Example Scenario Configuration:
version: v1
scenarios:
- name: scenario1
steps:
- requestName: GetFeature_18GO
- asserts: requests.GetFeature_18GO.res.name == "fake"
requests:
- name: GGetFeature_18GO
javascript: |-
function handler(req) {
// Your JavaScript logic here
return {
value: {
latitude: x,
longitude: y
}
};
}
- name: ListFeatures_5P9V
blob: |-
{
"hi": {
"latitude": 0,
"longitude": 0
},
"lo": {
"
latitude": 0,
"longitude": 0
}
}
endpointName: routeguide_jMBp
methodName: ListFeatures
# More request configurations...
In this example:
scenarios
: Allows you to define different test scenarios, and this example includes a scenario namedscenario1
.requests
: Contains configurations for individual requests (to be referenced in scenarios by name).
Some advanced capabilities of the scenario configuration include:
- Scenarios and Asserts: Creating end-user use cases and validating responses.
- Dynamic requests: Customize request handling logic.
- Chaining and overrides: Chain values between sequential requests and overriding them as needed.
Scenarios and Asserts
Scenarios are representations of end-user use cases that require testing, allowing you to define a sequence of actions to be performed, typically involving requests to specific endpoints. Each named scenario includes a steps
attribute, listing these actions, which can be executed sequentially or concurrently, simulating various usage patterns, including load tests.
Example Scenario Configuration:
version: v1
scenarios:
- name: scenario1
steps:
- requestName: GetFeature_18GO
- asserts: requests.GetFeature_18GO.res.name == "fake"
requests:
- name: GetFeature_18GO
endpointName: routeguide_jMBp
methodName: GetFeature
javascript: |-
function handler(req) {
// Your JavaScript logic here
return {
value: {
latitude: x,
longitude: y
}
};
}
# More request configurations...
In this example:
scenario1
is defined, containing two steps executed sequentially.- The first step utilizes the
requestName
attribute, referencing arequest
object previously defined in the configuration. - The second step is an
assert
statement, used to verify that the response from the request matches the expected value.
To use an assert, the asserts
parameter is defined within the step, with a value in the format:
requests.<name of request>.res.message == "<expected value>"
Where <name of request>
refers to the name of the previously defined request
, and <expected value>
is a string representing the expected return value from the request.
Note: Values returned by services are interpreted as JavaScript for evaluating assert
statements. The type may not always be a string
, but could be a boolean
or a number
, among other types. When working with a boolean
, the <expected value>
should be true
or false
and not "true"
or "false"
in the assert
statement.
In scenarios
, the associated steps are executed sequentially. To execute items in parallel, refer to the testPattern
defined in the test configuration.
Dynamic Requests
Dynamic requests provide the flexibility to customize request handling logic in your scenario configurations. You can use different attributes to specify dynamic request behavior, such as python
, pythonPath
, javascript
, or javascriptPath
. Each attribute allows you to define custom request handling logic and return a JSON representation of the response value.
JavaScript Dynamic Requests
-
Using the
javascript
AttributeTo create JavaScript-based dynamic requests, employ the
javascript
attribute within therequestValue
section of your request definition. Define a function calledhandler
that takes thereq
parameter. Implement your custom JavaScript logic within thehandler
function, and return a JSON object representing the response value.Example Scenario Configuration:
version: v1 requests: - name: GetFeature_18GO javascript: |- function handler(req) { // Your JavaScript logic here return { value: { latitude: x, longitude: y } }; } endpointName: routeguide_jMBp methodName: GetFeature # More request configurations...
-
Using the
javascriptPath
AttributeAlternatively, you can use the
javascriptPath
attribute to specify the path to an external JavaScript script file that contains your custom request handling logic.Example Scenario Configuration:
version: v1 requests: - name: GetFeature_18GO javascriptPath: scripts/getFeature.js endpointName: routeguide_jMBp methodName: GetFeature # More request configurations...
The external JavaScript script file
getFeature.js
defines ahandler
function to process incoming requests and generate appropriate responses.
Installing NPM-Based Packages
If your JavaScript-based dynamic requests require NPM-based packages, follow these steps:
Specify the required packages in the npmPackages
section of your test definition. The testing framework will automatically install these packages before running your test.
Example Test Configuration:
version: v1
test:
name: routeguide
testPattern:
- requestName: GetFeature_18GO
startAt: 1
- requestName: ListFeatures_5P9V
startAt: 1
- requestName: RecordRoute_MD5D
startAt: 1
- requestName: RouteChat_QQEf
startAt: 1
npmPackages:
- mathjs
- chart
Python Dynamic Requests
Note: If your dynamic Python request is dependent on additional Python modules, refer to the Installing Worker with Python Modules section to learn how to build the custom Skyramp Worker image.
-
Using the
python
AttributeYou can use the
python
attribute within therequestValue
section of your request definition. This attribute allows you to define a function calledhandler
that takes thereq
parameter, representing the incoming request. Within thehandler
function, you can implement your custom Python logic. Finally, return a JSON representation of the response value usingSkyrampValue
.Example Scenario Configuration:
version: v1 requests: - name: GetFeature_18GO python: |- def handler(req): // Your Python logic here return SkyrampValue( value={ "latitude": x, "longitude": y } ) endpointName: routeguide_jMBp methodName: GetFeature # More request configurations...
-
Using the
pythonPath
AttributeAlternatively, you can use the
pythonPath
attribute to specify the path to an external Python script file containing your custom request handling logic.Example Scenario Configuration:
version: v1 requests: - name: GetFeature_18GO pythonPath: scripts/get_feature.py endpointName: routeguide_jMBp methodName: GetFeature # More request configurations...
The external Python script file
get_feature.py
defines ahandler
function to process the request and generate the response.
Chaining and Overrides
Tester provides a powerful feature that allows you to chain values between sequential requests and override them.
Example Test Configuration:
version: v1
test:
name: routeguide
testPattern:
- startAt: 1
scenarioName: scenario1
override:
mock:
- endpointName: routeguide_jMBp
methodName: RouteChat
javascript: |-
function handler(req) {
return {
value: {
message: req.value.message + "temp",
location: {
latitude: req.value.latitude,
longitude: req.value.longitude
}
}
}
}
Example Scenario Configuration:
version: v1
scenarios:
- name: scenario1
steps:
- requestName: RouteChat_QQEf
- asserts: requests.RouteChat_QQEf.res.message == "message1temp"
- requestName: RouteChat_QQEf
override:
message: requests.RouteChat_QQEf.res.message
- asserts: requests.RouteChat_QQEf.res.message == "message1temp1temp"
- requestName: RouteChat_QQEf
override:
message: requests.RouteChat_QQEf.res.message
- asserts: requests.RouteChat_QQEf.res.message == "message1temp1temp1temp"
requests:
- name: RouteChat_QQEf
endpointName: routeguide_jMBp
methodName: RouteChat
vars:
message: "message"
javascript: |
i = 0
function handler() {
i++
return {
value: {
message: vars.message + i,
location: {
latitude: req.value.latitude,
longitude: req.value.longitude
}
}
}
}
# More request configurations...
In this example:
-
The
override
attribute in thetest
section allows you to customize the behavior of anendpoint
defined elsewhere in the project folder by specifying amock
. -
Within the
scenarios
section, multiple steps are defined. Each step calls theRouteChat_QQEf
request and includes anassert
. What sets them apart is that in subsequentrequestName
calls, themessage
variable is overridden. Themessage
variable takes on the value of the response returned by the request. This chaining is done multiple times to create a sequence of messages. -
The
requests
attribute shows how thevars
keyword is used to define a new variable calledmessage
in theRouteChat_QQEf
request. This variable is utilized in the JavaScript snippet usingvars.message
. By overriding this variable inscenario1
, you can modify the message content in the subsequent requests.
This feature allows you to create dynamic test scenarios where the output of one request influences the behavior of subsequent requests, making your testing more versatile and powerful.
Request Parameters and Headers
When making REST calls, requests often require headers, such as Basic Authentication information, and variables in the path. You can achieve this using the request object.
Example Scenario Configuration:
version: v1
requests:
- name: addCartRequest
endpointName: cart-service
methodName: cart-service-post
blob: |
{
"product_id": "OLJCESPC7Z",
"quantity": 1
}
headers:
Authorization: "Basic YWxhZGRpbjpvcGVuc2VzYW1l"
params:
- name: user_id
in: path
value: abcde
In this example:
-
Path Parameters: In the
cart-service
endpoint, if the path contains a path parameter (/cart/user_id/{user_id}
), we can define aparams
attribute and setuser_id
toabcde
. Importantly, because we setin
topath
, it’s treated as a path parameter. You can also setin
toquery
to make it a REST query parameter. -
Headers: The
headers
attribute adds a header with the keyAuthorization
and the value"Basic YWxhZGRpbjpvcGVuc2VzYW1l"
. It allows you to include headers in your requests, which is often required for authentication and other purposes.
Endpoint Configuration
The endpoint configuration file defines networking-level service details for an endpoint.
Example Endpoint Configuration:
version: v1
services:
- name: routeguide
port: 50051
alias: routeguide
protocol: grpc
endpoints:
- name: routeguide_jMBp
methods:
- name: GetFeature
- name: ListFeatures
- name: RecordRoute
- name: RouteChat
defined:
path: ./pb/route_guide.proto
name: RouteGuide
serviceName: routeguide
For configuring the endpoint file, we have a few key attributes:
-
services
: This section lists the services available in your project. In this example, there is one service namedrouteguide
. -
endpoints
: Under theendpoints
section, you define individual endpoints, specifying the available methods, the service definition path, and the service name. In the example, we have an endpoint namedrouteguide_jMBp
for theRouteGuide
service. -
methods
: Within each endpoint, you list the available methods. In this case, we have methods likeGetFeature
,ListFeatures
,RecordRoute
, andRouteChat
. This helps specify the details of each method and how it should behave. -
defined
: Here, you specify the service definition file’s path and the service name. The service definition file (route_guide.proto
) outlines the structure of the service and its methods.
By configuring endpoints, you define the available services and methods within your project, facilitating the testing of your distributed application. We recommend dynamically generating a test by providing service-level information.