How to Test Services

Learn how to use Tester to write and run tests for distributed applications.

Tester simplifies writing and running tests for complicated distributed apps in four simple steps:

Create a test
- Write a test from scratch or
- Generate a test from user input
Configure the test
Run the test
Analyze test results

Prerequisites

Before using Tester, you must install the Terminal Client on your machine and the Skyramp Worker in the environment where you want to run tests.

1. Create a Test

To create a test configuration, you have two options:

Write a Test from Scratch

Use the init test command to create a template test description. This template file will require user input to supply information about the services and endpoints you would like to test.

Follow the protocol sub-commands below to generate a test template based on your API protocol: gRPC or REST.

gRPC
REST

  skyramp init test grpc <test name>

  skyramp init test rest <test name>

Generate a Test from User Input

Use the generate command to generate a default test configuration to use and start with. These tests will work out-of-the-box but are intended to be a starting point for more complicated testing logic in different applications.

Follow the protocol sub-commands below to generate configurations based on your API protocol: gRPC or REST.

gRPC
REST

  skyramp tester generate grpc \
    --api-schema <path to .proto file> \
    --alias <name Docker alias to test> \
    --port <port number for the service> \
    --service <name of proto service>

  skyramp tester generate rest \
    --api-schema <path to OpenAPI schema file or URL (URL support for OpenAPI 3.x only)> \
    --alias <name of Docker alias to test> \
    --port <port number for the service> \
    --tag <optional OpenAPI tag to filter on> \
    --paths <optional REST paths to filter on>  \
    --crud-scenarios <enable generating CRUD (Create, Read, Update, Delete) scenarios for REST endpoints>

This command performs the following actions:

Creates a test configuration file in the tests folder for the specified service or alias.
Creates a scenario configuration file in the scenarios folder for each method defined in the service.
If the endpoint definition does not already exist, it creates an endpoint configuration file in the endpoints folder.

You can edit the generated .yaml files as explained in the next section.

Tip

We recommend running the generate command in your source code repo so tests are versioned and shareable across your team.

2. Configure the Test

The test description created with the init test command serves as a template for a test. It prompts you to fill in the necessary information to configure specific details such as the endpoint, scenario, and overall test configuration.

On the other hand, the test description generated using the tester generate command is designed to be ready-to-use out-of-the-box. Despite its immediate usability, you still retain the flexibility to add or modify configuration details. This allows you to customize and enhance your tests according to your specific requirements.

Below are examples of endpoint, scenario, and test configurations:

Endpoint Configuration

Endpoint configuration files can be found in the endpoints folder. Here’s an example of an endpoint configuration for a gRPC service:

version: v1
services:
    - name: routeguide
      port: 50051
      alias: routeguide
      protocol: grpc
endpoints:
    - name: routeguide_jMBp
      methods:
        - name: GetFeature
        - name: ListFeatures
        - name: RecordRoute
        - name: RouteChat
      defined:
        path: ./pb/route_guide.proto
        name: RouteGuide
      serviceName: routeguide

The generated endpoint configuration contains networking-level service details, including its name, port, alias, and protocol. Additionally, it contains metadata related to various endpoints that a service can have, including the methods it supports and the associated proto file.

Scenario Configuration

To configure scenarios, edit the scenario configuration file in the scenarios folder. Here’s an example of a scenario configuration for a gRPC service:

version: v1
requests:
    - name: GetFeature_18GO
      blob: |-
        {
          "latitude": 0,
          "longitude": 0
        }        
      endpointName: routeguide_jMBp
      methodName: GetFeature
    - name: ListFeatures_5P9V
      blob: |-
        {
          "hi": {
            "latitude": 0,
            "longitude": 0
          },
          "lo": {
            "latitude": 0,
            "longitude": 0
          }
        }        
      endpointName: routeguide_jMBp
      methodName: ListFeatures
    # More request configurations...

In this example, you define request behavior for a specific method of the service and specify the endpoint and method name as defined in the endpoint configuration. You can customize the request payload using a static JSON blob. To create more complex requests, you can add advanced capabilities such as defining dynamic requests and adding overrides and parameters.

The scenario file generated using tester generate by default only lists the aforementioned requests. To add advanced capabilities like assertions and chaining, you can create additional scenario files in the scenarios folder. These files can reference the generated requests and provide more complex test scenarios.

You can read about all advanced capabilities in the Test Description page.

Test Configuration

To configure the overall test behavior, edit the test configuration file in the tests folder. Here’s an example of a test configuration for a gRPC service:

version: v1
test:
    name: routeguide
    testPattern:
        - requestName: GetFeature_18GO
          startAt: 1
        - requestName: ListFeatures_5P9V
          startAt: 1
        - requestName: RecordRoute_MD5D
          startAt: 1
        - requestName: RouteChat_QQEf
          startAt: 1

In this example, you configure the top-level test’s behavior, including setting a name for the test and specifying the test pattern by referencing requests and scenarios as defined in the scenario configuration. Advanced capabilities like configuring load testing available in the Test Description page.

3. Run the Test

Once the test description is in place, you can run the test. To run the test, you first need to reference the specific test file that will be used. To do this, refer to the tests directory. The name of the test will be the name of the respective file containing the test, without the file extension. For example, if you have a file located at tests/checkout-test.yaml, the test file name will be checkout-test.

  skyramp tester start <test file name> \
    -a <ip:port of the Skyramp worker>

After running the command above to start the test, there will be output on the progress/status of the test until it finishes. Note that if you interrupt the command (such as by sending a SIGINT), the test will continue to run in the background in the Skyramp worker. For an ongoing test, you can check the status by running skyramp tester status <test file name> and providing either the address or namespace, as previously described.

Stop the Test

To stop an ongoing test, run skyramp tester stop <test file name>, once again with either the address or namespace arguments.

Learn more about what you can configure in your test description on the Test Description page.

4. Analyze Test Results

There are several ways to view and retain test results:

Generate an HTML test report by running skyramp tester start with the optional --html-report flag. The report is saved in the “results” directory of your working directory.
Start a Skyramp dashboard by running skyramp dashboard up. Refer to the dashboard documentation to learn more about managing test results.