Troubleshooting

This guide covers common setup problems, trace collection issues, and test generation/execution errors you might encounter.

2-Factor Authentication (2FA)

Because 2FA interrupts the login flow, automated tests cannot proceed past the verification step using the one-time code captured during trace collection. To handle this, Skyramp supports two main approaches — depending on your setup and level of integration.

Approach 1: Use a stored session (quick setup)

You can configure your test to launch from a pre-authenticated state instead of re-running the login and 2FA sequence each time.The recommended way to do this is with a wos-session storage state file, which securely stores your authenticated session. When applied, the test bypasses both login and 2FA and immediately loads the application in an authenticated state.A flow altogether and allows it to immediately load into the application in an authenticated state.

To create your own session file, use the following command below. Once the playwright browser opens, complete your Login Flow (Login + 2FA) ONLY. This will give you a session file.

Note: In order to save the session file, close the Playwright Browser that pops up, the session file will not save if you close via the terminal. You can rename session to any filename you wish.

npx playwright codegen --browser chromium --save-storage

To enable the session file, add the following to your global test configuration and comment out the initial login steps.

test.use({storageState: '<absolute path to session file>/session'

Approach 2: Generate Token via API call (recommended for integrated setups)

If your OAuth or identity provider supports programmatic token generation, Skyramp can authenticate by requesting the token directly through an API call instead of relying on a stored browser session.

During trace collection, you can send a request (for example, via curl) to your OAuth or OTP provider to retrieve a valid access token and use the response to complete the login flow. This allows the test to reproduce the full authentication sequence deterministically and stay aligned with your actual backend flow — removing the need to manually refresh or reuse sessions. Implementation depends on your provider, and the Skyramp team can assist in configuring this integration to ensure a smooth setup.

File Upload

If your test scenario involves uploading a file, Skyramp will automatically capture the step in both the trace and the generated test. In some cases, however, only the file name (not the full path) is recorded. This can cause the test to fail during execution.

To fix this and ensure the test runs out of the box, update the setFiles() function to use the absolute file path of the file you want to upload.

await fileChooser.setFiles("/Users/koljaheck/Documents/skyramp_test_file.pdf"

If your upload requires both a file selection and a separate upload confirmation, activate the Upload File option in the toolbar before selecting the file. This step ensures the file path is recorded as part of the test sequence, preventing mismatched or missing file references in the generated output.

Test Timeout

Skyramp’s runtime logic automatically identifies actions that require additional hydration. However, in some cases, a page or component may still take longer to load than expected. This may happen after operations such as submitting a form, deleting an object, or navigating to a new page.

If this happens, the test may proceed to the next step before the application is fully ready, which can result in timeouts and being unable to find the next component to navigate to. To reduce this risk, you can add a short wait before the action that fails, ensuring the application has enough time to finish loading or hydrating before moving forward.

Here’s an example of how to adjust your code:

// Additional wait for 1

Bad Selectors

Playwright’s default logic may sometimes generate selectors that are too generic or unstable, which can lead to intermittent failures if the element changes or multiple matches are found. Skyramp’s LLM runtime logic improves this by automatically searching for more reliable alternative selectors, allowing the test to continue where Playwright alone might get stuck.

But in rare cases, if no strong alternative exists or the locator remains too vague, you can ensure stability by adding explicit test IDs to your application code and recollecting traces with those attributes.

For example, Playwright might generate a fragile selector like:

// Generic selector chosen by Playwright
await page.getByRole("button").filter({ hasText: /^$/

Instead, by defining a data-testid for the element in your source code, you can improve resilience:

<button data-testid="submit-button"

And update the test to use it:

// Stable and explicit selector
await page.getByTestId("submit-button"

Network Correlation

When tests attempt to correlate backend traffic with UI interactions, it may sometimes generate overly specific or incorrect network call patterns. This can cause failures if the exact URL changes or does not match at runtime.

For example, the following network call is too specific and may not resolve correctly:

const responsePromise1 = page.waitForResponse("**/**/**/question-groups/bqg_001K5S0DH93B3MHT1J5V2GP74FV**"

You have three options to resolve this:

• Comment out the network correlation if it isn’t critical for validating your scenario:

Note: Make sure to also comment out the line where the variable is used, otherwise the test will break

// const responsePromise1 = page.waitForResponse("**/**/**/question-groups/bqg_001K5S0DH93B3MHT1J5V2GP74FV**");
await page.getByRole("button", { name: "Add Question"

• Simplify the pattern by removing the overly specific string but keeping the wildcards (**) to allow flexibility:

const responsePromise1 = page.waitForResponse("**/**/**/question-groups/**"

This ensures the test can still wait for the right type of request without being tied to a brittle or incorrect URL.

• Use include or exclude filters to improve network correlation:

If your application triggers a large number of background or unrelated network calls, you can use the --include or --exclude filters when generating a test to control which domains or endpoints are considered during correlation.

These filters don’t change the test flow itself but help Skyramp focus on the most relevant API calls, improving the accuracy of frontend-to-backend mapping and preventing noisy or irrelevant network events from causing false waits or mismatches at runtime

Generate a UI test from the collected traces in TypeScript but only include network traffic from "demoshop.skyramp.dev/*"

Duplicate Request Data Values

When you provide details for a new object in the UI (e.g., submitting a form to create a product), Skyramp captures the request body as a separate object for easy parameterization. This body should include all fields and their respective values.

Currently, Skyramp correlates the request body keys with UI selectors based on the values entered. If the same value is used for multiple fields, Skyramp may only keep one key and ignore the others. For example, if you enter “Skyramp” for name, description, and category, the generated test may only target the first matching field:

const playwrightRequest0: Record<string, any> = {
  "name": "Skyramp",
  "price": 150

We are working on improving this behavior. In the meantime, you can avoid it by using distinct values for each field when recording the trace (e.g., name: Skyramp, description: Testing platform, category: Tools). This ensures the values are properly mapped to their corresponding fields and makes the generated test easier to parameterize.

const playwrightRequest0: Record<string, any> = {
  "name": "Skyramp",
  "description": "Testing Platform",
  "category": "Tools",
  "price": 150

Trace Collection

Startup Delays

The very first time you start a trace collection, Skyramp needs to download several dependencies - including the Skyramp Docker image. This can take a bit of time, depending on your network speed. Because of this, your coding agent may time out and incorrectly report that the trace collection failed to start. In reality, the setup is still happening in the background.

If you see a timeout or failure message, simply wait a few minutes to allow everything to finish downloading.

Once the image has pulled:

1. Run Stop Trace Collection to clean up the partially initialized container.

2. Start the trace collection again — it should come up quickly and be recognized immediately.

If the issue persists even after retrying, please reach out to the Skyramp team for support.

Keyboard Interactions

When using keyboard inputs during trace collection, Playwright has limitations in capturing those interactions. As a result, when the test is generated from the trace, these keyboard actions (such as pressing Enter) may not appear in the output. This can lead to failures if the interaction was required for the flow to proceed. In some cases, the opposite can occur — the Enter key may not trigger any action during recording, prompting a manual button click instead, which causes both actions to appear in the test and leads to duplicate or conflicting steps at runtime.

To address this, manually insert the missing keyboard action in the appropriate place within your test code, specifically where the interaction occurred during the trace collection. For the duplicate steps, just comment out the keyboard action.

For example, if you pressed Enter on a button selected by role:

await page.getByRole("button", { name: "Search" }).focus();
await page.keyboard.press("Enter"

Or, if you were interacting with a text input using a locator:

await page.locator("#username").fill("skyramp_user");
await page.keyboard.press("Tab"

By adding these steps manually, you ensure the test accurately reflects the behavior you performed during trace collection and avoids failures due to missing interactions.

Snapshot Accuracy

When generating screenshots, we include a parameter called maxDiffPixelRatio, the option used in visual regression testing to define the acceptable ratio of differing pixels in a screenshot comparison.

await expect(page).toHaveScreenshot("page-001.png", { fullPage: true, maxDiffPixelRatio: 0

We default it 0.002 in our generated tests to provide a bit more flexibility during snapshot comparisons. If certain elements or pages require even more tolerance, for example, if small visual shifts occur between runs, you can increase this value as needed.

Popup Windows

Front-end trace collection can only access the DOM during test recording. Native browser UI elements—such as HTTP authentication challenges, JavaScript alert/confirm/prompt dialogs, and file selection dialogs—are not part of the DOM and therefore cannot be recorded or automatically handled during capture.

However, these dialogs can be handled programmatically during test execution by adding appropriate code to your test suite.

HTTP Authentication (401 Challenge)

When your application requires HTTP basic authentication, the browser displays a native login dialog that Playwright cannot record. Handle this by configuring credentials in your browser context fixture:

@pytest.fixture(scope="session")
def browser_context_args(browser_context_args, playwright):
    return {
        "ignore_https_errors": True,
        "service_workers": "block",
        # Credentials automatically used when receiving 401 challenge
        "http_credentials": {
            "username": "your_username", 
            "password": "your_password"

When to use: Your test encounters a 401 authentication challenge and cannot proceed.

JavaScript Alert Dialogs

Alert dialogs block test execution until dismissed. Handle them by expecting the dialog event before triggering the action:

with page.expect_event("dialog") as dlg_info:
    page.get_by_role("button", name="Show alert").click()

dialog = dlg_info.value
assert dialog.type == "alert"
assert dialog.message == "Operation completed"
dialog.accept()  # Click OK

When to use: Your application displays informational alerts that must be acknowledged.

JavaScript Confirm Dialogs

Confirm dialogs present OK/Cancel choices. You can accept or dismiss them:

with page.expect_event("dialog") as dlg_info:
    page.get_by_role("button", name="Delete").click()

dialog = dlg_info.value
assert dialog.type == "confirm"
assert "Are you sure" in dialog.message

# Choose one:
dialog.accept()   # Click OK
# dialog.dismiss() # Click Cancel

When to use: Your application requires user confirmation for destructive or significant actions.

JavaScript Prompt Dialogs

Prompt dialogs request text input from users. You can provide input or cancel:

with page.expect_event("dialog") as dlg_info:
    page.get_by_role("button", name="Enter name").click()

dialog = dlg_info.value
assert dialog.type == "prompt"
assert dialog.message == "Your name?"

# Choose one:
dialog.accept("Ami")  # Type into prompt and click OK
# dialog.dismiss()     # Click Cancel

When to use: Your application requests user input through a prompt dialog.

Beforeunload Dialogs

Beforeunload dialogs warn users before leaving a page with unsaved changes:

with page.expect_event("dialog") as dlg_info:
    page.get_by_role("link", name="Leave page").click()

dialog = dlg_info.value
assert dialog.type == "beforeunload"
dialog.accept()  # Equivalent to "Leave"

When to use: Your test needs to navigate away from a page that triggers an unsaved changes warning.

Native File Picker Dialogs

File upload dialogs can be handled by directly setting files on the input element:

# Method 1: Direct file path
page.set_input_files('input[type="file"]', '/path/to/file.png')

# Method 2: Using file chooser event
with page.expect_file_chooser() as fc:
    page.get_by_text("Upload").click()
fc.value.set_files('/path/to/file.png'

When to use: Your test needs to upload files through the browser's native file selection dialog.

Working with Localhost

When working with localhost, if you see issues with connection being refused and errors like dial tcp 192.168.65.254:4173: connect: connection refused or proxy connection errors, you may need to bind your server to 0.0.0.0.

Configure your dev server to listen on 0.0.0.0 instead of localhost. This makes it accessible to Docker containers while still working at localhost in your browser.

Vite

Add to vite.config.ts:

export default defineConfig({
  server: { host: '0.0.0.0' },
  preview: { host: '0.0.0.0'

Next.js

Update package.json:

{
  "scripts": {
    "dev": "next dev -H 0.0.0.0"
  }
}
```

**Create React App** - Add to `.env`:
```
HOST=0

Express.js

Update server code:

app.listen(3000, '0.0.0.0'

Other frameworks - Look for host, hostname, or binding configuration in your dev server settings and set it to 0.0.0.0.

Curl on MacOS Tahoe

On macOS Tahoe, users may encounter a curl/LibreSSL 3.3.6: bad decrypt error. This happens because the skyramp-worker image is based on linux/amd64, and when curl is executed inside the tracing shell on Apple Silicon, LibreSSL defaults to the TLS_CHACHA20_POLY1305_SHA256 cipher. Due to limited Rosetta support on macOS Tahoe, this cipher cannot complete the TLS handshake successfully.

To avoid this issue, we recommend explicitly specifying a cipher when running curl, such as TLS_AES_256_GCM_SHA384. Both ciphers are secure, but forcing curl to use TLS_AES_256_GCM_SHA384 ensures the handshake succeeds in this environment (Additional information).

Use the following format when running curl inside the tracing shell.

curl -k --ciphers 'TLS_AES_256_GCM_SHA384'