QA Testing

Previewops can run an automated QA analysis on any pull request with a single comment. You get a ready-to-read report posted directly in the PR — no CI configuration required.

Quick start

Run on demand

/qa-previewops

Previewops analyses the PR and posts a report. If a preview environment is already live for this PR, HTTP smoke tests run against it automatically.

Run against a specific URL

/qa-previewops --url=https://pr-42-preview.example.com

Use --url to target a preview that was deployed outside of Previewops, or to override the auto-detected URL.

What's in the report

The report is always posted as a PR comment and is organised into sections:

📁 Test coverage

Available on all plans — no AI key required.

Previewops inspects up to 300 changed files and maps each source file to its expected test counterpart. The result is a table showing which files are covered (a test file was found) or uncovered (no test file could be matched).

Supported languages:

Language Convention used
TypeScript / JavaScript src/X/Y.ts → src/__tests__/X/Y.test.ts or .spec.ts
Python auth.py → test_auth.py or auth_test.py
Go handler.go → handler_test.go (same directory)
Ruby auth.rb → auth_spec.rb or auth_test.rb
Java Auth.java → AuthTest.java or AuthTests.java
Playwright .spec.ts / .spec.js files are recognised as test files
Cypress .cy.ts / .cy.js files are recognised as test files

A fuzzy fallback handles cases where the test file name doesn't exactly mirror the source file name — for example, provider subdirectories (aws-ecs/index.ts matched to ecs.test.ts) or cross-root test layouts (Java src/main/ ↔ src/test/, Ruby app/ ↔ spec/).

Files that are not mappable — configuration files, migrations, Markdown, JSON — are silently excluded from the coverage table.

📝 Code review

Available on Pro, Enterprise, and Custom plans, and on Premium BYOC with the Custom LLM add-on activated.

An AI model reviews the full PR diff and produces:

The review covers any language in the diff — not just TypeScript.

🧪 HTTP smoke tests

Available on Pro, Enterprise, and Custom plans, and on Premium BYOC with the Custom LLM add-on activated. Requires a live preview URL.

The AI generates 3–6 HTTP test scenarios based on the PR diff, then executes them against the preview. Each scenario specifies a method, path, and expected status code. Results are shown in a pass/fail table.

If no preview URL is available (neither via --url nor an active preview for this PR), the HTTP test section is omitted.

⚠️ Regression risks

Available on Pro, Enterprise, and Custom plans, and on Premium BYOC with the Custom LLM add-on activated.

The AI identifies areas of the codebase that may regress based on what the PR touches — for example, auth changes raising risk for all authenticated routes. Each risk is rated high / medium / low with a brief explanation.

Plan comparison

Feature Free Premium BYOC Pro Enterprise Custom
Test coverage mapping ✅ ✅ ✅ ✅ ✅
AI code review ❌ ✅ ¹ ✅ ✅ ✅
HTTP smoke tests ❌ ✅ ¹ ✅ ✅ ✅
Regression risk analysis ❌ ✅ ¹ ✅ ✅ ✅

¹ Premium BYOC AI features require the Custom LLM add-on. Activate it in the dashboard and store your LLM_PROVIDER, LLM_API_KEY, and LLM_MODEL credentials. Without it, the command runs and returns test coverage, but AI analysis is skipped and an add-on activation prompt is shown in the report. See add-ons.md.

Free plan users will see an upgrade prompt in the report where the AI sections would appear.

Example report

> 🤖 QA Report for PR #42

### 📁 Test Coverage

| File | Status |
|---|---|
| `src/utils/auth.ts` | ✅ covered |
| `src/utils/rate-limit.ts` | ✅ covered |
| `src/deployer/index.ts` | ❌ uncovered |

**2/3** source files have corresponding test files.

### 📝 Code Review

The changes introduce a new rate-limiting layer in the auth utility and refactor
the deploy orchestrator to support concurrent provider calls.

| Severity | Issue |
|---|---|
| 🟡 medium | `rateLimiter` is not reset between test runs — consider clearing state in teardown. |
| 🟢 low | Minor: variable name `tmp` in `index.ts` could be more descriptive. |

### 🧪 HTTP Test Scenarios

| Scenario | Status | Details |
|---|---|---|
| GET /api/health | ✅ pass | HTTP 200 |
| POST /api/deploy | ✅ pass | HTTP 202 |
| GET /api/unknown-route | ❌ fail | HTTP 404 (expected 404) |

### ⚠️ Regression Risks

| Area | Risk | Reason |
|---|---|---|
| Auth middleware | 🟡 medium | Rate-limiting logic now wraps all authenticated routes. |
| Deploy orchestrator | 🔴 high | Concurrent provider path is new and lacks tests. |

_Testing against: https://pr-42-preview.fly.dev_
_Analyzed with: gpt-4o_

Options

Option Type Example Description
--url https://... --url=https://pr-42.example.com Explicitly set the preview URL for HTTP smoke tests. When omitted, Previewops auto-detects the active preview for this PR. Private/internal IP addresses are always rejected.

Who can trigger the command

Any collaborator with Write, Maintain, or Admin permission on the repository. Comments from users with read-only access are silently ignored.

Frequently asked questions

Does QA testing require a deployed preview?

No. The test coverage mapping and AI code review run on the PR diff — no live URL needed. HTTP smoke tests require a preview URL, but if one isn't available that section is simply omitted.

Which AI model is used?

This depends on how the operator has configured the Previewops instance. The model name is shown in the footer of every report (_Analyzed with: gpt-4o_). Contact your team's Previewops administrator or see plans.md to learn more.

Why is my file shown as uncovered when I know a test exists?

The mapping uses the test files that exist at the PR head. If your test file was added in the same PR and is already committed, it should be detected. If the test file uses an unusual naming convention or lives in an unexpected location, it may fall through the fuzzy fallback. Primary mapping conventions are listed in the Test coverage table above.

What happens if the AI is temporarily unavailable?

Previewops retries the AI call once before falling back. If both attempts fail, the code review section shows a short error message and the rest of the report (coverage, HTTP tests if applicable) is still posted.

Is the diff sent to a third-party AI service?

Yes — when AI features are enabled, the PR diff is sent to the configured LLM provider (OpenAI or Anthropic). File patches are truncated to 500 characters each, and no repository access tokens or credentials are included in the payload. Self-hosted operators control which provider is used. Contact your Previewops administrator for details.