QitOps CLI Tools

QitOps is a comprehensive Software Quality Assurance CLI tool for API, Performance, Security, and Web Testing. It provides a unified command-line interface with minimal dependencies and maximum flexibility.

Quick Start

# Install QitOps
cargo install --path .

# Run a basic API test
qitops api -c tests/configs/api_test.json

# Run an API collection (multiple requests with dependencies)
qitops collection -c tests/configs/api_collection.json

# Run a performance test
qitops performance -c tests/configs/performance_test.json -u 10 -d 30

# Run a security scan
qitops security -c tests/configs/security_test.json -d 3

# Run a web test
qitops web -c tests/configs/web_test.json

# Generate a report in HTML format
qitops -r html -o report.html api -c tests/configs/api_test.json

# Run in CI mode (reduced output, exit code based on test results)
qitops --ci-mode -r json -o results.json api -c tests/configs/api_test.json

# Run data-driven tests with CSV data
qitops data-driven -c tests/configs/data_driven_api_test.json -d tests/data/users.csv -t csv

# Run data-driven tests with JSON data
qitops data-driven -c tests/configs/data_driven_collection.json -d tests/data/products.json -t json

Features

API Testing

• HTTP method support (GET, POST, PUT, DELETE, etc.)
• URL configuration with environment-specific settings
• Custom headers and request body support
• Response validation (status codes, body, headers)
• Response time monitoring
• Configurable timeouts and retries
• Retry mechanism with exponential backoff and jitter
  • Configurable retry attempts
  • Customizable retry delay
  • Status code-based retry conditions
  • Connection error handling
  • Timeout handling

API Collections

• Group related API requests in a single configuration
• Define dependencies between requests
• Capture and use data from previous responses using JSONPath
• Variable interpolation with {{variable}} syntax
• Environment variables and environment-specific configurations
• Sequential request execution with dependency management
• Shared authentication across requests (Basic, Bearer, API Key)
• Default request configuration (headers, timeout, retries)
• Detailed collection reporting with captured variables
• Request chaining for complex workflows

Performance Testing

• Load testing with configurable concurrent users
• Response time analysis
• Success rate monitoring
• Ramp-up time configuration
• Detailed performance metrics
  • Average response time
  • Minimum response time
  • Maximum response time
  • Total requests
  • Success/error counts
  • Success rate percentage

Enhanced Performance Testing

• Multiple load profiles (constant, ramping, spike)
• Multi-stage test execution
• Multiple scenarios in a single test
• Weighted scenario distribution
• Detailed metrics collection and reporting
• Percentile calculations (p50, p90, p95, p99)
• Custom thresholds with pass/fail criteria
• Real-time metrics streaming
• Tagged metrics for detailed analysis
• Scenario-based reporting

CI/CD Integration

• GitHub Actions workflow templates
• CI mode with reduced output
• Exit codes based on test results
• JSON report generation for CI pipelines
• Parallel test execution
• Environment-specific configurations
• Scheduled test runs
• Artifact storage for test results and reports

Data-Driven Testing

• Parameterize tests with CSV and JSON datasets
• Variable interpolation with {{placeholder}} syntax
• Support for multiple iterations of the same test
• Configurable iteration limits
• Stop-on-failure option
• Detailed iteration reporting
• Support for all test types (API, Performance, Security, Web)
• CSV header row support
• JSON path extraction
• Inline data definition
• Customizable success thresholds

Security Testing

• Comprehensive security scanning
• Multiple scan types (headers, SSL, vulnerabilities, sensitive data)
• Severity-based reporting
• Authentication testing
• Common vulnerability checks
• Security header validation
• CSRF and XSS detection
• SQL injection testing
• JWT security analysis
• Access control verification

Web Testing

• Headless browser automation
• Viewport configuration
• Screenshot capture
• Element assertions
• Text content validation
• URL and title validation
• Action simulation (click, type, wait, navigate)
• Custom user agent configuration

AI-Powered Features

• Test configuration generation from natural language descriptions
• Test results analysis with insights and patterns
• Improvement suggestions based on test results
• Support for local LLM models (LLaMA, Mistral, GPT-J, Phi)
• Customizable model parameters (temperature, context size, etc.)
• Offline operation with no data sent to external services

Installation

From crates.io (Recommended)

# Install directly from crates.io
cargo install qitops

# Run QitOps
qitops --help

From GitHub Releases

1. Download the latest binary for your platform from the GitHub Releases page
2. Make the file executable (Linux/macOS): chmod +x qitops-*
3. Move it to a directory in your PATH:
   • Linux/macOS: sudo mv qitops-* /usr/local/bin/qitops
   • Windows: Add the directory containing the executable to your PATH

Using Docker

# Pull the Docker image
docker pull qitops/qitops:latest

# Run QitOps
docker run --rm qitops/qitops:latest --help

# Run with mounted volumes for configs and results
docker run --rm -v $(pwd)/configs:/workspace/configs -v $(pwd)/results:/workspace/results qitops/qitops:latest api -c /workspace/configs/api_test.json

From Source

# Clone the repository
git clone https://github.com/qitops/qitops-cli-tools.git
cd qitops-cli-tools

# Build the project
cargo build --release

# Install the binary (optional)
cargo install --path .

With AI Features (Optional)

# Install with AI features enabled
cargo install qitops --features ai

Configuration

API Test Configuration

{
    "name": "Example API Test",
    "description": "Test description",
    "timeout": 30,
    "retries": 3,
    "environment": "production",
    "url": "https://api.example.com",
    "method": "GET",
    "headers": {
        "Accept": "application/json",
        "User-Agent": "QitOps-Test"
    },
    "expected_status": 200,
    "expected_body": {
        "field1": "value1",
        "field2": "value2"
    },
    "max_response_time": 2,
    "expected_headers": {
        "content-type": "application/json",
        "cache-control": "no-cache"
    },
    "retry": {
        "max_retries": 3,
        "initial_delay_ms": 100,
        "max_delay_ms": 1000,
        "retry_status_codes": [408, 429, 500, 502, 503, 504],
        "retry_on_timeout": true,
        "retry_on_connection_error": true
    }
}

API Collection Configuration

{
    "name": "GitHub API Collection",
    "description": "A collection of GitHub API tests",
    "version": "1.0.0",
    "variables": {
        "base_url": "https://api.github.com",
        "username": "octocat",
        "repo": "Hello-World"
    },
    "auth": {
        "type": "bearer",
        "token": "{{GITHUB_TOKEN}}"
    },
    "defaults": {
        "headers": {
            "Accept": "application/vnd.github.v3+json",
            "User-Agent": "QitOps-Test"
        },
        "timeout": 30,
        "retries": 3
    },
    "requests": [
        {
            "name": "Get User",
            "description": "Get a GitHub user",
            "id": "get-user",
            "url": "{{base_url}}/users/{{username}}",
            "method": "GET",
            "expected_status": 200,
            "expected_body": {
                "login": "{{username}}",
                "type": "User"
            },
            "capture": {
                "user_id": "$.id",
                "user_url": "$.url"
            }
        },
        {
            "name": "Get User Repos",
            "description": "Get repositories for a user",
            "id": "get-user-repos",
            "url": "{{user_url}}/repos",
            "method": "GET",
            "depends_on": ["get-user"],
            "expected_status": 200
        }
    ],
    "environments": {
        "production": {
            "base_url": "https://api.github.com"
        },
        "staging": {
            "base_url": "https://api.staging.github.com"
        }
    },
    "run_options": {
        "sequential": true,
        "stop_on_failure": true,
        "delay_between_requests_ms": 500
    }
}

Performance Test Configuration

{
    "name": "Sample Performance Test",
    "description": "Load testing a public API endpoint",
    "timeout": 30,
    "retries": 3,
    "environment": "production",
    "target_url": "https://api.example.com/endpoint",
    "method": "GET",
    "headers": {
        "Accept": "application/json"
    },
    "success_threshold": 95.0,
    "ramp_up_time_secs": 5
}

Enhanced Performance Test Configuration

{
    "name": "Enhanced Performance Test",
    "description": "Testing API performance with multiple scenarios and load profiles",
    "timeout": 60,
    "retries": 0,
    "environment": "production",
    "load_profile": {
        "type": "ramping_vus",
        "initial": 1,
        "stages": [
            {
                "duration_secs": 30,
                "target": 10
            },
            {
                "duration_secs": 60,
                "target": 20
            },
            {
                "duration_secs": 30,
                "target": 0
            }
        ]
    },
    "scenarios": [
        {
            "name": "Get Request",
            "target_url": "https://httpbin.org/get",
            "method": "GET",
            "headers": {
                "Accept": "application/json",
                "User-Agent": "QitOps-Test/1.0"
            },
            "weight": 3,
            "tags": {
                "endpoint": "get",
                "category": "read"
            }
        },
        {
            "name": "Post Request",
            "target_url": "https://httpbin.org/post",
            "method": "POST",
            "headers": {
                "Content-Type": "application/json",
                "Accept": "application/json"
            },
            "body": {
                "test": "data",
                "number": 123
            },
            "weight": 1,
            "tags": {
                "endpoint": "post",
                "category": "write"
            }
        }
    ],
    "thresholds": [
        {
            "metric": "response_time.avg",
            "expression": "< 0.5",
            "abort_on_fail": false
        },
        {
            "metric": "response_time.p95",
            "expression": "< 1.0",
            "abort_on_fail": false
        },
        {
            "metric": "success.avg",
            "expression": "> 0.95",
            "abort_on_fail": true
        }
    ],
    "success_threshold": 95.0,
    "stream_metrics": true,
    "metrics_interval_secs": 5
}

Security Test Configuration

{
    "name": "Security Scan",
    "description": "Comprehensive security scan of the API",
    "timeout": 30,
    "retries": 3,
    "environment": "production",
    "target_url": "https://api.example.com",
    "headers": {
        "Accept": "application/json"
    },
    "auth": {
        "type": "bearer",
        "token": "your-token"
    },
    "scan_types": [
        "headers",
        "ssl",
        "vulnerabilities",
        "sensitive-data"
    ],
    "max_high_severity_findings": 0
}

Web Test Configuration

{
    "name": "Sample Web Test",
    "description": "Testing a public website",
    "timeout": 30,
    "retries": 3,
    "environment": "production",
    "target_url": "https://example.com",
    "viewport": {
        "width": 1280,
        "height": 800,
        "device_scale_factor": 1.0,
        "is_mobile": false
    },
    "wait_for_selector": "body",
    "wait_timeout_secs": 10,
    "screenshots": true,
    "user_agent": "QitOps-WebTester/1.0",
    "assertions": [
        {
            "assertion_type": "title",
            "expected_value": "Example Domain",
            "comparison": "contains"
        },
        {
            "assertion_type": "element",
            "selector": "h1",
            "expected_value": "true"
        }
    ],
    "actions": [
        {
            "action_type": "wait",
            "wait_time_ms": 1000
        },
        {
            "action_type": "click",
            "selector": "a"
        }
    ]
}

AI Configuration

{
    "model_type": "llama",
    "model_path": "/usr/local/share/models/llama-2-7b-chat.gguf",
    "context_size": 4096,
    "temperature": 0.7,
    "max_tokens": 2048,
    "system_prompt": "You are an AI assistant specialized in software testing. Your task is to help generate test configurations, analyze test results, and suggest improvements."
}

Usage

API Testing

# Run a single API test
qitops api -c tests/configs/api_test.json

# Run tests in a specific environment
qitops api -c tests/configs/api_test.json -e production

Example output:

Test Results:
Name: Sample API Test
Status: passed
Duration: 0.90s
Details: {
  "headers": {
    "content-type": "application/json",
    "cache-control": "no-cache",
    ...
  },
  "response_time": 0.903123787,
  "status_code": 200
}
Timestamp: 2025-05-09T21:06:33.923402733+00:00

API Collections

# Run an API collection
qitops collection -c tests/configs/api_collection.json

# Run in a specific environment
qitops collection -c tests/configs/api_collection.json -e staging

# Output in JSON format
qitops collection -c tests/configs/api_collection.json -f json

Example output:

Collection Results:
Name: HTTPBin API Collection
Status: passed
Duration: 2.35s
Timestamp: 2025-05-09T21:07:05.165804274+00:00

Request Results:
  1. Get IP Address - passed
  2. Post with JSON - passed
  3. Get with Headers - passed
  4. Get with Query Parameters - passed

Captured Variables:
  client_ip: 203.0.113.1
  request_id: 97e5b974-e2c3-4073-b45e-5bf5a7f3f0b2
  posted_data: {"test_value":"qitops_test_value","client_ip":"203.0.113.1"}

The API collections feature supports:

• Variable interpolation using {{variable}} syntax
• Capturing data from responses using JSONPath expressions
• Request dependencies to ensure proper execution order
• Environment-specific configurations
• Shared authentication and default headers

Performance Testing

# Run performance test with default settings
qitops performance -c tests/configs/performance_test.json

# Run with custom concurrent users and duration
qitops performance -c tests/configs/performance_test.json -u 50 -d 120

Example output:

Performance Test Results:
Name: Sample Performance Test
Status: passed
Duration: 10.04s
Details: {
  "average_response_time": 0.11007347446511631,
  "error_count": 0,
  "max_response_time": 0.521383212,
  "min_response_time": 0.057222978,
  "success_count": 215,
  "success_rate": 100.0,
  "total_requests": 215
}
Timestamp: 2025-05-09T21:06:50.438713024+00:00

Enhanced Performance Testing

# Run enhanced performance test with load profiles and scenarios
qitops performance-enhanced -c tests/configs/enhanced_performance_test.json

# Run in a specific environment
qitops performance-enhanced -c tests/configs/enhanced_performance_test.json -e staging

Example output:

Enhanced Performance Test Results:
Name: Enhanced Performance Test
Status: passed
Duration: 120.35s

Metrics Summary:
  Total Requests: 1250
  Success Count: 1245
  Error Count: 5
  Success Rate: 99.60%

Response Time:
  Average: 125.32ms
  Min: 57.89ms
  Max: 521.45ms
  p50: 115.67ms
  p90: 198.34ms
  p95: 245.78ms
  p99: 378.91ms

Scenario Results:
  Get Request: 937/940 requests successful (99.68%)
  Post Request: 308/310 requests successful (99.35%)

Thresholds:
  response_time.avg: < 0.5 - PASSED
  response_time.p95: < 1.0 - PASSED
  success.avg: > 0.95 - PASSED

For full details, use the --report option to generate a JSON report.
Timestamp: 2025-05-09T21:08:15.723654912+00:00

The enhanced performance testing feature supports:

• Multiple load profiles (constant, ramping, spike)
• Multi-stage test execution
• Multiple scenarios with weighted distribution
• Custom thresholds with pass/fail criteria
• Real-time metrics streaming
• Detailed metrics with percentiles

Data-Driven Testing

# Run data-driven API tests with CSV data
qitops data-driven -c tests/configs/data_driven_api_test.json -d tests/data/users.csv -t csv

# Run with JSON data
qitops data-driven -c tests/configs/data_driven_collection.json -d tests/data/products.json -t json

# Limit the number of iterations
qitops data-driven -c tests/configs/data_driven_api_test.json -d tests/data/users.csv -m 3

# Stop on first failure
qitops data-driven -c tests/configs/data_driven_api_test.json -d tests/data/users.csv -s

Example output:

Iteration 1 Results:
Name: User API Test for johndoe
Status: passed
Duration: 0.52s
Details: {
  "headers": {
    "content-type": "application/json",
    ...
  },
  "response_time": 0.523456,
  "status_code": 200
}
Timestamp: 2025-05-09T21:07:05.165804274+00:00

Data Row:
  username: johndoe
  email: john.doe@example.com
  user_id: 1001
  role: admin

...

Data-Driven Test Summary:
Total Iterations: 5
Successful: 5
Failed: 0
Success Rate: 100.00%

The data-driven testing feature supports:

• Parameterizing tests with CSV and JSON datasets
• Variable interpolation using {{placeholder}} syntax
• Running multiple iterations of the same test with different data
• Configurable iteration limits and stop-on-failure options
• Detailed reporting for each iteration

Security Testing

# Run security scan with default settings
qitops security -c tests/configs/security_test.json

# Run with custom scan depth and passive scanning
qitops security -c tests/configs/security_test.json -d 4 -p

Example output:

Security Test Results:
Name: API Security Test
Status: passed
Duration: 0.00s
Details: {
  "findings": [],
  "summary": {
    "critical_findings": 0,
    "high_findings": 0,
    "low_findings": 0,
    "medium_findings": 0,
    "total_findings": 0
  }
}
Timestamp: 2025-05-09T21:07:05.165804274+00:00

Web Testing

# Run web test with default settings
qitops web -c tests/configs/web_test.json

# Run with headless mode disabled and custom screenshot directory
qitops web -c tests/configs/web_test.json -h false -s ./screenshots

Example output:

Web Test Results:
Name: Sample Web Test
Status: passed
Duration: 1.25s
Details: {
  "action_results": [
    {
      "duration_ms": 1000,
      "success": true,
      "type": "wait"
    },
    {
      "selector": "a",
      "success": true,
      "type": "click"
    }
  ],
  "assertion_results": [
    {
      "details": "Title: Simulated Page Title",
      "passed": true,
      "type": "title"
    },
    {
      "details": "Element with selector 'h1' exists: true",
      "passed": true,
      "type": "element"
    }
  ],
  "content_length": 1256,
  "page_title": "Simulated Page Title",
  "screenshot": "./screenshots/screenshot_1715284025.png",
  "status_code": 200
}
Timestamp: 2025-05-09T21:07:05.165804274+00:00

Output Formats

QitOps provides both human-readable and machine-readable output formats to support both interactive use and CI integration:

# Default human-readable output to stdout
qitops api -c tests/configs/api_test.json

# Generate machine-readable JSON report for CI integration
qitops -r json -o report.json api -c tests/configs/api_test.json

# Generate XML report (JUnit format for CI integration)
qitops -r xml -o report.xml security -c tests/configs/security_test.json

# Generate HTML report for visual inspection
qitops -r html -o report.html performance -c tests/configs/performance_test.json

# Generate CSV report for data analysis
qitops -r csv -o report.csv web -c tests/configs/web_test.json

All outputs include consistent timestamping for audit trails and traceability.

AI-Powered Features

# Generate an API test configuration from a description
qitops generate -t api -d "Test the GitHub API to fetch user information" -o tests/configs/github_api_test.json

# Generate a web test configuration from a description
qitops generate -t web -d "Test the login form on example.com with valid credentials" -o tests/configs/login_test.json

# Analyze test results
qitops analyze -r results/api_test_result.json -o analysis.md

# Get improvement suggestions based on test results
qitops improve -r results/performance_test_result.json -o improvements.md

# Use a specific AI model
qitops generate -t security -d "Test for SQL injection vulnerabilities" -o tests/configs/sql_injection_test.json -m mistral

# Use a custom model
qitops analyze -r results/security_test_result.json -o analysis.md -m custom -p /path/to/custom/model.gguf

Command Line Options

Global Options

• -r, --report: Generate report in specified format (json, xml, html, csv)
• -o, --output: Output path for the report
• --ci-mode: Run in CI mode (reduced output, exit code based on test results)

API Testing

• -c, --config: Path to the test configuration file
• -e, --environment: Environment to run tests in (default: "production")

API Collections

• -c, --config: Path to the collection configuration file
• -e, --environment: Environment to run tests in (default: "production")
• -f, --format: Output format (human, json) (default: "human")

Performance Testing

• -c, --config: Path to the test configuration file
• -e, --environment: Environment to run tests in (default: "production")
• -u, --users: Number of concurrent users (default: 10)
• -d, --duration: Test duration in seconds (default: 60)

Enhanced Performance Testing

• -c, --config: Path to the test configuration file
• -e, --environment: Environment to run tests in (default: "production")

Data-Driven Testing

• -c, --config: Path to the test configuration file
• -d, --data: Path to the data source file (CSV or JSON)
• -t, --data-type: Data source type (csv, json) (default: "csv")
• -e, --environment: Environment to run tests in (default: "production")
• -m, --max-iterations: Maximum number of iterations to run
• -s, --stop-on-failure: Stop on first failure

Security Testing

• -c, --config: Path to the test configuration file
• -e, --environment: Environment to run tests in (default: "production")
• -d, --depth: Scan depth (1-5, default: 3)
  • Level 1: Basic security checks (headers, SSL)
  • Level 2: Common vulnerabilities
  • Level 3: Authentication and authorization
  • Level 4: Advanced vulnerability scanning
  • Level 5: Comprehensive security audit
• -p, --passive: Include passive scanning

Web Testing

• -c, --config: Path to the test configuration file
• -e, --environment: Environment to run tests in (default: "production")
• -h, --headless: Run in headless mode (default: true)
• -s, --screenshot_dir: Directory to save screenshots

AI Test Generation

• -t, --test_type: Type of test to generate (api, performance, security, web)
• -d, --description: Description of the test to generate
• -o, --output: Output file path for the generated configuration
• -m, --model: AI model to use (llama, mistral, gptj, phi, custom)
• -p, --model_path: Path to model weights (required for custom models)

AI Test Analysis

• -r, --results: Path to test results file(s)
• -o, --output: Output file path for the analysis
• -m, --model: AI model to use (llama, mistral, gptj, phi, custom)
• -p, --model_path: Path to model weights (required for custom models)

AI Improvement Suggestions

• -r, --results: Path to test results file(s)
• -o, --output: Output file path for the suggestions
• -m, --model: AI model to use (llama, mistral, gptj, phi, custom)
• -p, --model_path: Path to model weights (required for custom models)

Test Results

API Test Results

• Test name and status
• Duration
• Response details:
  • Status code
  • Response time
  • Headers
  • Retry attempts (if any)
• Timestamp
• Environment information

API Collection Results

• Collection name and status
• Overall duration
• Individual request results:
  • Request name
  • Status
  • Duration
  • Response details
• Captured variables
• Timestamp
• Environment information

Performance Test Results

• Test name and status
• Duration
• Performance metrics:
  • Total requests
  • Success/error counts
  • Success rate
  • Average response time
  • Minimum response time
  • Maximum response time
• Timestamp
• Environment information

Enhanced Performance Test Results

• Test name and status
• Duration
• Detailed metrics:
  • Total requests
  • Success/error counts
  • Success rate
  • Response time statistics (avg, min, max)
  • Percentile measurements (p50, p90, p95, p99)
• Scenario-specific metrics:
  • Per-scenario success rates
  • Per-scenario request counts
• Threshold evaluations:
  • Metric name
  • Expression
  • Pass/fail status
• Tagged metrics for detailed analysis
• Timestamp
• Environment information

Security Test Results

• Test name and status
• Duration
• Security findings:
  • Critical findings
  • High severity findings
  • Medium severity findings
  • Low severity findings
  • Total findings
• Detailed findings with:
  • Severity level
  • Category
  • Description
  • Recommendation
• Timestamp
• Environment information

Web Test Results

• Test name and status
• Duration
• Page information:
  • Page title
  • Status code
  • Content length
• Assertion results:
  • Type (title, url, element, text)
  • Pass/fail status
  • Details
• Action results:
  • Type (click, type, wait, navigate)
  • Success status
  • Details
• Screenshot path (if enabled)
• Timestamp
• Environment information

Best Practices

API Testing

• Use environment-specific configurations for different deployment stages
• Set appropriate timeouts based on your API's expected response times
• Configure retry mechanisms for transient failures
• Validate both success and error responses
• Use JSON Schema validation for complex response structures
• Monitor response times to catch performance degradation

API Collections

• Organize related requests into logical collections
• Use meaningful request IDs for better readability and dependencies
• Leverage JSONPath for precise data extraction from responses
• Use variable capture and reuse for efficient testing
• Define environment-specific variables for different deployment stages
• Use sequential execution for dependent requests
• Set appropriate delays between requests to avoid rate limiting
• Define shared authentication at the collection level
• Use defaults for common configuration across requests
• Structure collections to follow user journeys or business processes
• Use request dependencies to ensure proper execution order
• Keep collections focused on a specific testing goal

Performance Testing

• Start with a small number of concurrent users and gradually increase
• Use appropriate ramp-up times to avoid overwhelming the system
• Set realistic success thresholds based on your requirements
• Monitor system resources during tests
• Run tests in a controlled environment
• Consider network latency in your test environment

Enhanced Performance Testing

• Design multi-stage tests to simulate realistic user behavior
• Use different load profiles for different testing scenarios
• Define multiple scenarios to test different API endpoints
• Use weighted scenarios to simulate real-world usage patterns
• Set appropriate thresholds based on SLAs and performance requirements
• Use tags to categorize and analyze metrics
• Monitor real-time metrics during test execution
• Analyze percentile measurements for a better understanding of performance
• Use custom thresholds to catch performance regressions early
• Run tests in different environments to compare performance

Data-Driven Testing

• Organize test data in CSV or JSON format based on complexity
• Use CSV for simple tabular data with a consistent structure
• Use JSON for complex nested data structures
• Include a header row in CSV files for better readability
• Use meaningful placeholder names that match data column names
• Keep test configurations generic with placeholders
• Use stop-on-failure for dependent test iterations
• Set appropriate max iterations for large datasets
• Validate data files before running tests
• Use environment-specific configurations with data-driven tests
• Combine data-driven testing with API collections for complex workflows

Security Testing

• Start with passive scanning before active scanning
• Use appropriate scan depth based on your security requirements
• Review and address high-severity findings immediately
• Keep authentication tokens secure
• Run security tests in a controlled environment
• Regularly update security test configurations

Web Testing

• Use headless mode for CI/CD pipelines
• Configure appropriate viewport sizes for different devices
• Keep assertions focused and specific
• Use explicit waits for dynamic content
• Capture screenshots for visual verification
• Test across different browsers in production
• Organize tests by user journey or feature
• Use custom user agents when needed

AI-Powered Testing

• Use specific, detailed descriptions when generating tests
• Review and refine AI-generated test configurations before use
• Provide context in your descriptions (e.g., authentication requirements)
• Use lower temperature values (0.1-0.3) for more deterministic outputs
• Use higher temperature values (0.7-0.9) for more creative test scenarios
• Analyze test results regularly to identify patterns and issues
• Combine AI suggestions with human expertise for best results
• Keep model weights updated for best performance

CI/CD Integration

• Use --ci-mode for reduced output and exit codes in CI pipelines
• Generate JSON reports for machine-readable results
• Use GitHub Actions templates for quick setup
• Run different test types in parallel for faster feedback
• Set up scheduled runs for regular testing
• Use environment-specific configurations for different stages
• Store test results as artifacts for historical analysis
• Set appropriate timeouts for CI environments
• Use exit codes to gate deployments
• Integrate with notification systems for test failures

Configuration Reference

API Test Configuration

Retry Configuration

API Collection Configuration

Collection Auth

Collection Defaults

Collection Request

Data-Driven Testing Configuration

CSV Data File

username,email,user_id,role
johndoe,john.doe@example.com,1001,admin
janedoe,jane.doe@example.com,1002,user
bobsmith,bob.smith@example.com,1003,user

JSON Data File

[
  {
    "product_id": "P001",
    "name": "Smartphone",
    "price": 599.99,
    "category": "Electronics",
    "in_stock": true
  },
  {
    "product_id": "P002",
    "name": "Laptop",
    "price": 1299.99,
    "category": "Electronics",
    "in_stock": true
  }
]

Test Configuration with Placeholders

{
  "name": "User API Test for {{username}}",
  "description": "Test the user API for {{username}}",
  "url": "https://api.example.com/users/{{user_id}}",
  "method": "GET",
  "headers": {
    "Accept": "application/json",
    "X-User-Email": "{{email}}"
  },
  "expected_status": 200,
  "expected_body": {
    "id": "{{user_id}}",
    "role": "{{role}}"
  }
}

Run Options

Performance Test Configuration

Security Test Configuration

Web Test Configuration

Viewport Configuration

Web Assertion

Web Action

AI Configuration

Troubleshooting

Common Issues

API Testing

• Timeout Errors: Increase the timeout value in your configuration
• Connection Errors: Check network connectivity and retry settings
• Validation Failures: Verify expected response format and values
• Retry Loop: Check retry configuration and target system status

API Collections

• Dependency Errors: Ensure dependent requests are correctly defined and executed
• Variable Capture Failures: Check JSONPath expressions and response structure
• Variable Interpolation Issues: Verify variable names and syntax ({{variable}})
• Authentication Failures: Check auth configuration and token validity
• Sequential Execution Problems: Verify run_options and dependencies
• JSONPath Errors: Validate JSONPath expressions against actual response structure
• Missing Variables: Ensure all referenced variables are defined or captured
• Request Chaining Issues: Check that dependent requests are capturing the expected data
• Environment Configuration: Verify environment-specific variables are correctly set

Performance Testing

• High Error Rate: Reduce concurrent users or increase ramp-up time
• Slow Response Times: Check network latency and target system load
• Resource Exhaustion: Monitor system resources and adjust test parameters
• Inconsistent Results: Ensure test environment stability

Enhanced Performance Testing

• Stage Transition Issues: Check stage durations and target values
• Threshold Failures: Verify threshold expressions and metric names
• Scenario Distribution Problems: Check scenario weights and total count
• Metric Collection Issues: Ensure metrics are being properly captured
• High Resource Usage: Reduce the number of concurrent VUs or increase stage duration
• Percentile Calculation Errors: Ensure enough samples for accurate percentiles
• Tagged Metrics Missing: Verify tag names and values in scenario configuration

CI/CD Integration

• Exit Code Issues: Ensure test status is correctly reported
• GitHub Actions Failures: Check workflow YAML syntax and permissions
• Report Generation Failures: Verify output directory exists and is writable
• Parallel Test Conflicts: Ensure tests don't interfere with each other
• Timeout Errors: Increase CI job timeout or reduce test duration
• Environment Variable Issues: Check environment variable configuration
• Artifact Storage Problems: Verify artifact paths and retention settings

Data-Driven Testing

• CSV Parsing Errors: Check CSV format and delimiter settings
• JSON Parsing Errors: Validate JSON syntax and structure
• Placeholder Not Found: Ensure placeholder names match data column names
• Missing Data Columns: Verify data file contains all required columns
• Iteration Failures: Check if stop-on-failure is appropriate for your test
• Performance Issues: Reduce max iterations for large datasets
• File Not Found Errors: Verify data file paths
• JSON Path Errors: Check JSON path syntax for complex data structures
• Type Conversion Issues: Ensure data types match expected values
• Empty Data Sets: Verify data files contain valid test data

Security Testing

• False Positives: Review and adjust scan depth and types
• Authentication Failures: Verify auth configuration
• Scan Timeouts: Adjust timeout settings for complex scans
• Missing Findings: Check scan depth and types configuration

Web Testing

• Element Not Found: Check selectors and wait conditions
• Timeout Errors: Increase wait timeout for dynamic content
• Screenshot Issues: Verify screenshot directory permissions
• Assertion Failures: Check expected values and comparison types
• Action Failures: Verify element visibility and interactability

AI Features

• Model Loading Errors: Verify model path and format compatibility
• Out of Memory: Reduce context size or use a smaller model
• Poor Quality Output: Adjust temperature or provide more detailed prompts
• Slow Generation: Use a smaller model or reduce max tokens
• Missing Dependencies: Install AI dependencies with cargo build --features ai

Development

Prerequisites

• Rust 1.70 or higher
• Cargo
• Git

Constraints

QitOps is designed with the following constraints in mind:

• Minimal dependencies: Uses only the Rust standard library and well-known crates
• Static binary compilation: Can be compiled to a static binary for Linux
• CLI-only interface: All functionality is accessible via CLI flags with no UI dependencies
• Terminal-friendly output: Human-readable output for direct terminal use
• Machine-readable formats: Structured output for CI/CD integration

Building from Source

# Clone the repository
git clone https://github.com/yourusername/qitops.git
cd qitops

# Build the project
cargo build

# Run tests
cargo test

# Build documentation
cargo doc --no-deps

# Build static binary for Linux
cargo build --release --target x86_64-unknown-linux-musl

Architecture & Project Structure

QitOps follows a modular architecture with clear boundaries between components:

qitops/
├── src/
│   ├── main.rs        # CLI parsing using clap
│   ├── api.rs         # API testing implementation
│   ├── performance.rs # Performance testing implementation
│   ├── security.rs    # Security testing implementation
│   ├── web.rs         # Web testing implementation (extension)
│   ├── ai.rs          # AI-powered test generation (extension)
│   ├── reporting.rs   # Report generation (extension)
│   ├── common.rs      # Shared functionality and interfaces
│   └── error.rs       # Error handling
├── tests/
│   └── configs/       # JSON test configuration files
│       ├── api_test.json
│       ├── performance_test.json
│       ├── security_test.json
│       ├── web_test.json
│       └── ai_config.json
├── .github/
│   └── workflows/     # CI configuration
├── Dockerfile         # Container definition
├── docker-compose.yml # Container orchestration
└── Cargo.toml         # Dependencies (minimal and native)

The architecture is designed with the following principles:

• Clear module boundaries: Each testing type has its own module
• Common interfaces: All test runners implement the TestRunner trait
• JSON-based configuration: Tests are defined using structured JSON files
• Minimal dependencies: Uses Rust standard library and well-known crates
• CLI-first approach: All functionality accessible via command-line flags
• Extensibility: Designed for future expansion (TUI, AI integration)

CI/CD Integration

QitOps is designed to be CI/CD ready and can be easily integrated into your CI/CD pipeline:

GitHub Actions

The included GitHub Actions workflow (ci.yml) automatically:

• Builds and tests the project
• Runs linting and formatting checks
• Executes sample tests for each test type
• Creates release artifacts

Docker Integration

The included Dockerfile and docker-compose.yml allow you to:

• Build a containerized version of QitOps
• Run tests in isolated containers
• Mount configuration and result volumes
• Set environment variables for different environments

JUnit XML Reports

Generate JUnit XML reports for integration with CI/CD tools:

qitops -r xml -o test-results.xml api -c tests/configs/api_test.json

Most CI/CD platforms (Jenkins, GitHub Actions, GitLab CI, etc.) can automatically parse these reports to display test results.

Adding New Features

1. Create a new module in src/
2. Implement the TestRunner trait
3. Add CLI commands in main.rs
4. Add configuration structures
5. Write tests
6. Update documentation

QitOps OS Integration

QitOps CLI is designed to be bundled into QitOps OS, a custom bootable Linux distribution for QA professionals. When integrated into QitOps OS, the tool will be:

• Pre-installed in /usr/local/bin as a static binary
• Configured with default test configurations in /etc/qitops/configs
• Available directly from the terminal without additional setup
• Optimized for the QitOps OS environment

QA professionals can boot directly into QitOps OS and run tests from the terminal without any additional installation or configuration steps.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

When contributing, please keep in mind the core principles of the project:

• Maintain CLI-first approach with no UI dependencies
• Keep dependencies minimal and native
• Ensure compatibility with static binary compilation
• Preserve clear module boundaries
• Design for extensibility

License

This project is licensed under the MIT License - see the LICENSE file for details.