llm-api-benchmark-tool/memory-bank/systemPatterns.md

System Patterns

1. Overall Architecture

• Modular Go application following standard Go project layout (cmd, pkg, test).
• Configuration-driven behavior using YAML.
• Clear separation of concerns between packages:
  • cmd/benchmark: Main application entry point, CLI flags, orchestrates initialization.
  • pkg/config: Loading and validation of config.yaml.
  • pkg/client: Abstraction layer for HTTP requests.
  • pkg/tokenizer: Handles prompt generation and token counting.
  • pkg/concurrency: Manages concurrent workers and request scheduling.
  • pkg/stats: Collects and aggregates performance metrics.
  • pkg/report: (Planned) Generates final benchmark reports.

2. Key Technical Decisions & Patterns

• HTTP Client Abstraction:
  • HTTPClient interface (pkg/client/interface.go) defines Do (non-streaming) and Stream methods.
  • FastHTTPClient (pkg/client/fasthttp.go) is the primary implementation:
    • Uses fasthttp library for high performance in the Do method.
    • Uses Go's standard net/http library internally for the Stream method for better SSE handling stability.
    • Uses sync.Pool for fasthttp request/response object reuse.
• Concurrency Management:
  • pkg/concurrency/manager.go uses a goroutine pool (implemented via channels/semaphore pattern) to limit concurrent workers based on config.Concurrency.
  • Uses context.Context for managing benchmark duration and worker cancellation.
• Statistics Collection:
  • pkg/stats/collector.go uses a sync.Map for thread-safe storage of individual RequestResult structs.
  • Calculates final aggregate statistics (average, min/max, percentiles) upon benchmark completion.
  • Percentiles calculated using gonum/stat.
• Token Handling:
  • Uses github.com/tiktoken-go/tokenizer for generating prompts of specific token lengths and counting tokens in responses.
• Configuration:
  • Uses viper (or similar YAML parsing library) within pkg/config to load settings.

3. Component Relationships

flowchart TD
    A[cmd/benchmark/main.go] --> B(pkg/config)
    A --> C{client.HTTPClient}
    A --> D(pkg/stats)
    A --> E(pkg/concurrency)
    A --> F(pkg/tokenizer)

    C -->|Do uses| G[fasthttp]
    C -->|Stream uses| H[net/http]

    E --> C
    E --> F
    E --> D

    subgraph pkg/client
        C
        G
        H
    end

    subgraph pkg/stats
        D
        I[gonum/stat]
    end

    subgraph pkg/tokenizer
        F
        J[tiktoken-go]
    end