Add vendor dependencies and update project structure
This commit is contained in:
1 parent
191785021e
commit
7350331b3a
1077 files changed
+1082686
-357
No files matched your search
@@ -0,0 +1,21 @@
|
||||
# Active Context (As of 2025-04-23 ~08:04 UTC+8)
|
||||
|
||||
## 1. Current Focus
|
||||
Refining the core benchmarking logic and ensuring accurate metric collection, particularly after implementing and debugging streaming support.
|
||||
|
||||
## 2. Recent Changes
|
||||
- **Fixed Stats Recording:** Correctly implemented the recording of `stats.RequestResult` after each request (both streaming and non-streaming) completes within the `runWorker` function in `pkg/concurrency/manager.go`. This involved moving the `RecordResult` call inside the appropriate conditional blocks and using the correct field names (`IsSuccess`, `TimeToFirstToken`).
|
||||
- **Corrected Client Initialization:** Updated `cmd/benchmark/main.go` to correctly initialize the `client.FastHTTPClient`. Removed the flawed logic that attempted to switch client types based on the `streaming` flag, recognizing that `FastHTTPClient` internally handles both streaming (using `net/http`) and non-streaming (using `fasthttp`). Ensured the correct request timeout value is passed during initialization.
|
||||
- **Lint Fixes:** Addressed several lint warnings, including unused imports and tautological conditions.
|
||||
|
||||
## 3. Next Steps & Decisions
|
||||
- The core functionality of running concurrent streaming requests and recording basic results (Success, Latency, TTFT) is working.
|
||||
- The immediate next step is to choose which refinement to tackle next from the following options:
|
||||
1. Add detailed error recording to `RequestResult`.
|
||||
2. Add token statistics (e.g., average tokens/request, total tokens) to the final report.
|
||||
3. Improve/clarify TTFT calculation for non-streaming requests.
|
||||
4. Refactor `runWorker` in `pkg/concurrency/manager.go` for better readability.
|
||||
|
||||
## 4. Active Considerations
|
||||
- Ensuring the `net/http` client used internally for streaming has appropriate timeout settings (currently hardcoded in `Stream` method).
|
||||
- Planning the structure and implementation of the final report generation (console and potentially HTML).
|
||||
@@ -0,0 +1,21 @@
|
||||
# Product Context: LLM API Benchmark Tool
|
||||
|
||||
## 1. Problem Solved
|
||||
Evaluating and comparing the performance characteristics (speed, throughput, responsiveness) of different LLM APIs or different configurations of the same API can be difficult and time-consuming. This tool aims to provide a standardized and automated way to perform these benchmarks.
|
||||
|
||||
## 2. Target Users
|
||||
- Developers integrating LLM APIs who need to choose the best performing option.
|
||||
- Operations teams monitoring LLM API performance and reliability.
|
||||
- Researchers comparing different LLM models or hosting infrastructures.
|
||||
|
||||
## 3. How it Should Work
|
||||
- The user provides a `config.yaml` file specifying the target API, credentials, benchmark parameters (duration, concurrency, etc.), and desired prompt characteristics.
|
||||
- The tool runs the benchmark according to the configuration, sending requests and collecting performance data.
|
||||
- Upon completion (or interruption), the tool calculates and displays summary statistics for the key metrics.
|
||||
- Streaming requests should accurately measure TTFT and token throughput.
|
||||
|
||||
## 4. User Experience Goals
|
||||
- **Easy Configuration:** Simple and clear YAML configuration.
|
||||
- **Clear Output:** Understandable summary report of key metrics.
|
||||
- **Reliable Measurement:** Accurate and consistent performance data collection, especially for streaming TTFT.
|
||||
- **Informative Logging:** Useful logs during the benchmark run for diagnostics.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Progress (As of 2025-04-23 ~08:04 UTC+8)
|
||||
|
||||
## 1. What Works
|
||||
- **Core Benchmark Execution:** Can run concurrent requests (both streaming and non-streaming modes selectable via config) for a specified duration.
|
||||
- **Configuration Loading:** Reads parameters from `config.yaml`.
|
||||
- **Concurrency Control:** Limits active workers based on the `concurrency` setting.
|
||||
- **HTTP Client:** `FastHTTPClient` handles both request types:
|
||||
- Non-streaming (`Do`) uses `fasthttp`.
|
||||
- Streaming (`Stream`) uses `net/http` internally and processes SSE events.
|
||||
- **Tokenizer Integration:** Generates prompts and counts response tokens.
|
||||
- **Basic Stats Collection:**
|
||||
- Records individual `RequestResult` (IsSuccess, Latency, TimeToFirstToken, TotalTokens) correctly for both streaming and non-streaming requests.
|
||||
- Calculates and displays aggregate stats upon completion: Total Requests, Success/Fail counts, Success Rate, Avg QPS, Latency (Avg/Min/Max/Percentiles), TTFT (Avg/Min/Max/Percentiles), Avg Tokens/Second.
|
||||
|
||||
## 2. What's Left / Needs Refinement
|
||||
- **Error Reporting:** Currently only tracks success/failure boolean. Need to capture and potentially report specific error messages (`RequestResult.Error` field).
|
||||
- **Token Statistics:** While individual token counts are recorded, aggregate token stats (e.g., average tokens per successful request, total tokens generated) are not yet calculated or displayed in the final report.
|
||||
- **Non-Streaming TTFT:** The current TTFT for non-streaming requests is approximated as the total latency. This might need refinement or clearer definition.
|
||||
- **Streaming Timeout:** The timeout for the entire streaming request is hardcoded in `client.Stream`. Consider making this configurable.
|
||||
- **Report Generation:** Final report is currently basic console output. Planning to implement HTML report generation (possibly using `go-echarts`).
|
||||
- **Code Refinement:** `pkg/concurrency/manager.go`'s `runWorker` could be refactored for better readability/maintainability.
|
||||
- **Testing:** Need comprehensive integration tests covering various config scenarios (especially different prompt sizes, rate limits, etc.). Existing unit tests cover client and potentially other modules.
|
||||
|
||||
## 3. Current Status
|
||||
- The main bug related to recording streaming results has been fixed.
|
||||
- The client initialization logic is corrected.
|
||||
- The tool is functional for basic benchmarking scenarios, especially streaming.
|
||||
- Ready to proceed with further feature enhancements or refinements.
|
||||
|
||||
## 4. Known Issues
|
||||
- None critical at the moment. Previous lint warnings have been addressed.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Project Brief: LLM API Benchmark Tool
|
||||
|
||||
## 1. Core Goal
|
||||
Develop a command-line tool in Go to benchmark the performance of Large Language Model (LLM) APIs.
|
||||
|
||||
## 2. Key Features
|
||||
- Send configurable concurrent requests (streaming and non-streaming) to a specified LLM API endpoint.
|
||||
- Measure and report key performance metrics:
|
||||
- Request Success/Failure Rate
|
||||
- Queries Per Second (QPS)
|
||||
- Latency (min, max, average, percentiles)
|
||||
- Time To First Token (TTFT) for streaming and non-streaming (min, max, average, percentiles)
|
||||
- Token generation rate (tokens/second, average)
|
||||
- Support configurable parameters via a YAML file (API endpoint, key, model, concurrency, duration, rate limit, prompt size, streaming mode, timeouts, etc.).
|
||||
- Generate a summary report (initially console output, potentially HTML later).
|
||||
|
||||
## 3. Scope
|
||||
- Focus on HTTP-based LLM APIs (e.g., OpenAI compatible).
|
||||
- Initial implementation targets core metrics.
|
||||
- Extensibility for different client implementations or reporting formats is desirable.
|
||||
@@ -0,0 +1,65 @@
|
||||
# System Patterns
|
||||
|
||||
## 1. Overall Architecture
|
||||
- Modular Go application following standard Go project layout (`cmd`, `pkg`, `test`).
|
||||
- Configuration-driven behavior using YAML.
|
||||
- Clear separation of concerns between packages:
|
||||
- `cmd/benchmark`: Main application entry point, CLI flags, orchestrates initialization.
|
||||
- `pkg/config`: Loading and validation of `config.yaml`.
|
||||
- `pkg/client`: Abstraction layer for HTTP requests.
|
||||
- `pkg/tokenizer`: Handles prompt generation and token counting.
|
||||
- `pkg/concurrency`: Manages concurrent workers and request scheduling.
|
||||
- `pkg/stats`: Collects and aggregates performance metrics.
|
||||
- `pkg/report`: (Planned) Generates final benchmark reports.
|
||||
|
||||
## 2. Key Technical Decisions & Patterns
|
||||
- **HTTP Client Abstraction:**
|
||||
- `HTTPClient` interface (`pkg/client/interface.go`) defines `Do` (non-streaming) and `Stream` methods.
|
||||
- `FastHTTPClient` (`pkg/client/fasthttp.go`) is the primary implementation:
|
||||
- Uses `fasthttp` library for high performance in the `Do` method.
|
||||
- Uses Go's standard `net/http` library internally for the `Stream` method for better SSE handling stability.
|
||||
- Uses `sync.Pool` for `fasthttp` request/response object reuse.
|
||||
- **Concurrency Management:**
|
||||
- `pkg/concurrency/manager.go` uses a goroutine pool (implemented via channels/semaphore pattern) to limit concurrent workers based on `config.Concurrency`.
|
||||
- Uses `context.Context` for managing benchmark duration and worker cancellation.
|
||||
- **Statistics Collection:**
|
||||
- `pkg/stats/collector.go` uses a `sync.Map` for thread-safe storage of individual `RequestResult` structs.
|
||||
- Calculates final aggregate statistics (average, min/max, percentiles) upon benchmark completion.
|
||||
- Percentiles calculated using `gonum/stat`.
|
||||
- **Token Handling:**
|
||||
- Uses `github.com/tiktoken-go/tokenizer` for generating prompts of specific token lengths and counting tokens in responses.
|
||||
- **Configuration:**
|
||||
- Uses `viper` (or similar YAML parsing library) within `pkg/config` to load settings.
|
||||
|
||||
## 3. Component Relationships
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[cmd/benchmark/main.go] --> B(pkg/config)
|
||||
A --> C{client.HTTPClient}
|
||||
A --> D(pkg/stats)
|
||||
A --> E(pkg/concurrency)
|
||||
A --> F(pkg/tokenizer)
|
||||
|
||||
C -->|Do uses| G[fasthttp]
|
||||
C -->|Stream uses| H[net/http]
|
||||
|
||||
E --> C
|
||||
E --> F
|
||||
E --> D
|
||||
|
||||
subgraph pkg/client
|
||||
C
|
||||
G
|
||||
H
|
||||
end
|
||||
|
||||
subgraph pkg/stats
|
||||
D
|
||||
I[gonum/stat]
|
||||
end
|
||||
|
||||
subgraph pkg/tokenizer
|
||||
F
|
||||
J[tiktoken-go]
|
||||
end
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
# Technical Context
|
||||
|
||||
## 1. Core Technology
|
||||
- **Language:** Go (Golang)
|
||||
|
||||
## 2. Key Libraries & Dependencies
|
||||
- **HTTP:**
|
||||
- `github.com/valyala/fasthttp`: For high-performance non-streaming requests.
|
||||
- `net/http`: Go standard library, used for streaming (SSE) requests within the client implementation.
|
||||
- **Tokenization:**
|
||||
- `github.com/tiktoken-go/tokenizer`: For OpenAI-compatible token counting and prompt generation helper functions.
|
||||
- **Concurrency:**
|
||||
- Go standard library (`sync`, `context`, goroutines, channels).
|
||||
- **Statistics:**
|
||||
- `gonum.org/v1/gonum/stat`: For percentile calculations.
|
||||
- `sync/atomic`: Used within stats collector for thread-safe counters.
|
||||
- `sync.Map`: For thread-safe storage of request results.
|
||||
- **Configuration:**
|
||||
- `github.com/spf13/viper`: Likely used for loading YAML configuration (verify import).
|
||||
- `gopkg.in/yaml.v3` (or similar): For YAML parsing.
|
||||
- **CLI Flags:**
|
||||
- Go standard library `flag`.
|
||||
|
||||
## 3. Development Environment
|
||||
- **OS:** macOS (as per user information)
|
||||
- **Build/Dependency Management:** Go Modules (`go mod`)
|
||||
|
||||
## 4. Technical Constraints & Considerations
|
||||
- **HTTP Client Choice:** Balancing performance (`fasthttp`) and streaming stability (`net/http`). The current approach uses a hybrid within `FastHTTPClient`.
|
||||
- **Concurrency Limits:** Managing goroutine lifecycle and preventing resource exhaustion.
|
||||
- **Accurate Timing:** Measuring latency and TTFT precisely, especially across concurrent operations.
|
||||
- **Memory Management:** Using `sync.Pool` for `fasthttp` objects; being mindful of memory usage when handling large responses or many results.
|
||||
- **TDD:** Workspace rules mandate Test-Driven Development.
|
||||
Reference in new issue
Block a user