llm-api-benchmark-tool/memory-bank/productContext.md

Product Context: LLM API Benchmark Tool

1. Problem Solved

Evaluating and comparing the performance characteristics (speed, throughput, responsiveness) of different LLM APIs or different configurations of the same API can be difficult and time-consuming. This tool aims to provide a standardized and automated way to perform these benchmarks.

2. Target Users

• Developers integrating LLM APIs who need to choose the best performing option.
• Operations teams monitoring LLM API performance and reliability.
• Researchers comparing different LLM models or hosting infrastructures.

3. How it Should Work

• The user provides a config.yaml file specifying the target API, credentials, benchmark parameters (duration, concurrency, etc.), and desired prompt characteristics.
• The tool runs the benchmark according to the configuration, sending requests and collecting performance data.
• Upon completion (or interruption), the tool calculates and displays summary statistics for the key metrics.
• Streaming requests should accurately measure TTFT and token throughput.

4. User Experience Goals

• Easy Configuration: Simple and clear YAML configuration.
• Clear Output: Understandable summary report of key metrics.
• Reliable Measurement: Accurate and consistent performance data collection, especially for streaming TTFT.
• Informative Logging: Useful logs during the benchmark run for diagnostics.