Compare commits

...
4 Commits
1085 changed files with 1082914 additions and 2774 deletions

No files matched your search

+12
View File
@@ -0,0 +1,12 @@
# llm-api-benchmark-tool Project Rules & Patterns
1. **TDD is MANDATORY.** Write tests first (`test/` dir).
2. **Client Implementation:** Use `FastHTTPClient` (`pkg/client/fasthttp.go`).
3. `FastHTTPClient` uses `fasthttp` for `Do` (non-stream) and `net/http` for `Stream` (SSE).
4. Streaming stability relies on the internal `net/http` usage.
5. Configuration via `config/config.yaml` drives benchmark parameters.
6. Core metrics defined in `stats.RequestResult` (`pkg/stats/collector.go`).
7. Concurrency managed via goroutine pool in `pkg/concurrency/manager.go`.
8. Use `tiktoken-go/tokenizer` for token operations.
9. Record results (`statsCollector.RecordResult`) immediately after request completion in `runWorker`.
10. Prefer fixing core functionality/bugs before large refactors.
+136
View File
@@ -0,0 +1,136 @@
# Workspace Rules for `llm-api-benchmark-tool`
## 1. Project Structure
- **Root Directory**: `llm-api-benchmark-tool`
- **Subdirectories**:
- `cmd/`: Entry point for the main application.
- `pkg/`: Core functionality modules.
- `client/`: HTTP client abstraction layer.
- `tokenizer/`: Prompt generation using `github.com/tiktoken-go/tokenizer`.
- `concurrency/`: Concurrency management with goroutine pools and `sync.Pool`.
- `stats/`: Performance metrics collection with `sync.Map` and `gonum/stat`.
- `report/`: Report generation with `go-echarts`.
- `test/`: Test cases for all modules.
- `config/`: Configuration files (e.g., `config.yaml`).
- `docs/`: Documentation.
- `scripts/`: Utility scripts.
## 2. Test-Driven Development (TDD)
- **MANDATORY TDD**: The entire project must follow test-driven development. Write test cases first, lock them, then iteratively develop code until all tests pass.
- **MANDATORY TDD**: The entire project must follow test-driven development. Write test cases first, lock them, then iteratively develop code until all tests pass.
- **MANDATORY TDD**: The entire project must follow test-driven development. Write test cases first, lock them, then iteratively develop code until all tests pass.
- **Process**:
1. Write test cases defining expected behavior, including edge cases and error handling.
2. Run tests to ensure they fail (red).
3. Write minimal code to pass tests (green).
4. Refactor code while keeping tests passing.
5. Repeat for each feature.
## 3. Technical Stack
- **Language**: Go.
- **HTTP Client**:
- Interface: `HTTPClient` defined in `pkg/client/interface.go`.
- Implementations:
- `fasthttp` (`github.com/valyala/fasthttp`): High-performance, default choice.
- `net/http` (standard library): Fallback for compatibility.
- Configurable via `client: "fasthttp"` or `client: "net-http"` in `config.yaml`.
- **Tokenizer**: `github.com/tiktoken-go/tokenizer` for prompt generation.
- **Concurrency**: Goroutine pool (e.g., `ants` or custom), `sync.Pool` for object reuse, `context.Context` for control.
- **Data Processing**: `sync.Map` for response storage, `gonum/stat` for percentile calculations.
- **Charts**: `go-echarts` for visualizing metrics.
## 4. HTTP Client Abstraction Layer
- **Interface**: `HTTPClient` with:
- `Do(req *Request) (*Response, error)`: Non-streaming requests.
- `Stream(req *Request, callback func(chunk SSEChunk) error) error`: Streaming SSE responses.
- **Implementations**:
- `pkg/client/fasthttp.go`: Uses `fasthttp` for high throughput.
- `pkg/client/nethttp.go`: Uses `net/http` for standard compatibility.
- **Tests**: `test/client_test.go` verifies `Do` and `Stream`, including TTFT measurement with SSE.
- **Switching**: Config-driven client selection at runtime.
## 5. Prompt Generation
- **Library**: `github.com/tiktoken-go/tokenizer`.
- **Features**: Generate prompts of 50 tokens (±5%) and 1000 tokens (±5%).
- **Tests**: `test/tokenizer_test.go` ensures token counts are within tolerance.
## 6. Concurrency Management
- **Goroutine Pool**: Limit to 600 goroutines max, using `ants` or custom implementation.
- **Object Reuse**: `sync.Pool` for `Request` and `Response` objects.
- **Control**: `context.WithTimeout` for test duration (default 300s).
- **Tests**: `test/concurrency_test.go` validates concurrency levels, intervals, and termination.
## 7. Performance Metrics Collection
- **Storage**: `sync.Map` for thread-safe response data.
- **Calculations**: `gonum/stat` for P90/P95/P99 percentiles.
- **Metrics**:
- Request stats (total, successful, failed, timeout ratio).
- Response time (avg, min/max, percentiles).
- TTFT (min/max, percentiles) via SSE.
- QPS (avg, max).
- Token generation rate (avg, max) for `content` and `reasoning_content`.
- **Tests**: `test/stats_test.go` ensures accurate metrics.
## 8. Report Generation
- **Charts**: Use `go-echarts` for response time, TTFT, QPS, and token rate visualizations.
- **Format**: HTML/PDF reports via templates.
- **Tests**: `test/report_test.go` validates report content and charts.
## 9. Configuration Management
- **File**: `config/config.yaml` for API, prompts, timeouts, etc.
- **Parsing**: Use `viper` or `yaml` package.
- **Tests**: `test/config_test.go` verifies loading and parsing.
## 10. Logging and Monitoring
- **Logging**: Use `log` package, enable debug logs with `--debug`.
- **Monitoring**: Use `runtime` for CPU, memory, and goroutine tracking.
- **Tests**: `test/log_test.go` and `test/monitor_test.go` ensure proper output.
## 11. Continuous Integration
- **CI/CD**: GitHub Actions for automated testing.
- **Coverage**: Target 80%+ code coverage.
- **Automation**: Run tests on every commit.
## 12. Version Control
- **Tool**: Git.
- **Branches**: `main`, `develop`, `feature/*`, `bugfix/*`.
## 13. Code Standards
- **Formatting**: `gofmt` for consistency.
- **Linting**: `golint` or `staticcheck` for quality checks.
- **Naming**: Follow Go conventions.
## 14. Dependency Management
- **Tool**: `go mod` for dependencies.
- **Optional**: `go mod vendor` for offline builds.
## 15. Security
- **API Keys**: Use environment variables or config files, never hardcode.
- **Error Handling**: Catch and log all errors to prevent crashes.
## 16. Performance Optimization
- **Benchmarks**: Use `testing` package for performance tests.
- **Profiling**: Use `pprof` for analysis.
## 17. User Engagement
- **Feedback**: Encourage issues and PRs via GitHub.
- **Guidelines**: Provide `CONTRIBUTING.md`.
## 18. Release Process
- **Versioning**: Semantic versioning (e.g., `v1.0.0`).
- **Changelog**: Maintain `CHANGELOG.md`.
## 19. Community
- **Discussion**: Use GitHub Discussions.
- **License**: MIT.
## 20. TDD Workflow
- **Steps**:
1. Write and lock test cases in `test/`.
2. Implement code in `pkg/` or `cmd/`.
3. Iterate until tests pass.
4. Refactor and retest.
- **Examples**: See `test/client_test.go`, `test/tokenizer_test.go`, etc.
## Summary
These rules ensure a robust, testable, and maintainable `llm-api-benchmark-tool`. The TDD approach—writing tests first, locking them, and coding until they pass—is enforced throughout. Windsurf can leverage this structure to generate code, tests, and documentation systematically.
-52
View File
@@ -1,52 +0,0 @@
# LLM API 基准测试工具
这是一个用于测试兼容OpenAI风格API的LLM性能基准测试工具。该工具支持动态生成提示词、高并发请求和详细的性能报告。
## 功能特点
- **API兼容性**:支持OpenAI Compatible风格的LLM API
- **提示词生成**:
- 集成tiktoken-go/tokenizer
- 自动生成短咨询提示词(50 tokens ±5%)和长文档提示词(1000 tokens ±5%)
- 基于模板动态调整至目标Token数
- **并发请求**:支持高并发异步请求,模拟真实负载
- **场景建模**:支持混合负载(短咨询70%,长文档30%)
- **用户行为模拟**:请求间隔遵循泊松分布
- **梯度增压**:支持阶梯式加压(50→200→500并发)
- **性能指标收集**:响应时间、QPS、Token生成速率等
- **报告生成**:生成包含性能指标统计和可视化图表的质量报告
## 安装
```bash
git clone https://github.com/yourusername/llm-api-benchmark-tool.git
cd llm-api-benchmark-tool
go mod download
```
## 使用方法
1. 配置 `config.yaml` 文件:
```yaml
api:
endpoint: "https://api.example.com/v1/completions"
api_key: "your_api_key"
model: "gpt-3.5-turbo"
```
2. 运行测试:
```bash
go run main.go -config config.yaml
```
3. 查看生成的报告(位于 `reports` 目录)
## 配置说明
详细配置选项请参考 [配置文档](docs/configuration.md)。
## 许可证
MIT
-673
View File
@@ -1,673 +0,0 @@
package benchmark
import (
"bytes"
"context"
"encoding/json"
"fmt"
"math"
"math/rand"
"strings"
"sync"
"time"
"github.com/montanaflynn/stats"
"github.com/valyala/fasthttp"
"github.com/acmestudio/llm-api-benchmark-tool/config"
"github.com/acmestudio/llm-api-benchmark-tool/logger"
"github.com/acmestudio/llm-api-benchmark-tool/prompt"
)
// Benchmark 表示基准测试执行器
type Benchmark struct {
config *config.Config
generator *prompt.Generator
client *fasthttp.Client
rand *rand.Rand
}
// Response 表示API响应
type Response struct {
ID string `json:"id"`
Object string `json:"object"`
Created int64 `json:"created"`
Model string `json:"model"`
Choices []Choice `json:"choices"`
Usage Usage `json:"usage"`
}
// StreamResponse 表示流式API响应
type StreamResponse struct {
ID string `json:"id"`
Object string `json:"object"`
Created int64 `json:"created"`
Model string `json:"model"`
Choices []StreamChoice `json:"choices"`
}
// StreamChoice 表示流式响应中的选择
type StreamChoice struct {
Delta StreamDelta `json:"delta"`
Index int `json:"index"`
FinishReason string `json:"finish_reason"`
}
// StreamDelta 表示流式响应中的增量内容
type StreamDelta struct {
Role string `json:"role"`
Content string `json:"content"`
}
// Choice 表示API响应中的选择
type Choice struct {
Message Message `json:"message"`
Index int `json:"index"`
FinishReason string `json:"finish_reason"`
}
// Message 表示聊天消息
type Message struct {
Role string `json:"role"`
Content string `json:"content"`
}
// Usage 表示API响应中的使用情况
type Usage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
}
// Request 表示API请求
type Request struct {
Model string `json:"model"`
Messages []Message `json:"messages"`
MaxTokens int `json:"max_tokens"`
Temperature float64 `json:"temperature"`
Stream bool `json:"stream"`
}
// Result 表示单个API请求的结果
type Result struct {
Timestamp time.Time // 请求时间戳
Concurrency int // 并发级别
PromptType string // 提示词类型 (short/long)
PromptTokenCount int // 提示词token数量
PromptTokens int // 提示词token数量(API返回)
CompletionTokens int // 完成token数量
TotalTokens int // 总token数量
ResponseTime time.Duration // 响应时间
Response string // 响应内容
Error error // 错误信息
}
// BenchmarkResults 表示基准测试结果
type BenchmarkResults struct {
Results []Result // 所有请求结果
StartTime time.Time // 开始时间
EndTime time.Time // 结束时间
Config *config.Config // 配置信息
ConcurrencyData map[int][]Result // 按并发步骤分组的结果
TotalRequests int // 总请求数
SuccessRequests int // 成功请求数
FailedRequests int // 失败请求数
AvgResponseTime time.Duration // 平均响应时间
P50ResponseTime time.Duration // 响应时间50百分位
P90ResponseTime time.Duration // 响应时间90百分位
P99ResponseTime time.Duration // 响应时间99百分位
QPS float64 // 每秒请求数
AvgPromptTokens int // 平均提示词token数
AvgCompletionTokens int // 平均完成token数
AvgTotalTokens int // 平均总token数
AvgTokenRate float64 // 平均token生成速率
}
// NewBenchmark 创建新的基准测试执行器
func NewBenchmark(cfg *config.Config) (*Benchmark, error) {
// 初始化提示词生成器
logger.Debug("初始化提示词生成器...")
gen, err := prompt.NewGenerator(cfg)
if err != nil {
return nil, fmt.Errorf("初始化提示词生成器失败: %w", err)
}
logger.Debug("提示词生成器初始化成功")
// 初始化HTTP客户端
logger.Debug("初始化HTTP客户端...")
client := &fasthttp.Client{
MaxConnsPerHost: 10000,
ReadTimeout: time.Duration(cfg.Timeout) * time.Second,
WriteTimeout: time.Duration(cfg.Timeout) * time.Second,
}
logger.Debug("HTTP客户端初始化成功,超时设置: %d秒", cfg.Timeout)
// 初始化随机数生成器
source := rand.NewSource(time.Now().UnixNano())
return &Benchmark{
config: cfg,
generator: gen,
client: client,
rand: rand.New(source),
}, nil
}
// Run 执行基准测试
func (b *Benchmark) Run() (*BenchmarkResults, error) {
// 创建结果对象
results := &BenchmarkResults{
StartTime: time.Now(),
Config: b.config,
ConcurrencyData: make(map[int][]Result),
}
// 执行每个并发步骤
for _, concurrency := range b.config.Concurrency.Steps {
logger.Info("开始执行并发步骤: %d 并发用户", concurrency)
logger.Info("预计持续时间: %d秒", b.config.Concurrency.DurationPerStep)
// 执行单个并发步骤
stepResults, err := b.runConcurrencyStep(concurrency)
if err != nil {
return nil, fmt.Errorf("执行并发步骤 %d 失败: %w", concurrency, err)
}
// 统计成功率
successCount := 0
for _, result := range stepResults {
if result.Error == nil {
successCount++
}
// 添加到总结果
results.Results = append(results.Results, result)
}
// 记录步骤结果
results.ConcurrencyData[concurrency] = stepResults
// 输出步骤统计
successRate := 0.0
if len(stepResults) > 0 {
successRate = float64(successCount) / float64(len(stepResults)) * 100
}
logger.Info("完成并发步骤: %d 并发用户,共 %d 个请求,成功率: %.2f%%",
concurrency, len(stepResults), successRate)
}
results.EndTime = time.Now()
logger.Info("所有并发步骤测试完成")
return results, nil
}
// runConcurrencyStep 执行单个并发步骤
func (b *Benchmark) runConcurrencyStep(concurrencyStep int) ([]Result, error) {
var results []Result
var resultsMutex sync.Mutex
// 创建上下文,用于控制测试时间
ctx, cancel := context.WithTimeout(
context.Background(),
time.Duration(b.config.Concurrency.DurationPerStep)*time.Second,
)
defer cancel()
// 创建工作通道
workChan := make(chan struct{}, concurrencyStep)
// 统计变量
var statsMutex sync.Mutex
totalRequests := 0
successRequests := 0
failedRequests := 0
totalResponseTime := time.Duration(0)
// 启动统计协程
statsTicker := time.NewTicker(5 * time.Second)
defer statsTicker.Stop()
go func() {
startTime := time.Now()
for {
select {
case <-ctx.Done():
return
case <-statsTicker.C:
statsMutex.Lock()
currentTotal := totalRequests
currentSuccess := successRequests
currentFailed := failedRequests
var avgResponseTime time.Duration
if successRequests > 0 {
avgResponseTime = totalResponseTime / time.Duration(successRequests)
}
statsMutex.Unlock()
elapsedTime := time.Since(startTime).Seconds()
qps := float64(currentTotal) / elapsedTime
logger.Info("实时统计 - 请求: %d (成功: %d, 失败: %d), QPS: %.2f, 平均响应时间: %v",
currentTotal, currentSuccess, currentFailed, qps, avgResponseTime)
}
}
}()
// 启动工作协程
var wg sync.WaitGroup
logger.Debug("启动 %d 个工作协程...", concurrencyStep)
for i := 0; i < concurrencyStep; i++ {
wg.Add(1)
go func(workerID int) {
defer wg.Done()
logger.Debug("工作协程 #%d 已启动", workerID)
for {
select {
case <-ctx.Done():
// 测试时间结束
logger.Debug("工作协程 #%d 收到结束信号", workerID)
return
case workChan <- struct{}{}:
// 执行请求前先更新计数器
statsMutex.Lock()
totalRequests++
requestID := totalRequests
statsMutex.Unlock()
logger.Debug("工作协程 #%d 开始执行请求 #%d", workerID, requestID)
// 执行请求
result := b.executeRequest(concurrencyStep)
// 更新统计
statsMutex.Lock()
if result.Error == nil {
successRequests++
totalResponseTime += result.ResponseTime
logger.Debug("请求 #%d 成功,响应时间: %v", requestID, result.ResponseTime)
} else {
failedRequests++
logger.Debug("请求 #%d 失败: %v", requestID, result.Error)
}
statsMutex.Unlock()
// 添加结果
resultsMutex.Lock()
results = append(results, result)
resultsMutex.Unlock()
// 根据泊松分布等待
waitTime := b.getPoissonWaitTime()
logger.Debug("请求 #%d 完成,等待 %v 后发送下一个请求", requestID, waitTime)
time.Sleep(waitTime)
<-workChan
}
}
}(i)
}
// 等待所有工作协程完成
logger.Debug("等待所有工作协程完成...")
wg.Wait()
logger.Debug("所有工作协程已完成")
return results, nil
}
// executeRequest 执行单个请求
func (b *Benchmark) executeRequest(concurrencyStep int) Result {
result := Result{
Timestamp: time.Now(),
Concurrency: concurrencyStep,
}
// 选择提示词类型
promptType := b.generator.SelectPromptType()
result.PromptType = promptType
// 生成提示词
prompt, err := b.generator.GeneratePrompt(promptType)
if err != nil {
result.Error = fmt.Errorf("生成提示词失败: %w", err)
return result
}
// 创建请求
req := Request{
Model: b.config.API.Model,
Messages: []Message{
{
Role: "user",
Content: prompt,
},
},
MaxTokens: 2048,
Temperature: 0.7,
Stream: true, // 启用流式响应
}
// 序列化请求
reqBody, err := json.Marshal(req)
if err != nil {
result.Error = fmt.Errorf("序列化请求失败: %w", err)
return result
}
// 创建HTTP请求
httpReq := fasthttp.AcquireRequest()
httpResp := fasthttp.AcquireResponse()
defer fasthttp.ReleaseRequest(httpReq)
defer fasthttp.ReleaseResponse(httpResp)
httpReq.SetRequestURI(b.config.API.Endpoint)
httpReq.Header.SetMethod("POST")
httpReq.Header.Set("Content-Type", "application/json")
httpReq.Header.Set("Authorization", "Bearer "+b.config.API.APIKey)
httpReq.Header.Set("Accept", "text/event-stream") // 设置接受流式响应
httpReq.SetBody(reqBody)
// 记录开始时间
startTime := time.Now()
// 发送请求
logger.Debug("发送请求到 %s,提示词类型: %s,提示词长度: %d字符",
b.config.API.Endpoint, promptType, len(prompt))
err = b.client.Do(httpReq, httpResp)
// 计算响应时间
responseTime := time.Since(startTime)
result.ResponseTime = responseTime
// 处理错误
if err != nil {
result.Error = fmt.Errorf("发送请求失败: %w", err)
logger.Error("请求失败: %v", err)
return result
}
// 检查状态码
statusCode := httpResp.StatusCode()
if statusCode != 200 {
result.Error = fmt.Errorf("API返回错误状态码: %d, 响应: %s", statusCode, httpResp.Body())
logger.Error("API返回错误状态码: %d, 响应: %s", statusCode, httpResp.Body())
return result
}
// 处理流式响应
if req.Stream {
return b.handleStreamResponse(httpResp, result, startTime)
}
// 处理非流式响应
return b.handleNormalResponse(httpResp, result)
}
// handleStreamResponse 处理流式响应
func (b *Benchmark) handleStreamResponse(httpResp *fasthttp.Response, result Result, startTime time.Time) Result {
body := httpResp.Body()
responseContent := ""
tokenCount := 0
// 打印响应头和原始响应体的前100个字符,用于调试
logger.Debug("响应头: %s", httpResp.Header.String())
if len(body) > 0 {
previewLen := 100
if len(body) < previewLen {
previewLen = len(body)
}
logger.Debug("响应体预览: %s", string(body[:previewLen]))
} else {
logger.Warn("收到空响应体")
result.Error = fmt.Errorf("收到空响应体")
return result
}
// 解析流式响应
// 流式响应格式为: data: {...}\n\ndata: {...}\n\ndata: [DONE]\n\n
lines := bytes.Split(body, []byte("\n\n"))
if len(lines) == 0 {
logger.Warn("无法解析流式响应")
result.Error = fmt.Errorf("无法解析流式响应")
return result
}
logger.Debug("解析到 %d 个数据块", len(lines))
// 超时控制
timeout := time.After(30 * time.Second)
// 使用通道处理解析结果
doneChan := make(chan struct{})
go func() {
for _, line := range lines {
if len(line) == 0 {
continue
}
// 检查是否为数据行
if !bytes.HasPrefix(line, []byte("data: ")) {
continue
}
// 移除 "data: " 前缀
data := line[6:]
// 检查是否为结束标记
if bytes.Equal(data, []byte("[DONE]")) {
logger.Debug("收到流式响应结束标记")
break
}
// 解析JSON
var chunk map[string]interface{}
if err := json.Unmarshal(data, &chunk); err != nil {
logger.Warn("解析JSON数据块失败: %v, 数据: %s", err, string(data))
continue
}
// 提取内容
choices, ok := chunk["choices"].([]interface{})
if !ok || len(choices) == 0 {
logger.Warn("无法从数据块中提取choices")
continue
}
choice, ok := choices[0].(map[string]interface{})
if !ok {
logger.Warn("无法从choices中提取第一个元素")
continue
}
delta, ok := choice["delta"].(map[string]interface{})
if !ok {
logger.Warn("无法从choice中提取delta")
continue
}
content, ok := delta["content"].(string)
if !ok {
// 某些delta可能没有content,这是正常的
continue
}
// 累加内容
responseContent += content
tokenCount += len(strings.Split(content, " "))
}
close(doneChan)
}()
// 等待处理完成或超时
select {
case <-doneChan:
// 处理完成
case <-timeout:
logger.Warn("解析流式响应超时")
result.Error = fmt.Errorf("解析流式响应超时")
return result
}
// 计算响应时间
result.ResponseTime = time.Since(startTime)
// 设置结果
result.Response = responseContent
result.PromptTokenCount = len(result.PromptType) / 2 // 简单估计
result.PromptTokens = len(result.PromptType) / 2 // 简单估计
result.CompletionTokens = tokenCount
result.TotalTokens = result.PromptTokens + result.CompletionTokens
logger.Debug("流式响应处理完成,响应时间: %v, 提示词Token: %d, 完成Token: %d, 总Token: %d",
result.ResponseTime, result.PromptTokens, result.CompletionTokens, result.TotalTokens)
return result
}
// handleNormalResponse 处理普通响应
func (b *Benchmark) handleNormalResponse(httpResp *fasthttp.Response, result Result) Result {
// 解析响应
var response Response
if err := json.Unmarshal(httpResp.Body(), &response); err != nil {
result.Error = fmt.Errorf("解析响应失败: %w", err)
logger.Error("解析响应失败: %v", err)
return result
}
// 记录Token数
result.PromptTokens = response.Usage.PromptTokens
result.CompletionTokens = response.Usage.CompletionTokens
result.TotalTokens = response.Usage.TotalTokens
// 记录详细日志
logger.Debug("请求完成 - 类型: %s, 提示词Token: %d, 响应Token: %d, 响应时间: %v",
result.PromptType, result.PromptTokens, result.CompletionTokens, result.ResponseTime)
return result
}
// getPoissonWaitTime 根据泊松分布获取等待时间
func (b *Benchmark) getPoissonWaitTime() time.Duration {
// 生成泊松分布随机数
lambda := b.config.PoissonLambda
L := math.Exp(-lambda)
k := 0
p := 1.0
for p > L {
k++
p *= b.rand.Float64()
}
// 转换为等待时间(毫秒)
waitTime := time.Duration(k * 100) * time.Millisecond
// 限制最大等待时间为1秒
if waitTime > time.Second {
waitTime = time.Second
}
return waitTime
}
// convertToFloat64 将整数切片转换为浮点数切片
func convertToFloat64(ints []int) []float64 {
floats := make([]float64, len(ints))
for i, v := range ints {
floats[i] = float64(v)
}
return floats
}
// calculateStats 计算统计数据
func (b *Benchmark) calculateStats(results []Result) *BenchmarkResults {
var stats BenchmarkResults
// 过滤出成功的请求
var successResults []Result
for _, r := range results {
if r.Error == nil {
successResults = append(successResults, r)
}
}
// 如果没有成功的请求,返回空结果
if len(successResults) == 0 {
return &stats
}
// 计算基本统计数据
stats.TotalRequests = len(results)
stats.SuccessRequests = len(successResults)
stats.FailedRequests = stats.TotalRequests - stats.SuccessRequests
// 计算响应时间统计
var responseTimes []float64
for _, r := range successResults {
responseTimes = append(responseTimes, float64(r.ResponseTime.Milliseconds()))
}
// 计算平均响应时间
avg, _ := stats.Mean(responseTimes)
stats.AvgResponseTime = time.Duration(avg) * time.Millisecond
// 计算响应时间百分位数
p50, _ := stats.Percentile(responseTimes, 50)
p90, _ := stats.Percentile(responseTimes, 90)
p99, _ := stats.Percentile(responseTimes, 99)
stats.P50ResponseTime = time.Duration(p50) * time.Millisecond
stats.P90ResponseTime = time.Duration(p90) * time.Millisecond
stats.P99ResponseTime = time.Duration(p99) * time.Millisecond
// 计算QPS
totalDuration := successResults[len(successResults)-1].Timestamp.Sub(successResults[0].Timestamp).Seconds()
if totalDuration > 0 {
stats.QPS = float64(stats.SuccessRequests) / totalDuration
}
// 计算Token统计
var promptTokens, completionTokens, totalTokens []int
for _, r := range successResults {
promptTokens = append(promptTokens, r.PromptTokens)
completionTokens = append(completionTokens, r.CompletionTokens)
totalTokens = append(totalTokens, r.TotalTokens)
}
// 计算平均Token数
if len(promptTokens) > 0 {
promptAvg, _ := stats.Mean(convertToFloat64(promptTokens))
completionAvg, _ := stats.Mean(convertToFloat64(completionTokens))
totalAvg, _ := stats.Mean(convertToFloat64(totalTokens))
stats.AvgPromptTokens = int(promptAvg)
stats.AvgCompletionTokens = int(completionAvg)
stats.AvgTotalTokens = int(totalAvg)
}
// 计算Token生成速率 (tokens/s)
var tokenRates []float64
for _, r := range successResults {
if r.ResponseTime.Seconds() > 0 {
tokenRate := float64(r.CompletionTokens) / r.ResponseTime.Seconds()
tokenRates = append(tokenRates, tokenRate)
}
}
if len(tokenRates) > 0 {
tokenRateAvg, _ := stats.Mean(tokenRates)
stats.AvgTokenRate = tokenRateAvg
}
return &stats
}
func (b *BenchmarkResults) Mean(data []float64) (float64, error) {
return stats.Mean(data)
}
func (b *BenchmarkResults) Percentile(data []float64, p float64) (float64, error) {
return stats.Percentile(data, p)
}
Executable
BIN
View File
Binary file not shown.
+111
View File
@@ -0,0 +1,111 @@
package main
import (
"flag"
"log"
"time"
"context"
"llm-api-benchmark-tool/pkg/client"
"llm-api-benchmark-tool/pkg/config"
"llm-api-benchmark-tool/pkg/concurrency"
"llm-api-benchmark-tool/pkg/stats"
"llm-api-benchmark-tool/pkg/tokenizer"
"llm-api-benchmark-tool/pkg/report"
)
func main() {
// Define command-line flags
configPath := flag.String("config", "config/config.yaml", "Path to the configuration YAML file")
flag.Parse()
// Load configuration
cfg, cfgErr := config.LoadConfig(*configPath)
if cfgErr != nil {
log.Fatalf("Failed to load config: %v", cfgErr)
}
// Check Tokenizer Initialization Status (happens in init() func)
tknErr := tokenizer.CheckInitStatus()
if tknErr != nil {
log.Fatalf("Tokenizer initialization failed: %v", tknErr)
}
log.Println("Tokenizer initialization checked successfully.")
// Initialize HTTP Client
// Determine the effective request timeout
requestTimeout := cfg.RequestTimeout
if requestTimeout <= 0 {
requestTimeout = cfg.Timeout // Fallback to global timeout
}
// Always create FastHTTPClient; it handles both streaming (net/http) and non-streaming (fasthttp)
httpClient := client.NewFastHTTPClient(requestTimeout)
log.Println("Configuration loaded successfully.")
log.Printf("API Endpoint: %s", cfg.APIEndpoint)
// The actual client used (fasthttp for Do, net/http for Stream) is handled internally by FastHTTPClient
log.Printf("Using HTTP client implementation: FastHTTPClient (non-stream timeout: %s)", requestTimeout)
log.Println("HTTP Client initialized.")
// Initialize Stats Collector
statsCollector := stats.NewStatsCollector()
log.Println("Stats Collector initialized.")
// Initialize Concurrency Manager
benchmarkParams := concurrency.BenchmarkParams{
HTTPClient: httpClient,
TargetURL: cfg.APIEndpoint,
APIKey: cfg.APIKey,
TotalDuration: cfg.Timeout,
Concurrency: cfg.Concurrency,
RequestInterval: time.Duration(cfg.RateLimit) * time.Millisecond, // Convert ms to time.Duration
IsStreaming: cfg.Streaming,
PromptTokens: cfg.PromptTokens,
Model: cfg.Model,
// TODO: Add other params like API Key, Model from cfg if needed
}
concurrencyManager := concurrency.NewManager(benchmarkParams, statsCollector)
log.Println("Concurrency Manager initialized.")
log.Printf("Benchmark settings: Duration=%s, Concurrency=%d, RateLimit=%s, Streaming=%v, PromptTokens=%d, Model=%s",
benchmarkParams.TotalDuration, benchmarkParams.Concurrency, benchmarkParams.RequestInterval, benchmarkParams.IsStreaming, benchmarkParams.PromptTokens, benchmarkParams.Model)
// --- Run Benchmark ---
log.Println("Starting benchmark...")
ctx := context.Background()
if err := concurrencyManager.RunBenchmark(ctx); err != nil {
log.Fatalf("Benchmark run failed: %v", err)
}
log.Println("Benchmark finished.")
// --- Get Final Stats ---
finalStats := statsCollector.CalculateStats(benchmarkParams.TotalDuration)
log.Println("--- Benchmark Results ---")
log.Printf("Total Duration: %.2fs", finalStats.TotalDuration.Seconds())
log.Printf("Total Requests: %d", finalStats.TotalRequests)
log.Printf("Successful Requests: %d", finalStats.SuccessfulRequests)
log.Printf("Failed Requests: %d", finalStats.FailedRequests)
if finalStats.TotalRequests > 0 {
log.Printf("Success Rate: %.2f%%", float64(finalStats.SuccessfulRequests)/float64(finalStats.TotalRequests)*100)
}
log.Printf("Average QPS: %.2f", finalStats.AvgQPS)
if finalStats.SuccessfulRequests > 0 {
log.Printf("Average Latency (Successful): %s", finalStats.AvgLatency)
log.Printf("Latency (Min/Max): %s / %s", finalStats.MinLatency, finalStats.MaxLatency)
log.Printf("Latency (P90/P95/P99): %s / %s / %s", finalStats.P90Latency, finalStats.P95Latency, finalStats.P99Latency)
log.Printf("Average TTFT (Successful): %s", finalStats.AvgTimeToFirstToken)
log.Printf("TTFT (Min/Max): %s / %s", finalStats.MinTimeToFirstToken, finalStats.MaxTimeToFirstToken)
log.Printf("TTFT (P90/P95/P99): %s / %s / %s", finalStats.P90TimeToFirstToken, finalStats.P95TimeToFirstToken, finalStats.P99TimeToFirstToken)
log.Printf("Average Tokens/Second (Successful): %.2f", finalStats.AvgTokensPerSecond)
}
log.Println("-------------------------")
// --- Generate Report ---
if cfg.ReportFile != "" {
log.Printf("Generating HTML report to: %s", cfg.ReportFile)
if err := report.GenerateHTMLReport(finalStats, cfg.ReportFile); err != nil {
log.Printf("Error generating report: %v", err)
} else {
log.Println("Report generated successfully.")
}
}
}
-38
View File
@@ -1,38 +0,0 @@
api:
endpoint: "http://139.9.229.200:8080/v1/chat/completions"
api_key: "devserver-9aB3-7fG5q2Yd-8C4T-1R6k-Ew9p-3MnL7bZx4Wv2"
model: "DeepSeek-R1"
prompt_templates:
short:
target_tokens: 50
templates:
- "什么是{concept}?"
- "请简要解释{concept}的基本原理。"
- "简述{topic}的主要观点。"
- "{country}的首都是哪里?"
- "如何快速入门{concept}?"
long:
target_tokens: 1000
templates:
- "请撰写一篇关于{country}历史的详细介绍,包括主要历史事件和文化发展。"
- "请对{topic}进行深入分析,包括其背景、现状、问题和未来发展趋势。"
- "请撰写一份关于{event}影响的综合报告,包括社会、经济和政治层面的分析。"
- "请详细介绍{concept}的技术原理、应用场景和未来发展方向。"
- "请撰写一篇关于{topic}的学术论文,包括研究背景、方法、结果和讨论。"
requests:
- type: "short"
weight: 0.7
- type: "long"
weight: 0.3
concurrency:
steps: [1,2]
duration_per_step: 60 # 秒
timeout: 60 # 秒
poisson_lambda: 1.0 # 请求间隔泊松分布参数
tokenizer:
model: "gpt-3.5-turbo" # 用于tiktoken-go的分词器模型
-103
View File
@@ -1,103 +0,0 @@
package config
import (
"fmt"
"github.com/spf13/viper"
)
// Config 表示应用程序的配置
type Config struct {
API APIConfig `mapstructure:"api"`
PromptTemplates PromptTemplatesConfig `mapstructure:"prompt_templates"`
Requests []RequestConfig `mapstructure:"requests"`
Concurrency ConcurrencyConfig `mapstructure:"concurrency"`
Timeout int `mapstructure:"timeout"`
PoissonLambda float64 `mapstructure:"poisson_lambda"`
Tokenizer TokenizerConfig `mapstructure:"tokenizer"`
}
// APIConfig 表示API相关配置
type APIConfig struct {
Endpoint string `mapstructure:"endpoint"`
APIKey string `mapstructure:"api_key"`
Model string `mapstructure:"model"`
}
// PromptTemplatesConfig 表示提示词模板配置
type PromptTemplatesConfig struct {
Short PromptTypeConfig `mapstructure:"short"`
Long PromptTypeConfig `mapstructure:"long"`
}
// PromptTypeConfig 表示特定类型提示词的配置
type PromptTypeConfig struct {
TargetTokens int `mapstructure:"target_tokens"`
Templates []string `mapstructure:"templates"`
}
// RequestConfig 表示请求配置
type RequestConfig struct {
Type string `mapstructure:"type"`
Weight float64 `mapstructure:"weight"`
}
// ConcurrencyConfig 表示并发配置
type ConcurrencyConfig struct {
Steps []int `mapstructure:"steps"`
DurationPerStep int `mapstructure:"duration_per_step"`
}
// TokenizerConfig 表示分词器配置
type TokenizerConfig struct {
Model string `mapstructure:"model"`
}
// LoadConfig 从指定路径加载配置
func LoadConfig(path string) (*Config, error) {
viper.SetConfigFile(path)
if err := viper.ReadInConfig(); err != nil {
return nil, fmt.Errorf("读取配置文件失败: %w", err)
}
var config Config
if err := viper.Unmarshal(&config); err != nil {
return nil, fmt.Errorf("解析配置失败: %w", err)
}
// 验证配置
if err := validateConfig(&config); err != nil {
return nil, err
}
return &config, nil
}
// validateConfig 验证配置的有效性
func validateConfig(config *Config) error {
if config.API.Endpoint == "" {
return fmt.Errorf("API端点不能为空")
}
if config.API.APIKey == "" {
return fmt.Errorf("API密钥不能为空")
}
if config.API.Model == "" {
return fmt.Errorf("模型名称不能为空")
}
if len(config.Requests) == 0 {
return fmt.Errorf("请求配置不能为空")
}
if len(config.Concurrency.Steps) == 0 {
return fmt.Errorf("并发步骤不能为空")
}
if config.Timeout <= 0 {
return fmt.Errorf("超时时间必须大于0")
}
return nil
}
+27
View File
@@ -0,0 +1,27 @@
# Basic Benchmark Configuration
apiEndpoint: "http://139.9.229.200:8080/v1/chat/completions" # Replace with your actual API endpoint
apiKey: "devserver-9aB3-7fG5q2Yd-8C4T-1R6k-Ew9p-3MnL7bZx4Wv2" # Replace with your actual API key or load from environment
model: "DeepSeek-R1" # Replace with the model you want to benchmark
requests: 10 # Total number of requests to send
concurrency: 5 # Number of concurrent workers
rateLimit: 0 # Requests per second (0 means no limit)
duration: 10s # Short duration for testing
promptTokens: 50 # Target number of tokens for the generated prompt (e.g., 50 or 1000)
maxTokens: 150 # Maximum tokens allowed in the response (if applicable)
streaming: true # Enable streaming responses
timeout: 30s # Timeout for each individual request
outputReport: "benchmark_report.html" # Filename for the generated HTML report
client: "fasthttp" # Choose the HTTP client: "fasthttp" (default) or "nethttp"
headers: # Optional: Custom HTTP headers
# Authorization: "Bearer YOUR_OTHER_TOKEN" # Example
# Custom-Header: "Value"
payload: # Optional: Custom fields to add/override in the request payload
# temperature: 0.7
# top_p: 0.9
-130
View File
@@ -1,130 +0,0 @@
# 配置文档
本文档详细说明了LLM API基准测试工具的配置选项。
## 配置文件格式
配置文件使用YAML格式,默认文件名为`config.yaml`。您可以通过命令行参数`-config`指定其他配置文件路径。
## 配置选项
### API配置
```yaml
api:
endpoint: "https://api.example.com/v1/completions" # API端点URL
api_key: "your_api_key" # API密钥
model: "gpt-3.5-turbo" # 模型名称
```
### 提示词模板配置
```yaml
prompt_templates:
short: # 短咨询提示词配置
target_tokens: 50 # 目标Token数(±5%)
templates: # 提示词模板列表
- "什么是{concept}?"
- "请简要解释{concept}的基本原理。"
- "简述{topic}的主要观点。"
long: # 长文档提示词配置
target_tokens: 1000 # 目标Token数(±5%)
templates: # 提示词模板列表
- "请撰写一篇关于{country}历史的详细介绍,包括主要历史事件和文化发展。"
- "请对{topic}进行深入分析,包括其背景、现状、问题和未来发展趋势。"
```
提示词模板中可以使用以下占位符:
- `{country}`: 国家名称
- `{concept}`: 概念名称
- `{topic}`: 主题名称
- `{event}`: 事件名称
系统会自动替换这些占位符,并根据目标Token数动态调整提示词长度。
### 请求配置
```yaml
requests:
- type: "short" # 提示词类型
weight: 0.7 # 权重(占比)
- type: "long"
weight: 0.3
```
权重总和应为1.0,表示各类型提示词的比例。
### 并发配置
```yaml
concurrency:
steps: [50, 200, 500] # 并发步骤
duration_per_step: 300 # 每个步骤的持续时间(秒)
```
系统会按照指定的并发步骤逐步增加并发用户数,每个步骤持续指定的时间。
### 超时配置
```yaml
timeout: 60 # 请求超时时间(秒)
```
### 泊松分布参数
```yaml
poisson_lambda: 1.0 # 请求间隔泊松分布参数
```
该参数控制请求间隔的随机性,模拟真实用户行为。
### 分词器配置
```yaml
tokenizer:
model: "gpt-3.5-turbo" # 用于tiktoken-go的分词器模型
```
## 完整配置示例
```yaml
api:
endpoint: "https://api.example.com/v1/completions"
api_key: "your_api_key"
model: "gpt-3.5-turbo"
prompt_templates:
short:
target_tokens: 50
templates:
- "什么是{concept}?"
- "请简要解释{concept}的基本原理。"
- "简述{topic}的主要观点。"
- "{country}的首都是哪里?"
- "如何快速入门{concept}?"
long:
target_tokens: 1000
templates:
- "请撰写一篇关于{country}历史的详细介绍,包括主要历史事件和文化发展。"
- "请对{topic}进行深入分析,包括其背景、现状、问题和未来发展趋势。"
- "请撰写一份关于{event}影响的综合报告,包括社会、经济和政治层面的分析。"
- "请详细介绍{concept}的技术原理、应用场景和未来发展方向。"
- "请撰写一篇关于{topic}的学术论文,包括研究背景、方法、结果和讨论。"
requests:
- type: "short"
weight: 0.7
- type: "long"
weight: 0.3
concurrency:
steps: [50, 200, 500]
duration_per_step: 300 # 秒
timeout: 60 # 秒
poisson_lambda: 1.0 # 请求间隔泊松分布参数
tokenizer:
model: "gpt-3.5-turbo" # 用于tiktoken-go的分词器模型
```
+231
View File
@@ -0,0 +1,231 @@
### 技术选型
- **语言**:Go。
- **HTTP客户端**:
- 抽象接口:`HTTPClient`。
- 实现:
- `fasthttp`(`github.com/valyala/fasthttp`):高性能,优先选择。
- `net/http`(标准库):兼容性强,备选。
- **Tokenizer 库**:`github.com/tiktoken-go/tokenizer`。
- **并发管理**:goroutine池、`sync.Pool`、`context.Context`。
- **数据处理**:`sync.Map`存储响应数据,`gonum/stat`计算百分位数。
- **图表生成**:`go-echarts`。
### 产品需求设计
#### 功能需求
- **API兼容性**:
- 支持OpenAI Compatible风格的LLM API(包括`stream=true`的SSE流式响应)。
- 兼容推理模型(如DeepSeek-R1)的响应结构,包含`reasoning_content`字段(与`content`同级)。
- **HTTP客户端抽象层**:
- 定义`HTTPClient`接口,包含以下方法:
- `Do(req *Request) (*Response, error)`:发送非流式请求。
- `Stream(req *Request, callback func(chunk SSEChunk) error) error`:处理流式响应,逐块回调SSE数据。
- 实现:
- `fasthttp`:基于`fasthttp.RequestCtx`和`Response.BodyStream`。
- `net/http`:基于`http.Client`和`Response.Body`。
- 配置项`client: "fasthttp"`或`client: "net-http"`切换客户端。
- **提示词生成**:
- 使用`tiktoken-go/tokenizer`动态生成短咨询提示词(50 tokens ±5%)和长文档提示词(1000 tokens ±5%)。
- **并发请求**:
- 支持最大500并发,持续5分钟(300秒)或更长时间。
- 使用goroutine池(最大600 goroutine),`sync.Pool`重用请求对象。
- 通过`context.Context`支持任务终止。
- **场景建模**:
- 混合负载:短咨询(50 tokens,70%比例),长文档生成(1000 tokens,30%比例)。
- **用户行为模拟**:请求间隔遵循泊松分布(`math/rand`实现)。
- **梯度增压**:支持阶梯式加压(50→200→500并发),每阶段持续300秒。
- **性能指标收集**:
- **请求统计**:总请求数、成功请求数、失败请求数、超时比率(目标:<1%,默认超时30-60秒)。
- **响应时间**:平均、最小/最大、P90/P95/P99(目标:<2秒至<10秒,<3秒至<15秒,<5秒至<20秒)。
- **TTFT**:
- 最小/最大TTFT,P90/P95/P99 TTFT。
- 通过SSE流式响应测量,精度达毫秒,兼容`content`和`reasoning_content`。
- **QPS**:平均QPS(目标:50-200+),最大QPS。
- **Token生成速率**:平均速率(目标:>100 tokens/秒),最大速率,兼容`content`和`reasoning_content`。
- **最大有效并发用户数**:记录仍有请求成功的最大并发数(目标:50-500+)。
- **报告生成**:
- 生成质量报告,包含所有指标统计。
- 提供图表(如响应时间分布、TTFT分布、QPS曲线)。
#### 非功能需求
- **高性能**:
- 支持500并发,持续5分钟以上,`fasthttp`优化吞吐量,goroutine池降低GC压力。
- **统计准确性**:
- TTFT和响应时间测量精度达毫秒,使用`time.Now().UnixNano()`。
- 使用`sync.Mutex`或`sync/atomic`确保并发统计线程安全。
- **可扩展性**:支持新服务提供商、提示词模板、响应字段。
- **易用性**:配置文件指定API、提示词、超时、客户端类型(`fasthttp`或`net-http`)。
- **稳定性**:长时间运行无崩溃、死锁或内存泄漏。
- **可观测性**:
- 使用`log`包记录关键事件和错误,通过`--debug`参数,开启调试日志,记录详细信息。
- 使用`runtime`包监控CPU、内存、goroutine数。
- 使用`sync.Map`存储响应数据,`gonum/stat`计算百分位数。
#### 技术实现建议
- **HTTP客户端抽象层**:
- 定义`HTTPClient`接口:
```go
type HTTPClient interface {
Do(req *Request) (*Response, error)
Stream(req *Request, callback func(chunk SSEChunk) error) error
}
type Request struct {
Method string
URL string
Body []byte
Headers map[string]string
}
type Response struct {
StatusCode int
Body []byte
Headers map[string]string
}
type SSEChunk struct {
Data []byte // JSON数据
Timestamp int64 // 接收时间戳(UnixNano)
IsDone bool // 是否为[data: [DONE]]
}
```
- **fasthttp实现**:
- `Do`:使用`fasthttp.Do`,构造OpenAI Compatible JSON请求,解析响应。
- `Stream`:使用`fasthttp.Response.BodyStream`,逐行解析SSE(`data: {...}`),调用回调函数传递`SSEChunk`。
- 使用`sync.Pool`重用`fasthttp.Request`和`fasthttp.Response`。
- **net/http实现**:
- `Do`:使用`http.Client.Do`,构造请求,解析响应。
- `Stream`:使用`bufio.Scanner`读取`http.Response.Body`,解析SSE,调用回调函数。
- **切换逻辑**:
- 解析配置项`client`,实例化对应客户端:
```go
func NewHTTPClient(clientType string) (HTTPClient, error) {
switch clientType {
case "fasthttp":
return &FastHTTPClient{}, nil
case "net-http":
return &NetHTTPClient{}, nil
default:
return nil, fmt.Errorf("unsupported client: %s", clientType)
}
}
```
- **SSE与TTFT处理**:
- `Stream`方法逐块解析SSE,提取`choices[0].delta.content`或`choices[0].delta.reasoning_content`。
- 记录请求发送时间(`t0 = time.Now().UnixNano()`)。
- 检测首个非空Token块,记录时间(`t1`),TTFT = (`t1 - t0`) / 1e6(毫秒)。
- 兼容推理模型:优先检查`reasoning_content`,若为空则使用`content`。
- **Go性能优化**:
- **Goroutine池**:使用`ants`库或自定义池,限制最大600 goroutine。
- **对象池**:`sync.Pool`重用请求/响应对象。
- **上下文控制**:`context.WithTimeout`设置压测时长(默认300秒)。
- **资源监控**:使用`runtime`包记录CPU、内存、goroutine数。
- **统计准确性**:
- 使用`sync/atomic`累加请求计数。
- 使用`sync.Mutex`保护响应时间和TTFT切片。
- 使用`gonum/stat`计算P90/P95/P99。
- **提示词生成**:`tiktoken-go/tokenizer`动态调整模板至50或1000 tokens。
- **报告生成**:`go-echarts`生成图表,模板引擎生成HTML/PDF。
#### 压测流程
1. **配置加载**:读取API端点、客户端类型(`fasthttp`或`net-http`)、并发数、提示词模板等。
2. **初始化HTTP客户端**:根据配置项`client`实例化`fasthttp`或`net/http`客户端。
3. **提示词生成**:使用`tiktoken-go/tokenizer`生成短咨询(50 tokens)和长文档(1000 tokens)提示词。
4. **并发执行**:
- 初始化goroutine池(最大600)。
- 使用`HTTPClient.Stream`发送请求(`stream=true`),间隔符合泊松分布。
- 处理SSE流,记录TTFT。
5. **响应处理**:
- 解析`content`和`reasoning_content`,计算Token数。
- 记录响应时间、TTFT、成功/失败状态。
6. **指标计算**:
- 计算请求统计、响应时间、TTFT、QPS、Token生成速率、最大并发用户数。
7. **报告生成**:输出统计数据和图表。
#### 配置示例与报告内容
以下是更新后的配置和报告示例,包含HTTP客户端抽象层和切换逻辑:
```yaml
# 配置示例
api:
endpoint: "https://api.example.com/v1/completions"
api_key: "your_api_key"
model: "deepseek-r1" # 支持推理模型
streaming: true # 启用流式响应
client: "fasthttp" # HTTP客户端:fasthttp 或 net-http
prompt_templates:
short:
target_tokens: 50
templates:
- "What is the capital of {country}?"
- "Please briefly explain the concept of {concept}."
- "Summarize the main idea of {topic} in one sentence."
long:
target_tokens: 1000
templates:
- "Write a detailed history of {country} covering major events."
- "Compose an in-depth analysis of {topic} with examples."
- "Generate a comprehensive report on the impact of {event}."
requests:
- type: "short"
weight: 0.7
- type: "long"
weight: 0.3
concurrency:
steps: [50, 200, 500]
duration_per_step: 300 # 秒,5分钟
max_goroutines: 600 # 最大goroutine数
timeout: 60 # 秒
poisson_lambda: 1.0 # 请求间隔泊松分布参数
tokenizer:
model: "gpt-3.5-turbo" # 用于tiktoken-go的分词器模型
# 报告内容示例
report:
overview:
api_endpoint: "https://api.example.com/v1/completions"
model: "deepseek-r1"
concurrency_steps: [50, 200, 500]
duration_per_step: "300s"
request_mix: "70% short (50 tokens), 30% long (1000 tokens)"
tokenizer: "tiktoken-go (gpt-3.5-turbo)"
streaming_enabled: true
http_client: "fasthttp"
metrics:
total_requests: 15000
successful_requests: 14850
failed_requests: 150
timeout_ratio: "1.0%"
response_time:
avg: "1.8s"
min: "0.5s"
max: "10.2s"
p90: "2.5s"
p95: "3.0s"
p99: "4.5s"
ttft:
min: "0.1s"
max: "2.0s"
p90: "0.3s"
p95: "0.4s"
p99: "0.6s"
qps:
avg: 180
max: 220
token_rate:
avg: "120 tokens/s"
max: "150 tokens/s"
max_concurrent_users: 500
resource_usage:
avg_cpu: "60%"
max_memory: "1.1GB"
max_goroutines: 600
charts:
- "response_time_distribution.png"
- "ttft_distribution.png"
- "qps_over_time.png"
- "token_rate_over_time.png"
- "concurrency_vs_response_time.png"
```
+25 -25
View File
@@ -1,38 +1,38 @@
module github.com/acmestudio/llm-api-benchmark-tool module llm-api-benchmark-tool
go 1.21 go 1.24.1
require ( require (
github.com/go-echarts/go-echarts/v2 v2.3.3 github.com/go-echarts/go-echarts/v2 v2.5.3
github.com/montanaflynn/stats v0.7.1 github.com/stretchr/testify v1.10.0
github.com/pkoukk/tiktoken-go v0.1.6 github.com/tiktoken-go/tokenizer v0.6.1
github.com/spf13/viper v1.18.2 github.com/valyala/fasthttp v1.61.0
github.com/valyala/fasthttp v1.52.0
) )
require ( require (
github.com/andybalholm/brotli v1.1.0 // indirect github.com/andybalholm/brotli v1.1.1 // indirect
github.com/dlclark/regexp2 v1.10.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect github.com/dlclark/regexp2 v1.11.5 // indirect
github.com/google/uuid v1.4.0 // indirect github.com/fsnotify/fsnotify v1.8.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect github.com/go-viper/mapstructure/v2 v2.2.1 // indirect
github.com/klauspost/compress v1.17.6 // indirect github.com/klauspost/compress v1.18.0 // indirect
github.com/magiconair/properties v1.8.7 // indirect github.com/kr/text v0.2.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect github.com/pelletier/go-toml/v2 v2.2.3 // indirect
github.com/pelletier/go-toml/v2 v2.1.0 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/sagikazarmark/locafero v0.4.0 // indirect github.com/sagikazarmark/locafero v0.7.0 // indirect
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.11.0 // indirect github.com/spf13/afero v1.12.0 // indirect
github.com/spf13/cast v1.6.0 // indirect github.com/spf13/cast v1.7.1 // indirect
github.com/spf13/pflag v1.0.5 // indirect github.com/spf13/pflag v1.0.6 // indirect
github.com/spf13/viper v1.20.1 // indirect
github.com/subosito/gotenv v1.6.0 // indirect github.com/subosito/gotenv v1.6.0 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect github.com/valyala/bytebufferpool v1.0.0 // indirect
go.uber.org/atomic v1.9.0 // indirect go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.9.0 // indirect go.uber.org/multierr v1.9.0 // indirect
golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect golang.org/x/sys v0.32.0 // indirect
golang.org/x/sys v0.17.0 // indirect golang.org/x/text v0.24.0 // indirect
golang.org/x/text v0.14.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect
) )
// Explicitly map the module path to the local directory
replace llm-api-benchmark-tool => ./
+43 -67
View File
@@ -1,87 +1,63 @@
github.com/andybalholm/brotli v1.1.0 h1:eLKJA0d02Lf0mVpIDgYnqXcUn0GqVmEFny3VuID1U3M= github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7XdTA=
github.com/andybalholm/brotli v1.1.0/go.mod h1:sms7XGricyQI9K10gOSf56VKKWS4oLer58Q+mhRPtnY= github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/dlclark/regexp2 v1.10.0 h1:+/GIL799phkJqYW+3YbOd8LCcbHzT0Pbo8zl70MHsq0= github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/8M=
github.com/dlclark/regexp2 v1.10.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8= github.com/fsnotify/fsnotify v1.8.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= github.com/go-echarts/go-echarts/v2 v2.5.3 h1:5SFAA6bAIWz52VnVGlCM1UZXo8nSdN+H9E8Ysi4m5ec=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0= github.com/go-echarts/go-echarts/v2 v2.5.3/go.mod h1:56YlvzhW/a+du15f3S2qUGNDfKnFOeJSThBIrVFHDtI=
github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= github.com/go-viper/mapstructure/v2 v2.2.1 h1:ZAaOCxANMuZx5RCeg0mBdEZk7DZasvvZIxtHqx8aGss=
github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= github.com/go-viper/mapstructure/v2 v2.2.1/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/go-echarts/go-echarts/v2 v2.3.3 h1:uImZAk6qLkC6F9ju6mZ5SPBqTyK8xjZKwSmwnCg4bxg= github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/go-echarts/go-echarts/v2 v2.3.3/go.mod h1:56YlvzhW/a+du15f3S2qUGNDfKnFOeJSThBIrVFHDtI= github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/google/uuid v1.4.0 h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4=
github.com/google/uuid v1.4.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/klauspost/compress v1.17.6 h1:60eq2E/jlfwQXtvZEeBUYADs+BwKBWURIY+Gj2eRGjI=
github.com/klauspost/compress v1.17.6/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY= github.com/pelletier/go-toml/v2 v2.2.3 h1:YmeHyLY8mFWbdkNWwpr+qIL2bEqT0o95WSdkNHvL12M=
github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0= github.com/pelletier/go-toml/v2 v2.2.3/go.mod h1:MfCQTFTvCcUyyvvwm1+G6H/jORL20Xlb6rzQu9GuUkc=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/montanaflynn/stats v0.7.1 h1:etflOAAHORrCC44V+aR6Ftzort912ZU+YLiSTuV8eaE=
github.com/montanaflynn/stats v0.7.1/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt6R8Bnaayow=
github.com/pelletier/go-toml/v2 v2.1.0 h1:FnwAJ4oYMvbT/34k9zzHuZNrhlz48GB3/s6at6/MHO4=
github.com/pelletier/go-toml/v2 v2.1.0/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc=
github.com/pkoukk/tiktoken-go v0.1.6 h1:JF0TlJzhTbrI30wCvFuiw6FzP2+/bR+FIxUdgEAcUsw=
github.com/pkoukk/tiktoken-go v0.1.6/go.mod h1:9NiV+i9mJKGj1rYOT+njbv+ZwA/zJxYdewGl6qVatpg=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/sagikazarmark/locafero v0.7.0 h1:5MqpDsTGNDhY8sGp0Aowyf0qKsPrhewaLSsFaodPcyo=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/sagikazarmark/locafero v0.7.0/go.mod h1:2za3Cg5rMaTMoG/2Ulr9AwtFaIppKXTRYnozin4aB5k=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/sagikazarmark/locafero v0.4.0 h1:HApY1R9zGo4DBgr7dqsTH/JJxLTTsOt7u6keLGt6kNQ=
github.com/sagikazarmark/locafero v0.4.0/go.mod h1:Pe1W6UlPYUk/+wc/6KFhbORCfqzgYEpgQ3O5fPuL3H4=
github.com/sagikazarmark/slog-shim v0.1.0 h1:diDBnUNK9N/354PgrxMywXnAwEr1QZcOr6gto+ugjYE=
github.com/sagikazarmark/slog-shim v0.1.0/go.mod h1:SrcSrq8aKtyuqEI1uvTDTK1arOWRIczQRv+GVI1AkeQ=
github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo= github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo=
github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0= github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0=
github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8= github.com/spf13/afero v1.12.0 h1:UcOPyRBYczmFn6yvphxkn9ZEOY65cpwGKb5mL36mrqs=
github.com/spf13/afero v1.11.0/go.mod h1:GH9Y3pIexgf1MTIWtNGyogA5MwRIDXGUr+hbWNoBjkY= github.com/spf13/afero v1.12.0/go.mod h1:ZTlWwG4/ahT8W7T0WQ5uYmjI9duaLQGy3Q2OAl4sk/4=
github.com/spf13/cast v1.6.0 h1:GEiTHELF+vaR5dhz3VqZfFSzZjYbgeKDpBxQVS4GYJ0= github.com/spf13/cast v1.7.1 h1:cuNEagBQEHWN1FnbGEjCXL2szYEXqfJPbP2HNUaca9Y=
github.com/spf13/cast v1.6.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo= github.com/spf13/cast v1.7.1/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.18.2 h1:LUXCnvUvSM6FXAsj6nnfc8Q2tp1dIgUfY9Kc8GsSOiQ= github.com/spf13/viper v1.20.1 h1:ZMi+z/lvLyPSCoNtFCpqjy0S4kPbirhpTMwl8BkW9X4=
github.com/spf13/viper v1.18.2/go.mod h1:EKmWIqdnk5lOcmR72yw6hS+8OPYcwD0jteitLMVB+yk= github.com/spf13/viper v1.20.1/go.mod h1:P9Mdzt1zoHIG8m2eZQinpiBjo6kCmZSKBClNNqjJvu4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8= github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU= github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/tiktoken-go/tokenizer v0.6.1 h1:iuhZJUEYY4Fn5vYbDX0Clgqzus8w8rM410YbovUzyLM=
github.com/tiktoken-go/tokenizer v0.6.1/go.mod h1:6UCYI/DtOallbmL7sSy30p6YQv60qNyU/4aVigPOx6w=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw= github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc= github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.52.0 h1:wqBQpxH71XW0e2g+Og4dzQM8pk34aFYlA1Ga8db7gU0= github.com/valyala/fasthttp v1.61.0 h1:VV08V0AfoRaFurP1EWKvQQdPTZHiUzaVoulX1aBDgzU=
github.com/valyala/fasthttp v1.52.0/go.mod h1:hf5C4QnVMkNXMspnsUlfM3WitlgYflyhHYoKol/szxQ= github.com/valyala/fasthttp v1.61.0/go.mod h1:wRIV/4cMwUPWnRcDno9hGnYZGh78QzODFfo1LTUhBog=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE= go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE=
go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI= go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI=
go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ= go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ=
golang.org/x/exp v0.0.0-20230905200255-921286631fa9 h1:GoHiUyI/Tp2nVkLI2mCxVkOjsbSXD66ic0XW0js0R9g= golang.org/x/sys v0.32.0 h1:s77OFDvIQeibCmezSnk/q6iAfkdiQaJi4VzroCFrN20=
golang.org/x/exp v0.0.0-20230905200255-921286631fa9/go.mod h1:S2oDrQGGwySpoQPVqRShND87VCbxmc6bL1Yd2oYrm6k= golang.org/x/sys v0.32.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/sys v0.17.0 h1:25cE3gD+tdBA7lp7QfhuV+rJiE9YXTcS3VG1SqssI/Y= golang.org/x/text v0.24.0 h1:dd5Bzh4yt5KYA8f9CJHCP4FB4D51c2c6JvN37xJJkJ0=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/text v0.24.0/go.mod h1:L8rBsPeo2pSS+xqN0d5u2ikmjtmoJbDBT1b7nHvFCdU=
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
BIN
View File
Binary file not shown.
-200
View File
@@ -1,200 +0,0 @@
package logger
import (
"fmt"
"log"
"os"
"strings"
"time"
)
// 日志级别
const (
DEBUG = iota
INFO
WARN
ERROR
FATAL
)
// 日志级别名称
var levelNames = []string{
"调试",
"信息",
"警告",
"错误",
"致命",
}
// 日志级别颜色
var levelColors = []string{
"\033[36m", // 青色 - 调试
"\033[32m", // 绿色 - 信息
"\033[33m", // 黄色 - 警告
"\033[31m", // 红色 - 错误
"\033[35m", // 紫色 - 致命
}
// 重置颜色
const resetColor = "\033[0m"
// Logger 日志记录器
type Logger struct {
level int
logger *log.Logger
useColors bool
timeFormat string
}
// 全局日志记录器
var defaultLogger *Logger
// 初始化默认日志记录器
func init() {
defaultLogger = NewLogger(INFO, true)
}
// NewLogger 创建新的日志记录器
func NewLogger(level int, useColors bool) *Logger {
return &Logger{
level: level,
logger: log.New(os.Stdout, "", 0),
useColors: useColors,
timeFormat: "2006-01-02 15:04:05.000",
}
}
// SetLevel 设置日志级别
func SetLevel(level int) {
defaultLogger.level = level
}
// SetUseColors 设置是否使用颜色
func SetUseColors(useColors bool) {
defaultLogger.useColors = useColors
}
// formatLog 格式化日志消息
func (l *Logger) formatLog(level int, format string, args ...interface{}) string {
// 格式化消息
var msg string
if len(args) > 0 {
msg = fmt.Sprintf(format, args...)
} else {
msg = format
}
// 获取当前时间
now := time.Now().Format(l.timeFormat)
// 获取级别名称
levelName := levelNames[level]
// 构建日志前缀
prefix := fmt.Sprintf("[%s] [%s] ", now, levelName)
// 添加颜色
if l.useColors {
prefix = levelColors[level] + prefix
msg = msg + resetColor
}
// 处理多行消息,确保每行都有正确的前缀
lines := strings.Split(msg, "\n")
for i := 1; i < len(lines); i++ {
if len(lines[i]) > 0 {
padding := strings.Repeat(" ", len(now)+3) // +3 for "[] "
lines[i] = fmt.Sprintf("%s%s", padding, lines[i])
}
}
return prefix + strings.Join(lines, "\n")
}
// log 记录日志
func (l *Logger) log(level int, format string, args ...interface{}) {
if level >= l.level {
l.logger.Println(l.formatLog(level, format, args...))
}
}
// Debug 调试日志
func (l *Logger) Debug(format string, args ...interface{}) {
l.log(DEBUG, format, args...)
}
// Info 信息日志
func (l *Logger) Info(format string, args ...interface{}) {
l.log(INFO, format, args...)
}
// Warn 警告日志
func (l *Logger) Warn(format string, args ...interface{}) {
l.log(WARN, format, args...)
}
// Error 错误日志
func (l *Logger) Error(format string, args ...interface{}) {
l.log(ERROR, format, args...)
}
// Fatal 致命日志
func (l *Logger) Fatal(format string, args ...interface{}) {
l.log(FATAL, format, args...)
os.Exit(1)
}
// Debug 调试日志
func Debug(format string, args ...interface{}) {
defaultLogger.Debug(format, args...)
}
// Info 信息日志
func Info(format string, args ...interface{}) {
defaultLogger.Info(format, args...)
}
// Warn 警告日志
func Warn(format string, args ...interface{}) {
defaultLogger.Warn(format, args...)
}
// Error 错误日志
func Error(format string, args ...interface{}) {
defaultLogger.Error(format, args...)
}
// Fatal 致命日志
func Fatal(format string, args ...interface{}) {
defaultLogger.Fatal(format, args...)
}
// 进度条相关
const (
progressBarWidth = 50
)
// PrintProgressBar 打印进度条
func PrintProgressBar(current, total int, prefix string) {
percentage := float64(current) / float64(total)
filled := int(progressBarWidth * percentage)
empty := progressBarWidth - filled
// 构建进度条
bar := fmt.Sprintf("\r%s [%s%s] %.1f%% (%d/%d)",
prefix,
strings.Repeat("█", filled),
strings.Repeat("░", empty),
percentage*100,
current,
total,
)
// 打印进度条(不换行)
fmt.Print(bar)
// 如果完成,打印换行
if current == total {
fmt.Println()
}
}
-85
View File
@@ -1,85 +0,0 @@
package main
import (
"flag"
"fmt"
"os"
"time"
"github.com/acmestudio/llm-api-benchmark-tool/config"
"github.com/acmestudio/llm-api-benchmark-tool/benchmark"
"github.com/acmestudio/llm-api-benchmark-tool/report"
"github.com/acmestudio/llm-api-benchmark-tool/logger"
)
func main() {
// 解析命令行参数
configPath := flag.String("config", "config.yaml", "配置文件路径")
outputDir := flag.String("output", "reports", "报告输出目录")
debugMode := flag.Bool("debug", false, "启用调试模式")
noColor := flag.Bool("no-color", false, "禁用彩色日志输出")
flag.Parse()
// 配置日志级别
if *debugMode {
logger.SetLevel(logger.DEBUG)
} else {
logger.SetLevel(logger.INFO)
}
// 配置日志颜色
logger.SetUseColors(!*noColor)
logger.Info("LLM API基准测试工具启动")
logger.Info("版本: v0.1.0")
logger.Debug("调试模式已启用")
// 加载配置
logger.Info("正在加载配置文件: %s", *configPath)
cfg, err := config.LoadConfig(*configPath)
if err != nil {
logger.Fatal("加载配置失败: %v", err)
}
logger.Info("配置加载成功")
logger.Debug("API端点: %s", cfg.API.Endpoint)
logger.Debug("模型: %s", cfg.API.Model)
logger.Debug("并发步骤: %v", cfg.Concurrency.Steps)
logger.Debug("每步持续时间: %d秒", cfg.Concurrency.DurationPerStep)
// 创建输出目录
if err := os.MkdirAll(*outputDir, 0755); err != nil {
logger.Fatal("创建输出目录失败: %v", err)
}
logger.Debug("输出目录已创建: %s", *outputDir)
// 初始化基准测试
logger.Info("正在初始化基准测试...")
bench, err := benchmark.NewBenchmark(cfg)
if err != nil {
logger.Fatal("初始化基准测试失败: %v", err)
}
logger.Info("基准测试初始化成功")
// 运行基准测试
logger.Info("开始运行LLM API基准测试...")
startTime := time.Now()
results, err := bench.Run()
if err != nil {
logger.Fatal("运行基准测试失败: %v", err)
}
duration := time.Since(startTime)
logger.Info("基准测试完成,总耗时: %v", duration)
// 生成报告
logger.Info("正在生成性能报告...")
reportTime := time.Now().Format("2006-01-02_15-04-05")
reportPath := fmt.Sprintf("%s/report_%s", *outputDir, reportTime)
if err := report.GenerateReport(results, cfg, reportPath); err != nil {
logger.Fatal("生成报告失败: %v", err)
}
logger.Info("基准测试完成!报告已生成: %s", reportPath)
logger.Info("HTML报告: %s/report.html", reportPath)
logger.Info("CSV数据: %s/results.csv", reportPath)
}
+21
View File
@@ -0,0 +1,21 @@
# Active Context (As of 2025-04-23 ~08:04 UTC+8)
## 1. Current Focus
Refining the core benchmarking logic and ensuring accurate metric collection, particularly after implementing and debugging streaming support.
## 2. Recent Changes
- **Fixed Stats Recording:** Correctly implemented the recording of `stats.RequestResult` after each request (both streaming and non-streaming) completes within the `runWorker` function in `pkg/concurrency/manager.go`. This involved moving the `RecordResult` call inside the appropriate conditional blocks and using the correct field names (`IsSuccess`, `TimeToFirstToken`).
- **Corrected Client Initialization:** Updated `cmd/benchmark/main.go` to correctly initialize the `client.FastHTTPClient`. Removed the flawed logic that attempted to switch client types based on the `streaming` flag, recognizing that `FastHTTPClient` internally handles both streaming (using `net/http`) and non-streaming (using `fasthttp`). Ensured the correct request timeout value is passed during initialization.
- **Lint Fixes:** Addressed several lint warnings, including unused imports and tautological conditions.
## 3. Next Steps & Decisions
- The core functionality of running concurrent streaming requests and recording basic results (Success, Latency, TTFT) is working.
- The immediate next step is to choose which refinement to tackle next from the following options:
1. Add detailed error recording to `RequestResult`.
2. Add token statistics (e.g., average tokens/request, total tokens) to the final report.
3. Improve/clarify TTFT calculation for non-streaming requests.
4. Refactor `runWorker` in `pkg/concurrency/manager.go` for better readability.
## 4. Active Considerations
- Ensuring the `net/http` client used internally for streaming has appropriate timeout settings (currently hardcoded in `Stream` method).
- Planning the structure and implementation of the final report generation (console and potentially HTML).
+21
View File
@@ -0,0 +1,21 @@
# Product Context: LLM API Benchmark Tool
## 1. Problem Solved
Evaluating and comparing the performance characteristics (speed, throughput, responsiveness) of different LLM APIs or different configurations of the same API can be difficult and time-consuming. This tool aims to provide a standardized and automated way to perform these benchmarks.
## 2. Target Users
- Developers integrating LLM APIs who need to choose the best performing option.
- Operations teams monitoring LLM API performance and reliability.
- Researchers comparing different LLM models or hosting infrastructures.
## 3. How it Should Work
- The user provides a `config.yaml` file specifying the target API, credentials, benchmark parameters (duration, concurrency, etc.), and desired prompt characteristics.
- The tool runs the benchmark according to the configuration, sending requests and collecting performance data.
- Upon completion (or interruption), the tool calculates and displays summary statistics for the key metrics.
- Streaming requests should accurately measure TTFT and token throughput.
## 4. User Experience Goals
- **Easy Configuration:** Simple and clear YAML configuration.
- **Clear Output:** Understandable summary report of key metrics.
- **Reliable Measurement:** Accurate and consistent performance data collection, especially for streaming TTFT.
- **Informative Logging:** Useful logs during the benchmark run for diagnostics.
+31
View File
@@ -0,0 +1,31 @@
# Progress (As of 2025-04-23 ~08:04 UTC+8)
## 1. What Works
- **Core Benchmark Execution:** Can run concurrent requests (both streaming and non-streaming modes selectable via config) for a specified duration.
- **Configuration Loading:** Reads parameters from `config.yaml`.
- **Concurrency Control:** Limits active workers based on the `concurrency` setting.
- **HTTP Client:** `FastHTTPClient` handles both request types:
- Non-streaming (`Do`) uses `fasthttp`.
- Streaming (`Stream`) uses `net/http` internally and processes SSE events.
- **Tokenizer Integration:** Generates prompts and counts response tokens.
- **Basic Stats Collection:**
- Records individual `RequestResult` (IsSuccess, Latency, TimeToFirstToken, TotalTokens) correctly for both streaming and non-streaming requests.
- Calculates and displays aggregate stats upon completion: Total Requests, Success/Fail counts, Success Rate, Avg QPS, Latency (Avg/Min/Max/Percentiles), TTFT (Avg/Min/Max/Percentiles), Avg Tokens/Second.
## 2. What's Left / Needs Refinement
- **Error Reporting:** Currently only tracks success/failure boolean. Need to capture and potentially report specific error messages (`RequestResult.Error` field).
- **Token Statistics:** While individual token counts are recorded, aggregate token stats (e.g., average tokens per successful request, total tokens generated) are not yet calculated or displayed in the final report.
- **Non-Streaming TTFT:** The current TTFT for non-streaming requests is approximated as the total latency. This might need refinement or clearer definition.
- **Streaming Timeout:** The timeout for the entire streaming request is hardcoded in `client.Stream`. Consider making this configurable.
- **Report Generation:** Final report is currently basic console output. Planning to implement HTML report generation (possibly using `go-echarts`).
- **Code Refinement:** `pkg/concurrency/manager.go`'s `runWorker` could be refactored for better readability/maintainability.
- **Testing:** Need comprehensive integration tests covering various config scenarios (especially different prompt sizes, rate limits, etc.). Existing unit tests cover client and potentially other modules.
## 3. Current Status
- The main bug related to recording streaming results has been fixed.
- The client initialization logic is corrected.
- The tool is functional for basic benchmarking scenarios, especially streaming.
- Ready to proceed with further feature enhancements or refinements.
## 4. Known Issues
- None critical at the moment. Previous lint warnings have been addressed.
+20
View File
@@ -0,0 +1,20 @@
# Project Brief: LLM API Benchmark Tool
## 1. Core Goal
Develop a command-line tool in Go to benchmark the performance of Large Language Model (LLM) APIs.
## 2. Key Features
- Send configurable concurrent requests (streaming and non-streaming) to a specified LLM API endpoint.
- Measure and report key performance metrics:
- Request Success/Failure Rate
- Queries Per Second (QPS)
- Latency (min, max, average, percentiles)
- Time To First Token (TTFT) for streaming and non-streaming (min, max, average, percentiles)
- Token generation rate (tokens/second, average)
- Support configurable parameters via a YAML file (API endpoint, key, model, concurrency, duration, rate limit, prompt size, streaming mode, timeouts, etc.).
- Generate a summary report (initially console output, potentially HTML later).
## 3. Scope
- Focus on HTTP-based LLM APIs (e.g., OpenAI compatible).
- Initial implementation targets core metrics.
- Extensibility for different client implementations or reporting formats is desirable.
+65
View File
@@ -0,0 +1,65 @@
# System Patterns
## 1. Overall Architecture
- Modular Go application following standard Go project layout (`cmd`, `pkg`, `test`).
- Configuration-driven behavior using YAML.
- Clear separation of concerns between packages:
- `cmd/benchmark`: Main application entry point, CLI flags, orchestrates initialization.
- `pkg/config`: Loading and validation of `config.yaml`.
- `pkg/client`: Abstraction layer for HTTP requests.
- `pkg/tokenizer`: Handles prompt generation and token counting.
- `pkg/concurrency`: Manages concurrent workers and request scheduling.
- `pkg/stats`: Collects and aggregates performance metrics.
- `pkg/report`: (Planned) Generates final benchmark reports.
## 2. Key Technical Decisions & Patterns
- **HTTP Client Abstraction:**
- `HTTPClient` interface (`pkg/client/interface.go`) defines `Do` (non-streaming) and `Stream` methods.
- `FastHTTPClient` (`pkg/client/fasthttp.go`) is the primary implementation:
- Uses `fasthttp` library for high performance in the `Do` method.
- Uses Go's standard `net/http` library internally for the `Stream` method for better SSE handling stability.
- Uses `sync.Pool` for `fasthttp` request/response object reuse.
- **Concurrency Management:**
- `pkg/concurrency/manager.go` uses a goroutine pool (implemented via channels/semaphore pattern) to limit concurrent workers based on `config.Concurrency`.
- Uses `context.Context` for managing benchmark duration and worker cancellation.
- **Statistics Collection:**
- `pkg/stats/collector.go` uses a `sync.Map` for thread-safe storage of individual `RequestResult` structs.
- Calculates final aggregate statistics (average, min/max, percentiles) upon benchmark completion.
- Percentiles calculated using `gonum/stat`.
- **Token Handling:**
- Uses `github.com/tiktoken-go/tokenizer` for generating prompts of specific token lengths and counting tokens in responses.
- **Configuration:**
- Uses `viper` (or similar YAML parsing library) within `pkg/config` to load settings.
## 3. Component Relationships
```mermaid
flowchart TD
A[cmd/benchmark/main.go] --> B(pkg/config)
A --> C{client.HTTPClient}
A --> D(pkg/stats)
A --> E(pkg/concurrency)
A --> F(pkg/tokenizer)
C -->|Do uses| G[fasthttp]
C -->|Stream uses| H[net/http]
E --> C
E --> F
E --> D
subgraph pkg/client
C
G
H
end
subgraph pkg/stats
D
I[gonum/stat]
end
subgraph pkg/tokenizer
F
J[tiktoken-go]
end
```
+33
View File
@@ -0,0 +1,33 @@
# Technical Context
## 1. Core Technology
- **Language:** Go (Golang)
## 2. Key Libraries & Dependencies
- **HTTP:**
- `github.com/valyala/fasthttp`: For high-performance non-streaming requests.
- `net/http`: Go standard library, used for streaming (SSE) requests within the client implementation.
- **Tokenization:**
- `github.com/tiktoken-go/tokenizer`: For OpenAI-compatible token counting and prompt generation helper functions.
- **Concurrency:**
- Go standard library (`sync`, `context`, goroutines, channels).
- **Statistics:**
- `gonum.org/v1/gonum/stat`: For percentile calculations.
- `sync/atomic`: Used within stats collector for thread-safe counters.
- `sync.Map`: For thread-safe storage of request results.
- **Configuration:**
- `github.com/spf13/viper`: Likely used for loading YAML configuration (verify import).
- `gopkg.in/yaml.v3` (or similar): For YAML parsing.
- **CLI Flags:**
- Go standard library `flag`.
## 3. Development Environment
- **OS:** macOS (as per user information)
- **Build/Dependency Management:** Go Modules (`go mod`)
## 4. Technical Constraints & Considerations
- **HTTP Client Choice:** Balancing performance (`fasthttp`) and streaming stability (`net/http`). The current approach uses a hybrid within `FastHTTPClient`.
- **Concurrency Limits:** Managing goroutine lifecycle and preventing resource exhaustion.
- **Accurate Timing:** Measuring latency and TTFT precisely, especially across concurrent operations.
- **Memory Management:** Using `sync.Pool` for `fasthttp` objects; being mindful of memory usage when handling large responses or many results.
- **TDD:** Workspace rules mandate Test-Driven Development.
+196
View File
@@ -0,0 +1,196 @@
package client
import (
"bufio"
"bytes"
"context"
"fmt"
"io"
"net/http"
"sync"
"time"
"github.com/valyala/fasthttp"
)
// FastHTTPClient implements the HTTPClient interface using fasthttp for non-streaming
// and net/http for streaming, with configurable request timeout for non-streaming.
type FastHTTPClient struct {
reqPool sync.Pool
respPool sync.Pool
stdClient *http.Client // Standard client for streaming
requestTimeout time.Duration
}
// NewFastHTTPClient creates a new FastHTTPClient.
func NewFastHTTPClient(requestTimeout time.Duration) *FastHTTPClient {
return &FastHTTPClient{
reqPool: sync.Pool{
New: func() interface{} {
return fasthttp.AcquireRequest()
},
},
respPool: sync.Pool{
New: func() interface{} {
return fasthttp.AcquireResponse()
},
},
stdClient: &http.Client{
// Configure standard client (e.g., transport, timeout)
Timeout: 30 * time.Second, // Example timeout
},
requestTimeout: requestTimeout,
}
}
// Do sends a non-streaming request using fasthttp.
func (c *FastHTTPClient) Do(req *Request) (*Response, error) {
// 1. Acquire fasthttp request from pool
freq := c.reqPool.Get().(*fasthttp.Request)
defer c.reqPool.Put(freq) // Ensure request is put back even on error
freq.Reset()
// 2. Populate fasthttp request from input *Request
freq.SetRequestURI(req.URL)
freq.Header.SetMethod(req.Method)
if req.Body != nil {
freq.SetBodyRaw(req.Body) // Use SetBodyRaw to avoid copying
}
for k, v := range req.Headers {
freq.Header.Set(k, v)
}
// 3. Acquire fasthttp response from pool
fresp := c.respPool.Get().(*fasthttp.Response)
defer c.respPool.Put(fresp) // Ensure response is put back
fresp.Reset()
// 4. Perform the request
// Ensure request/response are released even if DoTimeout panics (though unlikely)
// Place deferred release calls *before* the operation that might fail
defer fasthttp.ReleaseRequest(freq)
defer fasthttp.ReleaseResponse(fresp)
// Perform the request using DoTimeout
err := fasthttp.DoTimeout(freq, fresp, c.requestTimeout)
if err != nil {
// Don't release again here, defer handles it
return nil, fmt.Errorf("fasthttp client timeout or connection error for %s %s: %w", req.Method, req.URL, err)
}
// 5. Populate output *Response from fasthttp response
resp := &Response{
StatusCode: fresp.StatusCode(),
Body: append([]byte(nil), fresp.Body()...), // Important: Copy body as fresp will be reused
Headers: make(map[string]string),
}
fresp.Header.VisitAll(func(key, value []byte) {
resp.Headers[string(key)] = string(value)
})
return resp, nil
}
// Stream handles streaming SSE requests using the standard net/http client.
func (c *FastHTTPClient) Stream(req *Request, callback func(chunk SSEChunk) error) error {
// 1. Create standard http request
var bodyReader io.Reader
if req.Body != nil {
bodyReader = bytes.NewReader(req.Body)
}
// Use context for potential cancellation/timeout
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) // Example: 60s timeout for the whole stream
defer cancel()
stdReq, err := http.NewRequestWithContext(ctx, req.Method, req.URL, bodyReader)
if err != nil {
return fmt.Errorf("failed to create standard http request for %s: %w", req.URL, err)
}
// Copy headers
for k, v := range req.Headers {
stdReq.Header.Set(k, v)
}
// Ensure Content-Type is set if needed (and not already set)
if bodyReader != nil && stdReq.Header.Get("Content-Type") == "" {
stdReq.Header.Set("Content-Type", "application/json")
}
// Ensure Accept header is set for SSE
if stdReq.Header.Get("Accept") == "" {
stdReq.Header.Set("Accept", "text/event-stream")
}
// Ensure Connection header for streaming
if stdReq.Header.Get("Connection") == "" {
stdReq.Header.Set("Connection", "keep-alive")
}
// 2. Perform the request
resp, err := c.stdClient.Do(stdReq)
if err != nil {
return fmt.Errorf("standard http client Do failed for %s: %w", req.URL, err)
}
defer resp.Body.Close() // Ensure body is closed
// 3. Check status code
if resp.StatusCode != http.StatusOK {
bodyBytes, readErr := io.ReadAll(resp.Body)
if readErr != nil {
return fmt.Errorf("http stream error status %d for %s, failed to read body: %w", resp.StatusCode, req.URL, readErr)
}
return fmt.Errorf("http stream error status %d for %s: %s", resp.StatusCode, req.URL, string(bodyBytes))
}
// 4. Process the stream
scanner := bufio.NewScanner(resp.Body)
ssePrefix := []byte("data: ")
var processingErr error
for scanner.Scan() {
line := scanner.Bytes()
if !bytes.HasPrefix(line, ssePrefix) {
continue
}
data := bytes.TrimPrefix(line, ssePrefix)
ts := time.Now().UnixNano()
var isDone bool
if bytes.Equal(data, []byte("[DONE]")) {
isDone = true
}
dataCopy := make([]byte, len(data))
copy(dataCopy, data)
chunk := SSEChunk{
Data: dataCopy,
Timestamp: ts,
IsDone: isDone,
}
if err := callback(chunk); err != nil {
if processingErr == nil {
processingErr = fmt.Errorf("sse stream user callback error: %w", err)
}
// Stop processing loop on user callback error
break
}
if isDone {
break
}
}
// Check for scanner errors
if err := scanner.Err(); err != nil {
if processingErr == nil {
processingErr = fmt.Errorf("error reading sse stream for %s: %w", req.URL, err)
}
}
// Return processing error if any occurred during scan or callback
if processingErr != nil {
return processingErr
}
return nil
}
+34
View File
@@ -0,0 +1,34 @@
package client
// HTTPClient defines the interface for sending HTTP requests.
// It supports both standard non-streaming and streaming (SSE) requests.
type HTTPClient interface {
// Do sends a standard non-streaming HTTP request.
Do(req *Request) (*Response, error)
// Stream sends an HTTP request and processes the response as a stream of Server-Sent Events (SSE).
// It calls the callback function for each received SSE chunk.
Stream(req *Request, callback func(chunk SSEChunk) error) error
}
// Request represents an HTTP request to be sent.
type Request struct {
Method string // HTTP method (e.g., "POST", "GET")
URL string // Request URL
Body []byte // Request body
Headers map[string]string // Request headers
IsStream bool // Indicates if this request expects a streaming response
}
// Response represents an HTTP response received from the server.
type Response struct {
StatusCode int // HTTP status code (e.g., 200, 404)
Body []byte // Response body (for non-streaming requests)
Headers map[string]string // Response headers
}
// SSEChunk represents a single chunk of data received in an SSE stream.
type SSEChunk struct {
Data []byte // Raw data content of the SSE event (typically JSON)
Timestamp int64 // Timestamp when the chunk was received (UnixNano)
IsDone bool // Flag indicating if this chunk represents the end of the stream (e.g., "data: [DONE]")
}
+308
View File
@@ -0,0 +1,308 @@
package concurrency
import (
"context"
"encoding/json"
"log"
"sync"
"time"
"llm-api-benchmark-tool/pkg/client"
"llm-api-benchmark-tool/pkg/stats"
"llm-api-benchmark-tool/pkg/tokenizer"
)
// BenchmarkParams holds the parameters for the benchmark run.
type BenchmarkParams struct {
HTTPClient client.HTTPClient // The HTTP client to use for requests
TargetURL string // The target API endpoint
APIKey string // API Key for authorization
Model string // Model name to use
PromptTokens int // Number of tokens for the generated prompt
TotalDuration time.Duration // How long the benchmark should run
Concurrency int // Number of concurrent workers/requests allowed simultaneously
RequestInterval time.Duration // Minimum delay between starting *any* two requests globally
IsStreaming bool // Indicates if the request expects a streaming response
Payload map[string]interface{} // Optional: Custom payload fields
}
// Manager coordinates the benchmark execution based on the parameters.
type Manager struct {
params BenchmarkParams
statsCollector *stats.StatsCollector
}
// NewManager creates a new concurrency manager.
func NewManager(params BenchmarkParams, statsCollector *stats.StatsCollector) *Manager {
return &Manager{
params: params,
statsCollector: statsCollector,
}
}
// RunBenchmark executes the benchmark according to the configured parameters.
// It uses the provided context for cancellation.
func (m *Manager) RunBenchmark(ctx context.Context) error {
log.Printf("RunBenchmark called. RateLimit: %d", m.params.RequestInterval)
if m.params.RequestInterval > 0 {
return m.runBenchmarkWithTicker(ctx)
}
return m.runBenchmarkWithoutTicker(ctx)
}
// runBenchmarkWithTicker executes the benchmark with a fixed rate limit.
func (m *Manager) runBenchmarkWithTicker(ctx context.Context) error {
log.Printf("Starting runBenchmarkWithTicker. Duration: %s, RateLimit: %d, Concurrency: %d",
m.params.TotalDuration, m.params.RequestInterval, m.params.Concurrency)
runCtx, cancel := context.WithTimeout(ctx, m.params.TotalDuration)
defer cancel()
var wg sync.WaitGroup
concurrencyLimiter := make(chan struct{}, m.params.Concurrency)
ticker := time.NewTicker(m.params.RequestInterval)
defer ticker.Stop()
log.Printf("Ticker interval: %s", m.params.RequestInterval)
wg.Add(1)
go func() {
defer wg.Done()
requestCounter := 0
for {
select {
case <-runCtx.Done():
return
case <-ticker.C:
select {
case concurrencyLimiter <- struct{}{}:
wg.Add(1)
go func(workerID int) {
defer wg.Done()
defer func() { <-concurrencyLimiter }()
if runCtx.Err() != nil {
return
}
promptText, err := tokenizer.GeneratePrompt(m.params.PromptTokens)
if err != nil {
log.Printf("Worker: Failed to generate prompt: %v\n", err)
return
}
req := m.createRequest(promptText)
startTime := time.Now()
var result stats.RequestResult
log.Printf("Worker %d making request (Streaming: %v)...", workerID, m.params.IsStreaming)
if m.params.IsStreaming {
var firstChunkTime time.Time
var totalTokens int
err = m.params.HTTPClient.Stream(req, func(chunk client.SSEChunk) error {
if firstChunkTime.IsZero() {
firstChunkTime = time.Now()
}
chunkTokens, err := tokenizer.CountTokensInText(string(chunk.Data))
if err != nil {
log.Printf("Worker %d: Error counting stream chunk tokens: %v", workerID, err)
} else {
totalTokens += chunkTokens
}
return nil
})
latency := time.Since(startTime)
result = stats.RequestResult{
IsSuccess: err == nil,
Latency: latency,
TimeToFirstToken: firstChunkTime.Sub(startTime),
TotalTokens: totalTokens,
}
log.Printf("Worker %d finished streaming request. Success: %v", workerID, result.IsSuccess)
m.statsCollector.RecordResult(result)
} else {
resp, err := m.params.HTTPClient.Do(req)
latency := time.Since(startTime)
var statusCode int
var totalTokens int
if resp != nil {
statusCode = resp.StatusCode
bodyBytes := resp.Body
bodyString := string(bodyBytes)
totalTokens, err = tokenizer.CountTokensInText(bodyString)
if err != nil {
log.Printf("Worker %d: Error counting non-stream response tokens: %v", workerID, err)
}
}
result = stats.RequestResult{
IsSuccess: err == nil && statusCode >= 200 && statusCode < 300,
Latency: latency,
TimeToFirstToken: latency,
TotalTokens: totalTokens,
}
log.Printf("Worker %d finished non-streaming request. Success: %v, Status: %d", workerID, result.IsSuccess, statusCode)
m.statsCollector.RecordResult(result)
}
}(requestCounter)
requestCounter++
default:
log.Printf("Skipping ticker event, concurrency limiter full (Size: %d)", len(concurrencyLimiter))
}
}
}
}()
wg.Wait()
if runCtx.Err() == context.DeadlineExceeded {
return nil
}
log.Printf("Ticker stopped. Waiting for %d workers to finish outstanding requests...", m.params.Concurrency)
wg.Wait()
return runCtx.Err()
}
// runBenchmarkWithoutTicker handles the benchmark execution when no rate limiting (RequestInterval <= 0) is needed.
// It runs requests as fast as possible, respecting the concurrency limit.
func (m *Manager) runBenchmarkWithoutTicker(ctx context.Context) error {
log.Printf("Starting runBenchmarkWithoutTicker. Duration: %s, Concurrency: %d",
m.params.TotalDuration, m.params.Concurrency)
runCtx, cancel := context.WithTimeout(ctx, m.params.TotalDuration)
defer cancel()
var wg sync.WaitGroup
concurrencyLimiter := make(chan struct{}, m.params.Concurrency)
for i := 0; i < m.params.Concurrency; i++ {
concurrencyLimiter <- struct{}{}
}
log.Printf("Starting %d workers", m.params.Concurrency)
for i := 0; i < m.params.Concurrency; i++ {
wg.Add(1)
go func(workerID int) {
defer wg.Done()
log.Printf("Worker %d started", workerID)
for {
select {
case <-runCtx.Done():
log.Printf("Worker %d exiting (context done before acquiring slot).", workerID)
return
case <-concurrencyLimiter:
log.Printf("Worker %d acquired concurrency slot.", workerID)
defer func() {
log.Printf("Worker %d releasing concurrency slot.", workerID)
concurrencyLimiter <- struct{}{}
}()
if runCtx.Err() != nil {
log.Printf("Worker %d exiting (context done after acquiring slot).", workerID)
return
}
promptText, err := tokenizer.GeneratePrompt(m.params.PromptTokens)
if err != nil {
log.Printf("Worker %d failed to generate prompt: %v. Skipping request.", workerID, err)
continue
}
req := m.createRequest(promptText)
startTime := time.Now()
var result stats.RequestResult
var totalTokens int
firstChunkTime := time.Time{}
log.Printf("Worker %d making request (Streaming: %v)...", workerID, m.params.IsStreaming)
if m.params.IsStreaming {
err = m.params.HTTPClient.Stream(req, func(chunk client.SSEChunk) error {
if firstChunkTime.IsZero() {
firstChunkTime = time.Now()
}
chunkTokens, err := tokenizer.CountTokensInText(string(chunk.Data))
if err != nil {
log.Printf("Worker %d: Error counting stream chunk tokens: %v", workerID, err)
} else {
totalTokens += chunkTokens
}
return nil
})
latency := time.Since(startTime)
result = stats.RequestResult{
IsSuccess: err == nil,
Latency: latency,
TimeToFirstToken: firstChunkTime.Sub(startTime),
TotalTokens: totalTokens,
}
log.Printf("Worker %d finished streaming request. Success: %v", workerID, result.IsSuccess)
m.statsCollector.RecordResult(result)
} else {
resp, doErr := m.params.HTTPClient.Do(req)
endTime := time.Now()
latency := endTime.Sub(startTime)
var statusCode int
var totalTokens int
if doErr != nil {
log.Printf("Worker %d failed to send request (non-stream): %v", workerID, doErr)
result = stats.RequestResult{
IsSuccess: false,
Latency: latency,
TimeToFirstToken: latency,
TotalTokens: 0,
}
} else {
statusCode = resp.StatusCode
bodyBytes := resp.Body
bodyString := string(bodyBytes)
totalTokens, err = tokenizer.CountTokensInText(bodyString)
if err != nil {
log.Printf("Worker %d: Error counting non-stream response tokens: %v", workerID, err)
}
result = stats.RequestResult{
IsSuccess: resp != nil && resp.StatusCode >= 200 && resp.StatusCode < 300,
Latency: latency,
TimeToFirstToken: latency,
TotalTokens: totalTokens,
}
log.Printf("Worker %d finished non-streaming request. Success: %v, Status: %d", workerID, result.IsSuccess, statusCode)
}
m.statsCollector.RecordResult(result)
}
}
}
}(i)
}
<-runCtx.Done()
log.Printf("Benchmark duration (%s) finished or context cancelled.", m.params.TotalDuration)
wg.Wait()
log.Println("runBenchmarkWithoutTicker finished.")
return nil
}
// createRequest creates a new client.Request object for the benchmark.
// It generates a prompt using the tokenizer and constructs the request body.
func (m *Manager) createRequest(prompt string) *client.Request {
payload := map[string]interface{}{
"model": m.params.Model,
"messages": []map[string]string{{"role": "user", "content": prompt}},
"stream": m.params.IsStreaming,
}
body, err := json.Marshal(payload)
if err != nil {
log.Printf("Error marshaling request body: %v", err)
return nil
}
return &client.Request{
URL: m.params.TargetURL,
Method: "POST",
Headers: map[string]string{"Content-Type": "application/json", "Authorization": "Bearer " + m.params.APIKey},
Body: body,
IsStream: m.params.IsStreaming,
}
}
+62
View File
@@ -0,0 +1,62 @@
package config
import (
"fmt"
"os"
"time"
"github.com/spf13/viper"
)
// Config holds the benchmark configuration.
type Config struct {
APIEndpoint string `yaml:"apiEndpoint" mapstructure:"apiEndpoint"`
APIKey string `yaml:"apiKey" mapstructure:"apiKey"` // Consider loading from env or secure store
Model string `yaml:"model" mapstructure:"model"`
Requests int `yaml:"requests"`
Concurrency int `yaml:"concurrency"`
RateLimit int `yaml:"rateLimit"` // Requests per second
Duration time.Duration `yaml:"duration"`
PromptTokens int `yaml:"promptTokens"`
MaxTokens int `yaml:"maxTokens"`
Streaming bool `yaml:"streaming" mapstructure:"streaming"`
Timeout time.Duration `yaml:"timeout"` // Request timeout
RequestTimeout time.Duration `yaml:"requestTimeout" mapstructure:"request_timeout"` // Timeout for individual HTTP requests (defaults to global Timeout if 0)
OutputReport string `yaml:"outputReport"` // Path to save the HTML report
Headers map[string]string `yaml:"headers"` // Custom HTTP headers
Payload map[string]any `yaml:"payload"` // Custom payload fields
Client string `yaml:"client"` // "fasthttp" or "nethttp"
ReportFile string `yaml:"reportFile" mapstructure:"report_file"` // Output file for the HTML report
// Add other configuration parameters as needed
}
// LoadConfig loads configuration from the specified file path.
func LoadConfig(filePath string) (*Config, error) {
// Check if the file exists
if _, err := os.Stat(filePath); os.IsNotExist(err) {
return nil, fmt.Errorf("config file not found at path: %s", filePath)
}
v := viper.New()
v.SetConfigFile(filePath)
v.SetConfigType("yaml")
// Read the configuration file
if err := v.ReadInConfig(); err != nil {
return nil, fmt.Errorf("failed to read config file '%s': %w", filePath, err)
}
var config Config
// Unmarshal the configuration into the Config struct
if err := v.Unmarshal(&config); err != nil {
return nil, fmt.Errorf("failed to unmarshal config from '%s': %w", filePath, err)
}
// Set default RequestTimeout if not provided
if config.RequestTimeout <= 0 {
config.RequestTimeout = config.Timeout
}
// TODO: Add validation for required fields if necessary
return &config, nil
}
+346
View File
@@ -0,0 +1,346 @@
package report
import (
"encoding/json"
"fmt"
"html/template"
"log"
"os"
"time"
"github.com/go-echarts/go-echarts/v2/charts"
"github.com/go-echarts/go-echarts/v2/opts"
"github.com/go-echarts/go-echarts/v2/render"
"llm-api-benchmark-tool/pkg/stats"
)
// ReportData holds the data passed to the HTML template.
type ReportData struct {
Stats stats.FinalStats
LatencyChart template.HTML // Already safe HTML
TTFTChart template.HTML // Already safe HTML
// Add other charts here (QPSChart, etc.)
}
// Basic HTML template string - No |safeHTML filter needed
const htmlTemplate = `<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>LLM API Benchmark Report</title>
<!-- Include ECharts JS directly for simplicity in this example -->
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/echarts@5.4.1/dist/echarts.min.js"></script>
<style> body { font-family: sans-serif; margin: 20px; } table { border-collapse: collapse; margin-bottom: 20px; } th, td { border: 1px solid #ddd; padding: 8px; text-align: left; } th { background-color: #f2f2f2; } .chart-container { width: 80%; min-height: 400px; margin: auto; margin-bottom: 30px; border: 1px solid #eee; padding: 15px; } </style>
</head>
<body>
<h1>LLM API Benchmark Report</h1>
<h2>Summary</h2>
<table>
<tr><th>Metric</th><th>Value</th></tr>
<tr><td>Total Duration:</td><td>{{ .Stats.TotalDuration }}</td></tr>
<tr><td>Total Requests:</td><td>{{ .Stats.TotalRequests }}</td></tr>
<tr><td>Successful Requests:</td><td>{{ .Stats.SuccessfulRequests }}</td></tr>
<tr><td>Failed Requests:</td><td>{{ .Stats.FailedRequests }}</td></tr>
<tr><td>Avg QPS:</td><td>{{ printf "%.2f" .Stats.AvgQPS }}</td></tr>
<tr><td>Avg Latency:</td><td>{{ .Stats.AvgLatency }}</td></tr>
<tr><td>Min Latency:</td><td>{{ .Stats.MinLatency }}</td></tr>
<tr><td>Max Latency:</td><td>{{ .Stats.MaxLatency }}</td></tr>
<tr><td>P90 Latency:</td><td>{{ .Stats.P90Latency }}</td></tr>
<tr><td>P95 Latency:</td><td>{{ .Stats.P95Latency }}</td></tr>
<tr><td>P99 Latency:</td><td>{{ .Stats.P99Latency }}</td></tr>
<tr><td>Avg TTFT:</td><td>{{ .Stats.AvgTimeToFirstToken }}</td></tr>
<tr><td>Min TTFT:</td><td>{{ .Stats.MinTimeToFirstToken }}</td></tr>
<tr><td>Max TTFT:</td><td>{{ .Stats.MaxTimeToFirstToken }}</td></tr>
<tr><td>P90 TTFT:</td><td>{{ .Stats.P90TimeToFirstToken }}</td></tr>
<tr><td>P95 TTFT:</td><td>{{ .Stats.P95TimeToFirstToken }}</td></tr>
<tr><td>P99 TTFT:</td><td>{{ .Stats.P99TimeToFirstToken }}</td></tr>
<tr><td>Avg Tokens/Second:</td><td>{{ printf "%.2f" .Stats.AvgTokensPerSecond }}</td></tr>
<!-- Add more stats as needed -->
</table>
<h2>Charts</h2>
<!-- Latency Histogram Chart -->
<div class="chart-container">
{{ .LatencyChart }}
</div>
<!-- TTFT Histogram Chart -->
<div class="chart-container">
{{ .TTFTChart }}
</div>
<!-- Add more chart containers here -->
</body>
</html>`
// GenerateHTMLReport generates an HTML report from the benchmark statistics.
func GenerateHTMLReport(finalStats stats.FinalStats, outputFilePath string) error {
// --- Create Charts ---
latencyChart := createLatencyHistogram(finalStats)
ttftChart := createTTFTHistogram(finalStats)
// --- Prepare Template Data ---
// Render charts to temporary buffers to get HTML content
latencyChartHTML, err := renderChartToHTMLFragment(latencyChart)
if err != nil {
return fmt.Errorf("failed to render latency chart: %w", err)
}
ttftChartHTML, err := renderChartToHTMLFragment(ttftChart)
if err != nil {
return fmt.Errorf("failed to render TTFT chart: %w", err)
}
summaryStr := fmt.Sprintf(`
Total Requests: %d
Successful Requests: %d
Failed Requests: %d
Total Duration: %s
Average Latency: %s
Min Latency: %s
Max Latency: %s
P90 Latency: %s
P95 Latency: %s
P99 Latency: %s
Average TTFT: %s
Min TTFT: %s
Max TTFT: %s
P90 TTFT: %s
P95 TTFT: %s
P99 TTFT: %s
Average QPS: %.2f
Average Tokens/Second: %.2f
`,
finalStats.TotalRequests,
finalStats.SuccessfulRequests,
finalStats.FailedRequests,
finalStats.TotalDuration,
finalStats.AvgLatency,
finalStats.MinLatency,
finalStats.MaxLatency,
finalStats.P90Latency,
finalStats.P95Latency,
finalStats.P99Latency,
finalStats.AvgTimeToFirstToken,
finalStats.MinTimeToFirstToken,
finalStats.MaxTimeToFirstToken,
finalStats.P90TimeToFirstToken,
finalStats.P95TimeToFirstToken,
finalStats.P99TimeToFirstToken,
finalStats.AvgQPS,
finalStats.AvgTokensPerSecond,
)
templateData := map[string]interface{}{
"Stats": finalStats,
"Summary": summaryStr,
"LatencyChart": latencyChartHTML,
"TTFTChart": ttftChartHTML,
}
// --- Parse and Execute Template ---
// No Funcs map needed as renderChartToHTMLFragment returns template.HTML
tmpl, err := template.New("report").Parse(htmlTemplate)
if err != nil {
return fmt.Errorf("failed to parse HTML template: %w", err)
}
f, err := os.Create(outputFilePath)
if err != nil {
return fmt.Errorf("failed to create report file '%s': %w", outputFilePath, err)
}
defer f.Close()
// log.Printf("DEBUG: Template Data Before Execution: %+v", templateData)
// Execute the template
err = tmpl.Execute(f, templateData)
if err != nil {
// Include the underlying template execution error for better debugging
return fmt.Errorf("failed to execute HTML template: %w", err)
}
fmt.Printf("HTML report generated successfully: %s\n", outputFilePath)
return nil
}
// getChartBaseConfig safely extracts the BaseConfiguration from a render.Renderer.
func getChartBaseConfig(chart render.Renderer) (*charts.BaseConfiguration, string, error) {
var baseConfig *charts.BaseConfiguration
var chartID string
switch c := chart.(type) {
case *charts.Bar:
baseConfig = &c.RectChart.RectConfiguration.BaseConfiguration
chartID = c.Initialization.ChartID // Access ChartID from embedded Initialization
case *charts.Line: // Add cases for other chart types if needed
baseConfig = &c.RectChart.RectConfiguration.BaseConfiguration
chartID = c.Initialization.ChartID
// Add other chart types here (e.g., Pie, Scatter, etc.)
default:
return nil, "", fmt.Errorf("unsupported chart type: %T", chart)
}
if chartID == "" {
// Generate a fallback ID if not set in Initialization
chartID = fmt.Sprintf("chart%d", time.Now().UnixNano())
log.Printf("Warning: ChartID not set for %T, using generated ID: %s", chart, chartID)
}
return baseConfig, chartID, nil
}
// renderChartToHTMLFragment renders a chart configuration to an HTML fragment.
func renderChartToHTMLFragment(chart render.Renderer) (template.HTML, error) {
baseConfig, chartID, err := getChartBaseConfig(chart)
if err != nil {
return "", err
}
// Serialize the BaseConfiguration which holds the options.
optJSONBytes, err := json.Marshal(baseConfig)
if err != nil {
return "", fmt.Errorf("failed to marshal chart options for %s (%T): %w", chartID, chart, err)
}
optJSON := string(optJSONBytes)
// Construct the HTML fragment (div and script)
script := fmt.Sprintf(`
<div id="%s" style="width:100%%; height:400px;"></div>
<script type="text/javascript">
"use strict";
// Check if element exists before initializing
if (document.getElementById('%s')) {
let chart_%s = echarts.init(document.getElementById('%s'), null, {renderer: '%s'});
let option_%s = %s;
chart_%s.setOption(option_%s);
} else {
console.error("Chart element not found: %s");
}
</script>`,
chartID, // div id
chartID, // Check element id
chartID, chartID, // script variable and element id
baseConfig.Initialization.Renderer, // Use renderer from initialization options
chartID, optJSON, // options variable and json content
chartID, chartID, // setOption call
chartID, // Error log element id
)
return template.HTML(script), nil
}
// calculateHistogram calculates histogram data (categories and counts) from a slice of durations.
func calculateHistogram(data []time.Duration, numBins int) ([]string, []opts.BarData) {
if len(data) == 0 || numBins <= 0 {
return []string{"No Data"}, []opts.BarData{{Value: 0}}
}
// Find min and max values
minVal, maxVal := data[0], data[0]
for _, d := range data {
if d < minVal {
minVal = d
}
if d > maxVal {
maxVal = d
}
}
// Handle case where all data points are the same
if minVal == maxVal {
return []string{fmt.Sprintf("%.2f ms", float64(minVal)/float64(time.Millisecond))},
[]opts.BarData{{Value: len(data)}}
}
binWidth := (maxVal - minVal) / time.Duration(numBins)
if binWidth == 0 {
binWidth = 1 // Avoid division by zero if range is very small relative to numBins
}
bins := make([]int, numBins) // Stores counts for each bin
categories := make([]string, numBins)
for i := 0; i < numBins; i++ {
binStart := minVal + time.Duration(i)*binWidth
binEnd := binStart + binWidth
// Ensure the last bin's end includes the max value exactly
if i == numBins-1 {
binEnd = maxVal
}
// Format category label (e.g., "50.00-100.00 ms")
// Use slightly different formatting for clarity: [start, end) or [start, end] for the last bin
labelFormat := "[%.2f, %.2f) ms"
if i == numBins-1 {
labelFormat = "[%.2f, %.2f] ms"
}
categories[i] = fmt.Sprintf(labelFormat,
float64(binStart)/float64(time.Millisecond),
float64(binEnd)/float64(time.Millisecond),
)
}
for _, d := range data {
// Determine the correct bin index
var binIndex int
if d == maxVal {
// Special case: max value goes into the last bin
binIndex = numBins - 1
} else {
// Calculate bin index based on width
binIndex = int((d - minVal) / binWidth)
// Clamp index to valid range (should not happen with maxVal check, but defensive)
if binIndex < 0 {
binIndex = 0
} else if binIndex >= numBins {
binIndex = numBins - 1
}
}
bins[binIndex]++
}
barData := make([]opts.BarData, numBins)
for i := 0; i < numBins; i++ {
barData[i] = opts.BarData{Value: bins[i]}
}
return categories, barData
}
// createLatencyHistogram creates a Bar chart for latency distribution.
func createLatencyHistogram(finalStats stats.FinalStats) *charts.Bar {
bar := charts.NewBar()
bar.SetGlobalOptions(
charts.WithTitleOpts(opts.Title{Title: "Latency Distribution"}),
// Ensure ID is set correctly via InitializationOpts
charts.WithInitializationOpts(opts.Initialization{ChartID: "latencyHistogram"}),
// Add other global options if needed, e.g., Toolbox
charts.WithToolboxOpts(opts.Toolbox{Show: opts.Bool(true)}),
)
// Calculate and set histogram data
categories, data := calculateHistogram(finalStats.LatencyData, 10) // Use 10 bins for example
bar.SetXAxis(categories).AddSeries("Latency", data)
return bar
}
// createTTFTHistogram creates a Bar chart for Time To First Token distribution.
func createTTFTHistogram(finalStats stats.FinalStats) *charts.Bar {
bar := charts.NewBar()
bar.SetGlobalOptions(
charts.WithTitleOpts(opts.Title{Title: "TTFT Distribution"}),
// Ensure ID is set correctly via InitializationOpts
charts.WithInitializationOpts(opts.Initialization{ChartID: "ttftHistogram"}),
// Add other global options if needed
charts.WithToolboxOpts(opts.Toolbox{Show: opts.Bool(true)}),
)
// Calculate and set histogram data (assuming TTFTData exists in FinalStats)
categories, data := calculateHistogram(finalStats.TTFTData, 10) // Use actual TTFT data
bar.SetXAxis(categories).AddSeries("TTFT", data)
return bar
}
+175
View File
@@ -0,0 +1,175 @@
package report
import (
"testing"
"time"
"llm-api-benchmark-tool/pkg/stats"
"github.com/go-echarts/go-echarts/v2/opts"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestCalculateHistogram(t *testing.T) {
// Helper function to create durations from milliseconds
dms := func(ms int) time.Duration {
return time.Duration(ms) * time.Millisecond
}
tests := []struct {
name string
data []time.Duration
numBins int
wantCats []string
wantDataVals []int // Just check the values
isErrorCase bool // Indicates if we expect the "No Data" case
}{
{
name: "Empty data",
data: []time.Duration{},
numBins: 5,
wantCats: []string{"No Data"},
wantDataVals: []int{0},
isErrorCase: true,
},
{
name: "Zero bins",
data: []time.Duration{dms(100), dms(200)},
numBins: 0,
wantCats: []string{"No Data"},
wantDataVals: []int{0},
isErrorCase: true,
},
{
name: "Single data point",
data: []time.Duration{dms(150)},
numBins: 5,
wantCats: []string{"150.00 ms"}, // Special case format
wantDataVals: []int{1},
},
{
name: "All data points same",
data: []time.Duration{dms(100), dms(100), dms(100)},
numBins: 3,
wantCats: []string{"100.00 ms"},
wantDataVals: []int{3},
},
{
name: "Simple case, 2 bins",
data: []time.Duration{dms(50), dms(60), dms(110), dms(120), dms(130)},
// Range: 50 to 130 (80ms)
// Bin width: 80 / 2 = 40
// Bins: [50, 90), [90, 130]
numBins: 2,
wantCats: []string{"[50.00, 90.00) ms", "[90.00, 130.00] ms"},
wantDataVals: []int{2, 3},
},
{
name: "Even distribution, 5 bins",
data: []time.Duration{dms(10), dms(25), dms(35), dms(45), dms(55), dms(65), dms(75), dms(85), dms(95), dms(100)},
// Range: 10 to 100 (90ms) -> max should be 100? Let's make max 110 for simpler bins
// Let's adjust data slightly for easier math: min 10, max 110
// data: {10, 25, 35, 45, 55, 65, 75, 85, 95, 110}
// Range: 10 to 110 (100ms)
// Bin width: 100 / 5 = 20
// Bins: [10,30), [30,50), [50,70), [70,90), [90,110]
numBins: 5,
wantCats: []string{"[10.00, 28.00) ms", "[28.00, 46.00) ms", "[46.00, 64.00) ms", "[64.00, 82.00) ms", "[82.00, 100.00] ms"},
wantDataVals: []int{2, 2, 1, 2, 3},
},
{
name: "Data clustered at ends",
data: []time.Duration{dms(10), dms(15), dms(20), dms(90), dms(95), dms(100)},
// Range: 10 to 100 (90). Width (3 bins): 30.
// Bins: [10, 40), [40, 70), [70, 100]
numBins: 3,
wantCats: []string{"[10.00, 40.00) ms", "[40.00, 70.00) ms", "[70.00, 100.00] ms"},
wantDataVals: []int{3, 0, 3},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
gcats, gdata := calculateHistogram(tt.data, tt.numBins)
require.Equal(t, len(tt.wantCats), len(gcats), "Number of categories mismatch")
assert.Equal(t, tt.wantCats, gcats, "Category labels mismatch")
require.Equal(t, len(tt.wantDataVals), len(gdata), "Number of data points mismatch")
// Extract values from opts.BarData
gotDataVals := make([]int, len(gdata))
for i, item := range gdata {
// Check if item.Value is an int or can be converted
if val, ok := item.Value.(int); ok {
gotDataVals[i] = val
} else if valF, ok := item.Value.(float64); ok {
gotDataVals[i] = int(valF) // Or handle float differently if needed
} else {
t.Fatalf("Unexpected data type in BarData: %T", item.Value)
}
}
assert.Equal(t, tt.wantDataVals, gotDataVals, "Data values mismatch")
})
}
}
// Helper to create durations
func dms(ms int) time.Duration {
return time.Duration(ms) * time.Millisecond
}
// TestCreateLatencyHistogram checks if the latency histogram chart is created correctly.
func TestCreateLatencyHistogram(t *testing.T) {
mockStats := stats.FinalStats{
LatencyData: []time.Duration{dms(50), dms(100), dms(110), dms(150), dms(200), dms(210), dms(220)},
}
chart := createLatencyHistogram(mockStats)
require.NotNil(t, chart, "Chart should not be nil")
require.NotNil(t, chart.BaseConfiguration.Title, "Chart title configuration should not be nil")
// Check title
assert.Equal(t, "Latency Distribution", chart.BaseConfiguration.Title.Title, "Chart title mismatch")
// Check series data (basic check - assuming one series with correct name)
require.Len(t, chart.MultiSeries, 1, "Expected one series")
assert.Equal(t, "Latency", chart.MultiSeries[0].Name, "Series name mismatch")
// Optionally, check the number of data points generated by calculateHistogram via the chart
// This depends on the numBins used internally by createLatencyHistogram (currently 10)
// For the mock data {50, 100, 110, 150, 200, 210, 220}, range=170, 10 bins -> width=17
// Bins: [50, 67), [67, 84), [84, 101), [101, 118), [118, 135), [135, 152), [152, 169), [169, 186), [186, 203), [203, 220]
// Counts: 1, 0, 1, 1, 0, 1, 0, 0, 1, 2 -> Total 7 points, 10 bars
seriesData, ok := chart.MultiSeries[0].Data.([]opts.BarData)
require.True(t, ok, "Series data should be of type []opts.BarData")
assert.Len(t, seriesData, 10, "Expected 10 data points (bins) in the series")
}
// TestCreateTTFTHistogram checks if the TTFT histogram chart is created correctly.
func TestCreateTTFTHistogram(t *testing.T) {
mockStats := stats.FinalStats{
TTFTData: []time.Duration{dms(20), dms(30), dms(35), dms(40), dms(60)},
}
chart := createTTFTHistogram(mockStats)
require.NotNil(t, chart, "Chart should not be nil")
require.NotNil(t, chart.BaseConfiguration.Title, "Chart title configuration should not be nil")
// Check title
assert.Equal(t, "TTFT Distribution", chart.BaseConfiguration.Title.Title, "Chart title mismatch")
// Check series data
require.Len(t, chart.MultiSeries, 1, "Expected one series")
assert.Equal(t, "TTFT", chart.MultiSeries[0].Name, "Series name mismatch")
// Optionally, check the number of data points
// Data: {20, 30, 35, 40, 60}, Range=40, 10 bins -> width=4
// Bins: [20,24), [24,28), [28,32), [32,36), [36,40), [40,44), [44,48), [48,52), [52,56), [56,60]
// Counts: 1, 0, 1, 1, 0, 1, 0, 0, 0, 1 -> Total 5 points, 10 bars
seriesData, ok := chart.MultiSeries[0].Data.([]opts.BarData)
require.True(t, ok, "Series data should be of type []opts.BarData")
assert.Len(t, seriesData, 10, "Expected 10 data points (bins) in the series")
}
+227
View File
@@ -0,0 +1,227 @@
package stats
import (
"log"
"sort"
"sync"
"time"
// "gonum.org/v1/gonum/stat" // Keep commented until needed for percentiles
)
// RequestResult holds the metrics for a single request.
type RequestResult struct {
IsSuccess bool
Latency time.Duration
TimeToFirstToken time.Duration // Relevant for streaming and non-streaming (time to first byte of response body)
TotalTokens int // Tokens in the response content
StreamLatency []time.Duration // Latency for each chunk in a stream (if streaming)
// TODO: Add Error details? Model name?
}
// FinalStats holds the aggregated results of the benchmark run.
type FinalStats struct {
TotalRequests int64
SuccessfulRequests int64
FailedRequests int64
TotalDuration time.Duration
AvgLatency time.Duration // Average latency of successful requests
MinLatency time.Duration
MaxLatency time.Duration
P90Latency time.Duration
P95Latency time.Duration
P99Latency time.Duration
AvgTimeToFirstToken time.Duration // Average TTFT of successful requests
MinTimeToFirstToken time.Duration
MaxTimeToFirstToken time.Duration
P90TimeToFirstToken time.Duration
P95TimeToFirstToken time.Duration
P99TimeToFirstToken time.Duration
AvgQPS float64 // Average Queries Per Second (TotalRequests / TotalDuration)
AvgTokensPerSecond float64 // Average tokens per second (TotalTokens / Total Successful Latency)
// Data for histograms
LatencyData []time.Duration `json:"-"` // Raw latency data points (exclude from default JSON)
TTFTData []time.Duration `json:"-"` // Raw TTFT data points (exclude from default JSON)
// TODO: Add MaxQPS? Token Rate per model?
}
// StatsCollector collects and aggregates statistics during the benchmark.
// It needs to be thread-safe.
type StatsCollector struct {
mutex sync.Mutex
results []RequestResult // Store individual results for percentile calculation
startTime time.Time
totalTokens int64
totalReqTime time.Duration // Sum of latencies for successful requests
successCount int64
failCount int64
minLatency time.Duration
maxLatency time.Duration
minTTFT time.Duration
maxTTFT time.Duration
initialized bool
// Raw data storage
latencies []time.Duration // Store raw latencies for histogram
ttfts []time.Duration // Store raw TTFTs for histogram
}
// NewStatsCollector creates a new thread-safe StatsCollector.
func NewStatsCollector() *StatsCollector {
return &StatsCollector{
results: make([]RequestResult, 0, 1000), // Pre-allocate some capacity
latencies: make([]time.Duration, 0, 1000), // Initialize
ttfts: make([]time.Duration, 0, 1000), // Initialize
startTime: time.Now(),
minLatency: time.Duration(1<<63 - 1), // Max duration as initial min
minTTFT: time.Duration(1<<63 - 1), // Max duration as initial min
initialized: true,
}
}
// RecordResult records the outcome of a single request. Thread-safe.
func (sc *StatsCollector) RecordResult(result RequestResult) {
sc.mutex.Lock()
defer sc.mutex.Unlock()
if !sc.initialized {
// This shouldn't happen with NewStatsCollector, but defensive check
return
}
log.Printf("[StatsCollector] Recording result: Success=%t, Latency=%s, TTFT=%s, Tokens=%d", result.IsSuccess, result.Latency, result.TimeToFirstToken, result.TotalTokens)
sc.results = append(sc.results, result) // Append for percentile calculation
if result.IsSuccess {
sc.successCount++
sc.totalTokens += int64(result.TotalTokens)
sc.totalReqTime += result.Latency
if result.Latency < sc.minLatency {
sc.minLatency = result.Latency
}
if result.Latency > sc.maxLatency {
sc.maxLatency = result.Latency
}
sc.latencies = append(sc.latencies, result.Latency)
// TTFT calculations only for successful requests
if result.TimeToFirstToken > 0 { // Ensure valid TTFT
if result.TimeToFirstToken < sc.minTTFT {
sc.minTTFT = result.TimeToFirstToken
}
if result.TimeToFirstToken > sc.maxTTFT {
sc.maxTTFT = result.TimeToFirstToken
}
sc.ttfts = append(sc.ttfts, result.TimeToFirstToken)
}
} else {
sc.failCount++
}
}
// CalculateStats computes the final statistics based on recorded results.
// totalDurationOverride allows specifying the exact benchmark duration used for QPS calculation.
func (sc *StatsCollector) CalculateStats(totalDurationOverride time.Duration) FinalStats {
sc.mutex.Lock()
defer sc.mutex.Unlock()
stats := FinalStats{
TotalDuration: totalDurationOverride,
}
if !sc.initialized || len(sc.results) == 0 {
return stats // Return empty stats if no results
}
stats.SuccessfulRequests = sc.successCount
stats.FailedRequests = sc.failCount
stats.TotalRequests = sc.successCount + sc.failCount
if totalDurationOverride.Seconds() > 0 {
stats.AvgQPS = float64(stats.TotalRequests) / totalDurationOverride.Seconds()
}
if sc.successCount > 0 {
stats.AvgLatency = sc.totalReqTime / time.Duration(sc.successCount)
stats.MinLatency = sc.minLatency
stats.MaxLatency = sc.maxLatency
// Filter successful results for latency/TTFT percentile calculations
successfulLatencies := make([]float64, 0, sc.successCount)
successfulTTFTs := make([]float64, 0, sc.successCount)
var totalTTFT time.Duration
for _, res := range sc.results {
if res.IsSuccess {
successfulLatencies = append(successfulLatencies, float64(res.Latency.Nanoseconds()))
if res.TimeToFirstToken > 0 {
successfulTTFTs = append(successfulTTFTs, float64(res.TimeToFirstToken.Nanoseconds()))
totalTTFT += res.TimeToFirstToken
}
}
}
// Calculate Avg TTFT
if len(successfulTTFTs) > 0 {
stats.AvgTimeToFirstToken = totalTTFT / time.Duration(len(successfulTTFTs))
stats.MinTimeToFirstToken = sc.minTTFT
stats.MaxTimeToFirstToken = sc.maxTTFT
}
// Calculate average tokens/sec based on successful requests' total time
if sc.totalReqTime.Seconds() > 0 {
stats.AvgTokensPerSecond = float64(sc.totalTokens) / sc.totalReqTime.Seconds()
}
// --- Percentile Calculations (using basic sort for now) ---
sort.Float64s(successfulLatencies)
if len(successfulLatencies) > 0 {
stats.P90Latency = time.Duration(percentile(successfulLatencies, 90)) * time.Nanosecond
stats.P95Latency = time.Duration(percentile(successfulLatencies, 95)) * time.Nanosecond
stats.P99Latency = time.Duration(percentile(successfulLatencies, 99)) * time.Nanosecond
}
sort.Float64s(successfulTTFTs)
if len(successfulTTFTs) > 0 {
stats.P90TimeToFirstToken = time.Duration(percentile(successfulTTFTs, 90)) * time.Nanosecond
stats.P95TimeToFirstToken = time.Duration(percentile(successfulTTFTs, 95)) * time.Nanosecond
stats.P99TimeToFirstToken = time.Duration(percentile(successfulTTFTs, 99)) * time.Nanosecond
}
} else {
// Handle cases where there are no successful requests
stats.MinLatency = 0
stats.MaxLatency = 0
stats.MinTimeToFirstToken = 0
stats.MaxTimeToFirstToken = 0
}
// Assign raw data for histograms
stats.LatencyData = make([]time.Duration, len(sc.latencies))
copy(stats.LatencyData, sc.latencies)
stats.TTFTData = make([]time.Duration, len(sc.ttfts))
copy(stats.TTFTData, sc.ttfts)
return stats
}
// percentile calculates the p-th percentile of a sorted float64 slice.
// Basic implementation, assumes data is already sorted.
func percentile(data []float64, p float64) float64 {
if len(data) == 0 {
return 0
}
if p <= 0 {
return data[0]
}
if p >= 100 {
return data[len(data)-1]
}
index := (p / 100.0) * float64(len(data)-1)
lower := int(index)
upper := lower + 1
if upper >= len(data) {
return data[lower] // Should only happen if index is exactly len(data)-1
}
weight := index - float64(lower)
return data[lower]*(1-weight) + data[upper]*weight
}
+112
View File
@@ -0,0 +1,112 @@
package tokenizer
import (
"errors"
"fmt"
"sync"
"github.com/tiktoken-go/tokenizer"
)
// A sample text to generate prompts from. Using Moby Dick chapter 1 start.
const sampleText = `Call me Ishmael. Some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen and regulating the circulation. Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin warehouses, and bringing up the rear of every funeral I meet; and especially whenever my hypos get such an upper hand of me, that it requires a strong moral principle to prevent me from deliberately stepping into the street, and methodically knocking people’s hats off—then, I account it high time to get to sea as soon as I can. This is my substitute for pistol and ball. With a philosophical flourish Cato throws himself upon his sword; I quietly take to the ship. There is nothing surprising in this. If they but knew it, almost all men in their degree, some time or other, cherish very nearly the same feelings towards the ocean with me. There now is your insular city of the Manhattoes, belted round by wharves as Indian isles by coral reefs—commerce surrounds it with her surf. Right and left, the streets take you waterward. Its extreme downtown is the battery, where that noble mole is washed by waves, and cooled by breezes, which a few hours previous were out of sight of land. Look at the crowds of water-gazers there. Circumambulate the city of a dreamy Sabbath afternoon. Go from Corlears Hook to Coenties Slip, and from thence, by Whitehall, northward. What do you see?—Posted like silent sentinels all around the town, stand thousands upon thousands of mortal men fixed in ocean reveries. Some leaning against the spiles; some seated upon the pier-heads; some looking over the bulwarks of ships from China; some high aloft in the rigging, as if striving to get a still better seaward peep. But these are all landsmen; of week days pent up in lath and plaster—tied to counters, nailed to benches, worried by ledgers. How then is this? Are the green fields gone? What do they here? But look! here come more crowds, pacing straight for the water, and seemingly bound for a dive. Strange! Nothing will content them but the extremest limit of the land; loitering under the shady lee of yonder warehouses will not suffice. No. They must get just as nigh the water as they possibly can without falling in. And there they stand—miles of them—leagues. Inlanders all, they come from lanes and alleys, streets and avenues—north, east, south, and west. Yet here they all unite. Tell me, does the magnetic virtue of the needles of the compasses of all those ships attract them thither?`
var (
codec tokenizer.Codec
encodedText []uint
initErr error
codecMutex sync.Mutex // Mutex to protect concurrent access to the codec
)
func init() {
// Initialize the tokenizer and encode the sample text once
codec, initErr = tokenizer.Get(tokenizer.Cl100kBase)
if initErr != nil {
initErr = fmt.Errorf("failed to get tokenizer codec: %w", initErr)
return
}
ids, _, err := codec.Encode(sampleText)
if err != nil {
initErr = fmt.Errorf("failed to encode sample text: %w", err)
return
}
encodedText = ids
}
// GeneratePrompt generates a text prompt with approximately the target number of tokens.
func GeneratePrompt(targetTokenCount int) (string, error) {
// Check initialization errors first
if initErr != nil {
return "", fmt.Errorf("tokenizer initialization failed: %w", initErr)
}
if codec == nil {
// This case should theoretically be covered by initErr check, but belt and suspenders
return "", errors.New("tokenizer codec is not initialized")
}
if targetTokenCount <= 0 {
return "", errors.New("target token count must be positive")
}
var selectedTokens []uint
// Repeat or truncate the encoded text to match the target count
if targetTokenCount > len(encodedText) {
// Repeat tokens until target is met
selectedTokens = make([]uint, 0, targetTokenCount)
for len(selectedTokens) < targetTokenCount {
remaining := targetTokenCount - len(selectedTokens)
if remaining >= len(encodedText) {
selectedTokens = append(selectedTokens, encodedText...)
} else {
selectedTokens = append(selectedTokens, encodedText[:remaining]...)
}
}
} else {
// Select the first N tokens
selectedTokens = encodedText[:targetTokenCount]
}
// --- Protect the call to Decode ---
codecMutex.Lock()
defer codecMutex.Unlock() // Ensure unlock happens even on error
// Decode the selected tokens back to text
prompt, err := codec.Decode(selectedTokens)
if err != nil {
// Log the problematic token slice for debugging if needed
// fmt.Printf("Failed to decode tokens: %v\n", selectedTokens)
return "", fmt.Errorf("failed to decode %d selected tokens: %w", len(selectedTokens), err)
}
// ---------------------------------
return prompt, nil
}
// CountTokensInText counts the number of tokens in the given text using the initialized codec.
func CountTokensInText(text string) (int, error) {
// Check initialization errors first
if initErr != nil {
return 0, fmt.Errorf("tokenizer initialization failed: %w", initErr)
}
if codec == nil {
return 0, errors.New("tokenizer codec is not initialized")
}
// --- Protect the call to Encode --- Must protect even read-only operations if codec has internal state?
codecMutex.Lock()
defer codecMutex.Unlock()
ids, _, err := codec.Encode(text)
if err != nil {
return 0, fmt.Errorf("failed to encode text for token counting: %w", err)
}
// ---------------------------------
return len(ids), nil
}
// CheckInitStatus returns any error that occurred during package initialization.
func CheckInitStatus() error {
return initErr
}
-327
View File
@@ -1,327 +0,0 @@
package prompt
import (
"fmt"
"math/rand"
"strings"
"time"
"github.com/acmestudio/llm-api-benchmark-tool/config"
"github.com/acmestudio/llm-api-benchmark-tool/logger"
"github.com/pkoukk/tiktoken-go"
)
// 填充词,用于调整提示词长度
var fillerWords = []string{
"详细", "简要", "请", "麻烦", "帮忙", "认真", "仔细", "全面", "系统性地",
"清晰地", "准确地", "深入地", "专业地", "客观地", "科学地", "有条理地",
"具体地", "简洁地", "完整地", "精确地", "有逻辑地", "有说服力地",
}
// 长文本填充段落,用于扩展长文档提示词
var longFillerParagraphs = []string{
"请确保分析全面深入,考虑各个方面的影响和变化。需要包含具体的数据和案例来支持你的观点,同时也要有理论依据。",
"在回答中,请注意逻辑性和连贯性,确保各部分之间有合理的过渡。同时,请尽量使用专业术语,但也要确保非专业人士能够理解。",
"希望你能够从多个角度进行分析,包括但不限于社会学、经济学、政治学、心理学等视角。同时,也请考虑不同群体、不同地区的差异性。",
"在分析中,请注意历史背景和未来趋势,将当前情况放在更广阔的时空背景下考量。同时,也请关注国际比较,了解全球范围内的相似与差异。",
"请在回答中提供一些可行的建议或解决方案,这些建议应该是具体的、可操作的,并且考虑到实施过程中可能遇到的困难和阻力。",
"在分析问题时,请注意考虑各种可能的因素,包括社会、经济、政治、文化、技术等方面。同时,也请思考这些因素之间的相互作用和影响。",
"请在回答中使用清晰的结构,可以考虑使用小标题、要点列表等方式组织内容,使读者能够更容易地理解和把握核心内容。",
"在分析过程中,请注意区分事实和观点,确保事实准确无误,观点有理有据。同时,也请尽量避免个人偏见,保持客观中立的态度。",
"请确保你的回答既有理论深度,又有实践指导意义。理论分析应该有学术依据,实践建议应该考虑可行性和有效性。",
"在回答中,请注意时效性,考虑最新的发展和变化。同时,也请关注长期趋势和基本规律,避免被短期波动所干扰。",
}
// 替换词,用于模板中的占位符
var replacements = map[string][]string{
"country": {"中国", "美国", "日本", "英国", "法国", "德国", "俄罗斯", "加拿大", "澳大利亚", "巴西", "印度"},
"concept": {"人工智能", "机器学习", "区块链", "量子计算", "云计算", "边缘计算", "物联网", "大数据", "虚拟现实", "增强现实"},
"topic": {"全球变暖", "可持续发展", "数字化转型", "远程工作", "网络安全", "太空探索", "生物技术", "清洁能源", "智慧城市", "数字隐私"},
"event": {"工业革命", "互联网诞生", "冷战", "登月", "柏林墙倒塌", "9/11事件", "COVID-19疫情", "阿拉伯之春", "2008金融危机", "人类基因组计划"},
}
// Generator 提示词生成器
type Generator struct {
config *config.Config
tokenizer *tiktoken.Tiktoken
rand *rand.Rand
}
// NewGenerator 创建新的提示词生成器
func NewGenerator(cfg *config.Config) (*Generator, error) {
// 初始化分词器
logger.Debug("初始化分词器,模型: %s", cfg.Tokenizer.Model)
tkm, err := tiktoken.EncodingForModel(cfg.Tokenizer.Model)
if err != nil {
return nil, fmt.Errorf("初始化分词器失败: %w", err)
}
logger.Debug("分词器初始化成功")
// 初始化随机数生成器
source := rand.NewSource(time.Now().UnixNano())
logger.Debug("提示词模板配置 - 短咨询: %d个模板, 目标Token: %d",
len(cfg.PromptTemplates.Short.Templates), cfg.PromptTemplates.Short.TargetTokens)
logger.Debug("提示词模板配置 - 长文档: %d个模板, 目标Token: %d",
len(cfg.PromptTemplates.Long.Templates), cfg.PromptTemplates.Long.TargetTokens)
return &Generator{
config: cfg,
tokenizer: tkm,
rand: rand.New(source),
}, nil
}
// GeneratePrompt 生成指定类型的提示词
func (g *Generator) GeneratePrompt(promptType string) (string, error) {
var templateConfig config.PromptTypeConfig
// 根据类型选择模板配置
switch promptType {
case "short":
templateConfig = g.config.PromptTemplates.Short
case "long":
templateConfig = g.config.PromptTemplates.Long
default:
return "", fmt.Errorf("未知的提示词类型: %s", promptType)
}
// 检查模板是否存在
if len(templateConfig.Templates) == 0 {
return "", fmt.Errorf("提示词模板为空: %s", promptType)
}
// 随机选择一个模板
templateIndex := g.rand.Intn(len(templateConfig.Templates))
template := templateConfig.Templates[templateIndex]
logger.Debug("为%s类型选择模板 #%d: %s", promptType, templateIndex, template)
// 替换模板中的占位符
prompt := g.replacePlaceholders(template)
logger.Debug("替换占位符后: %s", prompt)
// 调整提示词长度至目标Token数
result, err := g.adjustPromptLength(prompt, templateConfig.TargetTokens)
if err != nil {
return "", err
}
// 计算最终Token数
tokens := g.tokenizer.Encode(result, nil, nil)
logger.Debug("生成%s类型提示词成功,目标Token: %d, 实际Token: %d",
promptType, templateConfig.TargetTokens, len(tokens))
return result, nil
}
// replacePlaceholders 替换模板中的占位符
func (g *Generator) replacePlaceholders(template string) string {
result := template
// 替换所有占位符
for placeholder, options := range replacements {
placeholderTag := fmt.Sprintf("{%s}", placeholder)
if strings.Contains(result, placeholderTag) {
// 随机选择一个替换选项
replacementIndex := g.rand.Intn(len(options))
replacement := options[replacementIndex]
logger.Debug("替换占位符 %s -> %s", placeholderTag, replacement)
result = strings.Replace(result, placeholderTag, replacement, -1)
}
}
return result
}
// adjustPromptLength 调整提示词长度至目标Token数
func (g *Generator) adjustPromptLength(prompt string, targetTokens int) (string, error) {
// 计算当前Token数
tokens := g.tokenizer.Encode(prompt, nil, nil)
currentTokens := len(tokens)
// 计算目标范围(±5%)
minTokens := int(float64(targetTokens) * 0.95)
maxTokens := int(float64(targetTokens) * 1.05)
logger.Debug("调整提示词长度 - 当前: %d, 目标范围: %d-%d", currentTokens, minTokens, maxTokens)
// 如果当前Token数在目标范围内,直接返回
if currentTokens >= minTokens && currentTokens <= maxTokens {
logger.Debug("提示词长度已在目标范围内,无需调整")
return prompt, nil
}
// 如果Token数过少,添加填充词
if currentTokens < minTokens {
logger.Debug("提示词长度过短,需要扩展")
return g.extendPrompt(prompt, targetTokens)
}
// 如果Token数过多,截断提示词
logger.Debug("提示词长度过长,需要截断")
return g.truncatePrompt(prompt, targetTokens)
}
// extendPrompt 通过添加填充词扩展提示词
func (g *Generator) extendPrompt(prompt string, targetTokens int) (string, error) {
result := prompt
// 计算当前Token数
tokens := g.tokenizer.Encode(result, nil, nil)
currentTokens := len(tokens)
// 计算目标范围
minTokens := int(float64(targetTokens) * 0.95)
// 根据目标token数选择不同的扩展策略
if targetTokens > 500 {
// 长文档提示词,使用段落填充
logger.Debug("使用段落填充扩展长文档提示词")
return g.extendPromptWithParagraphs(prompt, targetTokens)
}
// 短咨询提示词,使用填充词
logger.Debug("使用填充词扩展短咨询提示词")
// 添加填充词,直到达到目标Token数
fillerCount := 0
for currentTokens < minTokens {
// 随机选择一个填充词
fillerIndex := g.rand.Intn(len(fillerWords))
filler := fillerWords[fillerIndex]
// 在提示词前添加填充词
result = filler + " " + result
fillerCount++
// 重新计算Token数
tokens = g.tokenizer.Encode(result, nil, nil)
currentTokens = len(tokens)
// 防止无限循环
if currentTokens > int(float64(targetTokens)*1.05) {
// 如果添加填充词后超过了最大Token数,回退并尝试截断
logger.Debug("添加%d个填充词后超过最大Token数,尝试截断", fillerCount)
return g.truncatePrompt(result, targetTokens)
}
if fillerCount > 100 {
logger.Warn("添加填充词超过100次,尝试使用段落填充")
return g.extendPromptWithParagraphs(prompt, targetTokens)
}
}
logger.Debug("扩展提示词成功,添加了%d个填充词,当前Token数: %d", fillerCount, currentTokens)
return result, nil
}
// extendPromptWithParagraphs 通过添加段落扩展长文档提示词
func (g *Generator) extendPromptWithParagraphs(prompt string, targetTokens int) (string, error) {
result := prompt
// 计算当前Token数
tokens := g.tokenizer.Encode(result, nil, nil)
currentTokens := len(tokens)
// 计算目标范围
minTokens := int(float64(targetTokens) * 0.95)
// 添加段落,直到达到目标Token数
paragraphCount := 0
usedParagraphs := make(map[int]bool)
for currentTokens < minTokens && paragraphCount < 20 {
// 随机选择一个未使用的段落
var paragraphIndex int
for {
paragraphIndex = g.rand.Intn(len(longFillerParagraphs))
if !usedParagraphs[paragraphIndex] {
usedParagraphs[paragraphIndex] = true
break
}
// 如果所有段落都已使用,重置使用记录
if len(usedParagraphs) >= len(longFillerParagraphs) {
usedParagraphs = make(map[int]bool)
}
}
paragraph := longFillerParagraphs[paragraphIndex]
// 添加段落
if paragraphCount == 0 {
result = result + "\n\n" + paragraph
} else {
result = result + "\n\n" + paragraph
}
paragraphCount++
// 重新计算Token数
tokens = g.tokenizer.Encode(result, nil, nil)
currentTokens = len(tokens)
logger.Debug("添加第%d个段落后,当前Token数: %d", paragraphCount, currentTokens)
// 如果已经达到目标范围,退出循环
if currentTokens >= minTokens {
break
}
}
// 如果添加了最大段落数仍未达到目标,给出警告
if currentTokens < minTokens {
logger.Warn("即使添加了%d个段落,仍未达到目标Token数(当前: %d, 目标: %d)",
paragraphCount, currentTokens, targetTokens)
} else {
logger.Debug("扩展长文档提示词成功,添加了%d个段落,当前Token数: %d", paragraphCount, currentTokens)
}
return result, nil
}
// truncatePrompt 通过截断缩短提示词
func (g *Generator) truncatePrompt(prompt string, targetTokens int) (string, error) {
// 计算目标范围
maxTokens := int(float64(targetTokens) * 1.05)
// 编码提示词
tokens := g.tokenizer.Encode(prompt, nil, nil)
// 如果Token数超过最大值,截断
if len(tokens) > maxTokens {
// 保留目标Token数
originalLength := len(tokens)
tokens = tokens[:maxTokens]
// 解码截断后的Token
result := g.tokenizer.Decode(tokens)
logger.Debug("截断提示词成功,从%d个Token截断到%d个Token", originalLength, len(tokens))
return result, nil
}
return prompt, nil
}
// SelectPromptType 根据权重随机选择提示词类型
func (g *Generator) SelectPromptType() string {
// 计算累积权重
var cumulativeWeights []float64
var sum float64
for _, req := range g.config.Requests {
sum += req.Weight
cumulativeWeights = append(cumulativeWeights, sum)
}
// 生成随机数
r := g.rand.Float64() * sum
// 根据随机数选择提示词类型
for i, weight := range cumulativeWeights {
if r <= weight {
return g.config.Requests[i].Type
}
}
// 默认返回第一个类型
return g.config.Requests[0].Type
}
-635
View File
@@ -1,635 +0,0 @@
package report
import (
"fmt"
"os"
"path/filepath"
"sort"
"time"
"github.com/acmestudio/llm-api-benchmark-tool/benchmark"
"github.com/acmestudio/llm-api-benchmark-tool/config"
"github.com/acmestudio/llm-api-benchmark-tool/logger"
"github.com/go-echarts/go-echarts/v2/charts"
"github.com/go-echarts/go-echarts/v2/components"
"github.com/go-echarts/go-echarts/v2/opts"
"github.com/montanaflynn/stats"
)
// Metrics 表示性能指标
type Metrics struct {
TotalRequests int // 总请求数
SuccessfulRequests int // 成功请求数
FailedRequests int // 失败请求数
TimeoutRatio float64 // 超时比率
AvgResponseTime time.Duration // 平均响应时间
P90ResponseTime time.Duration // P90响应时间
P95ResponseTime time.Duration // P95响应时间
P99ResponseTime time.Duration // P99响应时间
MaxResponseTime time.Duration // 最大响应时间
MinResponseTime time.Duration // 最小响应时间
AvgQPS float64 // 平均QPS
MaxQPS float64 // 最大QPS
AvgTokenRate float64 // 平均Token生成速率
MaxTokenRate float64 // 最大Token生成速率
MaxConcurrency int // 最大有效并发用户数
}
// GenerateReport 生成性能报告
func GenerateReport(results *benchmark.BenchmarkResults, cfg *config.Config, outputPath string) error {
// 创建输出目录
logger.Info("创建报告输出目录: %s", outputPath)
if err := os.MkdirAll(outputPath, 0755); err != nil {
return fmt.Errorf("创建输出目录失败: %w", err)
}
// 检查结果是否为空
if len(results.Results) == 0 {
logger.Warn("没有测试结果数据,无法生成详细报告")
} else {
logger.Debug("共有 %d 条测试结果记录", len(results.Results))
}
// 计算性能指标
logger.Info("计算性能指标...")
metrics := calculateMetrics(results)
// 输出主要性能指标
logger.Info("性能指标摘要:")
logger.Info(" 总请求数: %d (成功: %d, 失败: %d)",
metrics.TotalRequests, metrics.SuccessfulRequests, metrics.FailedRequests)
logger.Info(" 超时比率: %.2f%%", metrics.TimeoutRatio * 100)
logger.Info(" 平均响应时间: %v", metrics.AvgResponseTime)
logger.Info(" P90响应时间: %v", metrics.P90ResponseTime)
logger.Info(" P95响应时间: %v", metrics.P95ResponseTime)
logger.Info(" P99响应时间: %v", metrics.P99ResponseTime)
logger.Info(" 平均QPS: %.2f", metrics.AvgQPS)
logger.Info(" 最大QPS: %.2f", metrics.MaxQPS)
logger.Info(" 平均Token生成速率: %.2f tokens/s", metrics.AvgTokenRate)
logger.Info(" 最大Token生成速率: %.2f tokens/s", metrics.MaxTokenRate)
logger.Info(" 最大有效并发用户数: %d", metrics.MaxConcurrency)
// 生成HTML报告
logger.Info("生成HTML报告...")
if err := generateHTMLReport(results, metrics, cfg, outputPath); err != nil {
return fmt.Errorf("生成HTML报告失败: %w", err)
}
// 生成CSV数据
logger.Info("生成CSV数据...")
if err := generateCSVData(results, outputPath); err != nil {
return fmt.Errorf("生成CSV数据失败: %w", err)
}
logger.Info("报告生成完成")
return nil
}
// calculateMetrics 计算性能指标
func calculateMetrics(results *benchmark.BenchmarkResults) *Metrics {
metrics := &Metrics{}
// 总请求数
metrics.TotalRequests = len(results.Results)
logger.Debug("计算指标 - 总请求数: %d", metrics.TotalRequests)
// 统计成功和失败请求
var successfulRequests []benchmark.Result
for _, result := range results.Results {
if result.Error == nil {
successfulRequests = append(successfulRequests, result)
}
}
metrics.SuccessfulRequests = len(successfulRequests)
metrics.FailedRequests = metrics.TotalRequests - metrics.SuccessfulRequests
logger.Debug("计算指标 - 成功请求: %d, 失败请求: %d",
metrics.SuccessfulRequests, metrics.FailedRequests)
// 计算超时比率
metrics.TimeoutRatio = float64(metrics.FailedRequests) / float64(metrics.TotalRequests)
logger.Debug("计算指标 - 超时比率: %.2f%%", metrics.TimeoutRatio * 100)
// 如果没有成功的请求,返回
if len(successfulRequests) == 0 {
logger.Warn("没有成功的请求,无法计算详细指标")
return metrics
}
// 计算响应时间指标
var responseTimes []float64
for _, result := range successfulRequests {
responseTimes = append(responseTimes, float64(result.ResponseTime.Milliseconds()))
}
// 排序响应时间
sort.Float64s(responseTimes)
// 计算响应时间百分位数
p90, _ := stats.Percentile(responseTimes, 90)
p95, _ := stats.Percentile(responseTimes, 95)
p99, _ := stats.Percentile(responseTimes, 99)
avg, _ := stats.Mean(responseTimes)
max := responseTimes[len(responseTimes)-1]
min := responseTimes[0]
metrics.AvgResponseTime = time.Duration(avg) * time.Millisecond
metrics.P90ResponseTime = time.Duration(p90) * time.Millisecond
metrics.P95ResponseTime = time.Duration(p95) * time.Millisecond
metrics.P99ResponseTime = time.Duration(p99) * time.Millisecond
metrics.MaxResponseTime = time.Duration(max) * time.Millisecond
metrics.MinResponseTime = time.Duration(min) * time.Millisecond
logger.Debug("计算指标 - 响应时间(ms) - 平均: %.2f, P90: %.2f, P95: %.2f, P99: %.2f, 最小: %.2f, 最大: %.2f",
avg, p90, p95, p99, min, max)
// 计算QPS
totalDuration := results.EndTime.Sub(results.StartTime).Seconds()
metrics.AvgQPS = float64(metrics.SuccessfulRequests) / totalDuration
logger.Debug("计算指标 - 平均QPS: %.2f (总持续时间: %.2f秒)", metrics.AvgQPS, totalDuration)
// 计算每个并发步骤的QPS
logger.Debug("计算各并发步骤的QPS...")
for concurrency, stepResults := range results.ConcurrencyData {
// 找出该步骤的第一个和最后一个请求
if len(stepResults) == 0 {
logger.Debug(" 并发步骤 %d: 没有请求数据", concurrency)
continue
}
// 排序结果,按时间戳
sort.Slice(stepResults, func(i, j int) bool {
return stepResults[i].Timestamp.Before(stepResults[j].Timestamp)
})
firstRequest := stepResults[0]
lastRequest := stepResults[len(stepResults)-1]
// 计算持续时间
stepDuration := lastRequest.Timestamp.Sub(firstRequest.Timestamp).Seconds()
if stepDuration <= 0 {
logger.Debug(" 并发步骤 %d: 持续时间过短", concurrency)
continue
}
// 计算成功请求数
var successCount int
for _, result := range stepResults {
if result.Error == nil {
successCount++
}
}
// 计算QPS
stepQPS := float64(successCount) / stepDuration
logger.Debug(" 并发步骤 %d: QPS = %.2f (成功请求: %d, 持续时间: %.2f秒)",
concurrency, stepQPS, successCount, stepDuration)
if stepQPS > metrics.MaxQPS {
metrics.MaxQPS = stepQPS
logger.Debug(" 更新最大QPS: %.2f (并发步骤: %d)", metrics.MaxQPS, concurrency)
}
// 更新最大并发用户数
if concurrency > metrics.MaxConcurrency && stepQPS > 0 {
metrics.MaxConcurrency = concurrency
logger.Debug(" 更新最大有效并发用户数: %d", metrics.MaxConcurrency)
}
}
// 计算Token生成速率
var totalTokens int
for _, result := range successfulRequests {
totalTokens += result.CompletionTokens
}
metrics.AvgTokenRate = float64(totalTokens) / totalDuration
logger.Debug("计算指标 - 平均Token生成速率: %.2f tokens/s (总Token: %d)",
metrics.AvgTokenRate, totalTokens)
// 计算每个并发步骤的Token生成速率
logger.Debug("计算各并发步骤的Token生成速率...")
for concurrency, stepResults := range results.ConcurrencyData {
// 找出该步骤的第一个和最后一个请求
if len(stepResults) == 0 {
continue
}
// 排序结果,按时间戳
sort.Slice(stepResults, func(i, j int) bool {
return stepResults[i].Timestamp.Before(stepResults[j].Timestamp)
})
firstRequest := stepResults[0]
lastRequest := stepResults[len(stepResults)-1]
// 计算持续时间
stepDuration := lastRequest.Timestamp.Sub(firstRequest.Timestamp).Seconds()
if stepDuration <= 0 {
continue
}
// 计算总Token数
var stepTokens int
for _, result := range stepResults {
if result.Error == nil {
stepTokens += result.CompletionTokens
}
}
// 计算Token生成速率
stepTokenRate := float64(stepTokens) / stepDuration
logger.Debug(" 并发步骤 %d: Token生成速率 = %.2f tokens/s (总Token: %d, 持续时间: %.2f秒)",
concurrency, stepTokenRate, stepTokens, stepDuration)
if stepTokenRate > metrics.MaxTokenRate {
metrics.MaxTokenRate = stepTokenRate
logger.Debug(" 更新最大Token生成速率: %.2f tokens/s (并发步骤: %d)",
metrics.MaxTokenRate, concurrency)
}
}
return metrics
}
// generateHTMLReport 生成HTML报告
func generateHTMLReport(results *benchmark.BenchmarkResults, metrics *Metrics, cfg *config.Config, outputPath string) error {
// 创建页面
logger.Debug("创建HTML报告页面...")
page := components.NewPage()
page.PageTitle = "LLM API 基准测试报告"
page.Theme = "white"
// 添加概述
logger.Debug("添加概述图表...")
page.AddCharts(createOverviewChart(results, metrics, cfg))
// 添加响应时间分布图
logger.Debug("添加响应时间分布图...")
page.AddCharts(createResponseTimeDistributionChart(results))
// 添加QPS随时间变化图
logger.Debug("添加QPS随时间变化图...")
page.AddCharts(createQPSOverTimeChart(results))
// 添加Token生成速率随时间变化图
logger.Debug("添加Token生成速率随时间变化图...")
page.AddCharts(createTokenRateOverTimeChart(results))
// 添加并发与响应时间关系图
logger.Debug("添加并发与响应时间关系图...")
page.AddCharts(createConcurrencyVsResponseTimeChart(results))
// 保存HTML报告
htmlPath := filepath.Join(outputPath, "report.html")
logger.Debug("保存HTML报告: %s", htmlPath)
f, err := os.Create(htmlPath)
if err != nil {
return err
}
defer f.Close()
return page.Render(f)
}
// createOverviewChart 创建概述图表
func createOverviewChart(results *benchmark.BenchmarkResults, metrics *Metrics, cfg *config.Config) *charts.Bar {
// 创建图表
chart := charts.NewBar()
chart.SetGlobalOptions(
charts.WithTitleOpts(opts.Title{
Title: "性能指标概述",
}),
)
// 添加数据
chart.SetXAxis([]string{"平均响应时间 (ms)", "P90响应时间 (ms)", "P95响应时间 (ms)", "P99响应时间 (ms)"})
chart.AddSeries("响应时间", []opts.BarData{
{Value: metrics.AvgResponseTime.Milliseconds()},
{Value: metrics.P90ResponseTime.Milliseconds()},
{Value: metrics.P95ResponseTime.Milliseconds()},
{Value: metrics.P99ResponseTime.Milliseconds()},
})
return chart
}
// createResponseTimeDistributionChart 创建响应时间分布图
func createResponseTimeDistributionChart(results *benchmark.BenchmarkResults) *charts.Bar {
// 创建图表
chart := charts.NewBar()
chart.SetGlobalOptions(
charts.WithTitleOpts(opts.Title{
Title: "响应时间分布",
}),
)
// 统计响应时间分布
timeRanges := []int{100, 200, 500, 1000, 2000, 5000, 10000}
counts := make([]int, len(timeRanges)+1)
for _, result := range results.Results {
if result.Error != nil {
continue
}
responseTime := result.ResponseTime.Milliseconds()
found := false
for i, timeRange := range timeRanges {
if responseTime <= int64(timeRange) {
counts[i]++
found = true
break
}
}
if !found {
counts[len(timeRanges)]++
}
}
// 构建X轴标签
xAxisLabels := make([]string, len(timeRanges)+1)
for i, timeRange := range timeRanges {
if i == 0 {
xAxisLabels[i] = fmt.Sprintf("≤%dms", timeRange)
} else {
xAxisLabels[i] = fmt.Sprintf("%d-%dms", timeRanges[i-1], timeRange)
}
}
xAxisLabels[len(timeRanges)] = fmt.Sprintf(">%dms", timeRanges[len(timeRanges)-1])
// 添加数据
chart.SetXAxis(xAxisLabels)
barData := make([]opts.BarData, len(counts))
for i, count := range counts {
barData[i] = opts.BarData{Value: count}
}
chart.AddSeries("请求数", barData)
return chart
}
// createQPSOverTimeChart 创建QPS随时间变化图
func createQPSOverTimeChart(results *benchmark.BenchmarkResults) *charts.Line {
// 创建图表
chart := charts.NewLine()
chart.SetGlobalOptions(
charts.WithTitleOpts(opts.Title{
Title: "QPS随时间变化",
}),
)
// 按并发步骤分组
for concurrency, stepResults := range results.ConcurrencyData {
// 按时间窗口统计QPS
windowSize := 5 * time.Second
// 找出该步骤的第一个和最后一个请求
if len(stepResults) == 0 {
continue
}
// 排序结果,按时间戳
sort.Slice(stepResults, func(i, j int) bool {
return stepResults[i].Timestamp.Before(stepResults[j].Timestamp)
})
firstRequest := stepResults[0]
lastRequest := stepResults[len(stepResults)-1]
// 计算时间窗口数
duration := lastRequest.Timestamp.Sub(firstRequest.Timestamp)
windowCount := int(duration / windowSize) + 1
// 初始化时间窗口
windows := make([]time.Time, windowCount)
counts := make([]int, windowCount)
for i := 0; i < windowCount; i++ {
windows[i] = firstRequest.Timestamp.Add(time.Duration(i) * windowSize)
}
// 统计每个时间窗口的请求数
for _, result := range stepResults {
if result.Error != nil {
continue
}
windowIndex := int(result.Timestamp.Sub(firstRequest.Timestamp) / windowSize)
if windowIndex >= 0 && windowIndex < windowCount {
counts[windowIndex]++
}
}
// 计算QPS
qpsData := make([]opts.LineData, windowCount)
xAxisData := make([]string, windowCount)
for i := 0; i < windowCount; i++ {
qps := float64(counts[i]) / windowSize.Seconds()
qpsData[i] = opts.LineData{Value: qps}
xAxisData[i] = windows[i].Format("15:04:05")
}
// 添加数据
chart.SetXAxis(xAxisData)
chart.AddSeries(fmt.Sprintf("并发 %d", concurrency), qpsData)
}
return chart
}
// createTokenRateOverTimeChart 创建Token生成速率随时间变化图
func createTokenRateOverTimeChart(results *benchmark.BenchmarkResults) *charts.Line {
// 创建图表
chart := charts.NewLine()
chart.SetGlobalOptions(
charts.WithTitleOpts(opts.Title{
Title: "Token生成速率随时间变化",
}),
)
// 按并发步骤分组
for concurrency, stepResults := range results.ConcurrencyData {
// 按时间窗口统计Token生成速率
windowSize := 5 * time.Second
// 找出该步骤的第一个和最后一个请求
if len(stepResults) == 0 {
continue
}
// 排序结果,按时间戳
sort.Slice(stepResults, func(i, j int) bool {
return stepResults[i].Timestamp.Before(stepResults[j].Timestamp)
})
firstRequest := stepResults[0]
lastRequest := stepResults[len(stepResults)-1]
// 计算时间窗口数
duration := lastRequest.Timestamp.Sub(firstRequest.Timestamp)
windowCount := int(duration / windowSize) + 1
// 初始化时间窗口
windows := make([]time.Time, windowCount)
tokens := make([]int, windowCount)
for i := 0; i < windowCount; i++ {
windows[i] = firstRequest.Timestamp.Add(time.Duration(i) * windowSize)
}
// 统计每个时间窗口的Token数
for _, result := range stepResults {
if result.Error != nil {
continue
}
windowIndex := int(result.Timestamp.Sub(firstRequest.Timestamp) / windowSize)
if windowIndex >= 0 && windowIndex < windowCount {
tokens[windowIndex] += result.CompletionTokens
}
}
// 计算Token生成速率
tokenRateData := make([]opts.LineData, windowCount)
xAxisData := make([]string, windowCount)
for i := 0; i < windowCount; i++ {
tokenRate := float64(tokens[i]) / windowSize.Seconds()
tokenRateData[i] = opts.LineData{Value: tokenRate}
xAxisData[i] = windows[i].Format("15:04:05")
}
// 添加数据
chart.SetXAxis(xAxisData)
chart.AddSeries(fmt.Sprintf("并发 %d", concurrency), tokenRateData)
}
return chart
}
// createConcurrencyVsResponseTimeChart 创建并发与响应时间关系图
func createConcurrencyVsResponseTimeChart(results *benchmark.BenchmarkResults) *charts.Line {
// 创建图表
chart := charts.NewLine()
chart.SetGlobalOptions(
charts.WithTitleOpts(opts.Title{
Title: "并发与响应时间关系",
}),
)
// 准备数据
var concurrencies []int
var avgResponseTimes []opts.LineData
var p90ResponseTimes []opts.LineData
var p95ResponseTimes []opts.LineData
// 按并发步骤排序
for concurrency := range results.ConcurrencyData {
concurrencies = append(concurrencies, concurrency)
}
sort.Ints(concurrencies)
// 计算每个并发步骤的响应时间
for _, concurrency := range concurrencies {
stepResults := results.ConcurrencyData[concurrency]
// 收集成功请求的响应时间
var responseTimes []float64
for _, result := range stepResults {
if result.Error == nil {
responseTimes = append(responseTimes, float64(result.ResponseTime.Milliseconds()))
}
}
// 如果没有成功的请求,跳过
if len(responseTimes) == 0 {
avgResponseTimes = append(avgResponseTimes, opts.LineData{Value: 0})
p90ResponseTimes = append(p90ResponseTimes, opts.LineData{Value: 0})
p95ResponseTimes = append(p95ResponseTimes, opts.LineData{Value: 0})
continue
}
// 计算响应时间指标
avg, _ := stats.Mean(responseTimes)
p90, _ := stats.Percentile(responseTimes, 90)
p95, _ := stats.Percentile(responseTimes, 95)
avgResponseTimes = append(avgResponseTimes, opts.LineData{Value: avg})
p90ResponseTimes = append(p90ResponseTimes, opts.LineData{Value: p90})
p95ResponseTimes = append(p95ResponseTimes, opts.LineData{Value: p95})
}
// 将并发数转换为字符串
xAxisData := make([]string, len(concurrencies))
for i, concurrency := range concurrencies {
xAxisData[i] = fmt.Sprintf("%d", concurrency)
}
// 添加数据
chart.SetXAxis(xAxisData)
chart.AddSeries("平均响应时间 (ms)", avgResponseTimes)
chart.AddSeries("P90响应时间 (ms)", p90ResponseTimes)
chart.AddSeries("P95响应时间 (ms)", p95ResponseTimes)
return chart
}
// generateCSVData 生成CSV数据
func generateCSVData(results *benchmark.BenchmarkResults, outputPath string) error {
// 创建CSV文件
csvPath := filepath.Join(outputPath, "results.csv")
logger.Debug("创建CSV文件: %s", csvPath)
f, err := os.Create(csvPath)
if err != nil {
return err
}
defer f.Close()
// 写入CSV头
logger.Debug("写入CSV头...")
_, err = f.WriteString("Timestamp,PromptType,PromptTokens,CompletionTokens,ResponseTime,Concurrency,Error\n")
if err != nil {
return err
}
// 检查结果是否为空
if len(results.Results) == 0 {
logger.Warn("没有测试结果数据,CSV文件将只包含表头")
return nil
}
// 写入CSV数据
logger.Debug("写入CSV数据,共%d条记录...", len(results.Results))
for _, result := range results.Results {
// 格式化错误信息
errorMsg := ""
if result.Error != nil {
errorMsg = result.Error.Error()
}
// 写入一行数据
line := fmt.Sprintf("%s,%s,%d,%d,%d,%d,%s\n",
result.Timestamp.Format(time.RFC3339),
result.PromptType,
result.PromptTokens,
result.CompletionTokens,
result.ResponseTime.Milliseconds(),
result.Concurrency,
errorMsg,
)
_, err = f.WriteString(line)
if err != nil {
return err
}
}
logger.Debug("CSV数据写入完成")
return nil
}
@@ -1,104 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>LLM API 基准测试报告</title>
<script src="https://go-echarts.github.io/go-echarts-assets/assets/echarts.min.js"></script>
</head>
<body>
<style> .container {display: flex;justify-content: center;align-items: center;} .item {margin: auto;} </style>
<div class="container">
<div class="item" id="LNbBAPYnepJk" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_LNbBAPYnepJk = echarts.init(document.getElementById('LNbBAPYnepJk'), "white");
let option_LNbBAPYnepJk = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"响应时间","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":40867},{"value":55877},{"value":57093},{"value":59096}]}],"title":{"text":"性能指标概述"},"tooltip":{"show":false},"xAxis":[{"data":["平均响应时间 (ms)","P90响应时间 (ms)","P95响应时间 (ms)","P99响应时间 (ms)"]}],"yAxis":[{}]}
;
let action_LNbBAPYnepJk = {"areas":{},"type":""}
;
goecharts_LNbBAPYnepJk.setOption(option_LNbBAPYnepJk);
goecharts_LNbBAPYnepJk.dispatchAction(action_LNbBAPYnepJk);
</script>
<div class="container">
<div class="item" id="GOPBARBCnPTU" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_GOPBARBCnPTU = echarts.init(document.getElementById('GOPBARBCnPTU'), "white");
let option_GOPBARBCnPTU = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"请求数","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":1},{"value":108}]}],"title":{"text":"响应时间分布"},"tooltip":{"show":false},"xAxis":[{"data":["≤100ms","100-200ms","200-500ms","500-1000ms","1000-2000ms","2000-5000ms","5000-10000ms",">10000ms"]}],"yAxis":[{}]}
;
let action_GOPBARBCnPTU = {"areas":{},"type":""}
;
goecharts_GOPBARBCnPTU.setOption(option_GOPBARBCnPTU);
goecharts_GOPBARBCnPTU.dispatchAction(action_GOPBARBCnPTU);
</script>
<div class="container">
<div class="item" id="XkXZxRAArLPb" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_XkXZxRAArLPb = echarts.init(document.getElementById('XkXZxRAArLPb'), "white");
let option_XkXZxRAArLPb = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"并发 1","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0}]},{"name":"并发 5","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":0.8,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.4,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2Line truncated
;
let action_XkXZxRAArLPb = {"areas":{},"type":""}
;
goecharts_XkXZxRAArLPb.setOption(option_XkXZxRAArLPb);
goecharts_XkXZxRAArLPb.dispatchAction(action_XkXZxRAArLPb);
</script>
<div class="container">
<div class="item" id="OFgvWOHKVKpD" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_OFgvWOHKVKpD = echarts.init(document.getElementById('OFgvWOHKVKpD'), "white");
let option_OFgvWOHKVKpD = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"并发 1","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":353.4,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":248,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":316,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":347.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":237,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":223.6,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":215,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":338.6,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":289.8,"XAxisIndex":0,"YAxisIndex":0}]},{"name":"并发 5","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":1158.8,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":295.8,"XAxisIndex":0,"YAxisIndex":0},{"value":235.4,"XAxisIndex":0,"YAxisIndex":0},{"value":355.6,"XAxisIndex":0,"YAxisIndex":0},{"value":237.8,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":278,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":132.4,"XAxisIndex":0,"YAxisIndex":0},{"value":273.2,"XAxisIndex":0,"YAxisIndex":0},{"value":348.6,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":710.4,"XAxisIndex":0,"YAxisIndex":0},{"value":268.4,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":353.8,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":125,"XAxisIndex":0,"YAxisIndex":0},{"value":298.8,"XAxisIndex":0,"YAxisIndex":0},{"value":306,"XAxisIndex":0,"YAxisIndex":0},{"value":41.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":292.6,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxiLine truncated
;
let action_OFgvWOHKVKpD = {"areas":{},"type":""}
;
goecharts_OFgvWOHKVKpD.setOption(option_OFgvWOHKVKpD);
goecharts_OFgvWOHKVKpD.dispatchAction(action_OFgvWOHKVKpD);
</script>
<div class="container">
<div class="item" id="PSQeXGkwaeXv" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_PSQeXGkwaeXv = echarts.init(document.getElementById('PSQeXGkwaeXv'), "white");
let option_PSQeXGkwaeXv = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"平均响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":41457,"XAxisIndex":0,"YAxisIndex":0},{"value":40918.05263157895,"XAxisIndex":0,"YAxisIndex":0},{"value":40750.27419354839,"XAxisIndex":0,"YAxisIndex":0}]},{"name":"P90响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":50759,"XAxisIndex":0,"YAxisIndex":0},{"value":55646.5,"XAxisIndex":0,"YAxisIndex":0},{"value":55877.5,"XAxisIndex":0,"YAxisIndex":0}]},{"name":"P95响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":50759,"XAxisIndex":0,"YAxisIndex":0},{"value":56847.5,"XAxisIndex":0,"YAxisIndex":0},{"value":57093,"XAxisIndex":0,"YAxisIndex":0}]}],"title":{"text":"并发与响应时间关系"},"tooltip":{"show":false},"xAxis":[{"data":["1","5","10"]}],"yAxis":[{}]}
;
let action_PSQeXGkwaeXv = {"areas":{},"type":""}
;
goecharts_PSQeXGkwaeXv.setOption(option_PSQeXGkwaeXv);
goecharts_PSQeXGkwaeXv.dispatchAction(action_PSQeXGkwaeXv);
</script>
</body>
</html>
@@ -1,125 +0,0 @@
Timestamp,PromptType,PromptTokens,ResponseTokens,ResponseTime,ConcurrencyStep,Error
2025-04-21T15:29:08+08:00,long,290,1767,50692,1,
2025-04-21T15:29:59+08:00,short,32,1240,35591,1,
2025-04-21T15:30:35+08:00,long,299,1580,46800,1,
2025-04-21T15:31:22+08:00,long,285,1736,50826,1,
2025-04-21T15:32:13+08:00,long,294,1185,35257,1,
2025-04-21T15:32:48+08:00,long,278,1118,33428,1,
2025-04-21T15:33:22+08:00,short,30,1075,31594,1,
2025-04-21T15:33:54+08:00,short,35,1693,48817,1,
2025-04-21T15:34:43+08:00,short,30,1449,40108,1,
2025-04-21T15:35:23+08:00,short,31,1267,39528,5,
2025-04-21T15:35:23+08:00,short,34,1331,43050,5,
2025-04-21T15:35:23+08:00,long,281,1455,47254,5,
2025-04-21T15:35:23+08:00,short,34,1741,54033,5,
2025-04-21T15:35:23+08:00,short,0,0,60000,5,发送请求失败: timeout
2025-04-21T15:36:07+08:00,short,34,1177,37663,5,
2025-04-21T15:36:03+08:00,long,298,1479,47924,5,
2025-04-21T15:36:18+08:00,short,36,1189,39376,5,
2025-04-21T15:36:11+08:00,long,300,1778,56227,5,
2025-04-21T15:36:45+08:00,short,30,662,22887,5,
2025-04-21T15:36:23+08:00,long,295,1390,44672,5,
2025-04-21T15:36:51+08:00,long,306,1366,44047,5,
2025-04-21T15:37:08+08:00,long,298,1342,43062,5,
2025-04-21T15:36:57+08:00,long,291,1743,56131,5,
2025-04-21T15:37:08+08:00,short,31,1675,52352,5,
2025-04-21T15:37:07+08:00,long,294,1877,58539,5,
2025-04-21T15:37:52+08:00,short,31,625,19803,5,
2025-04-21T15:38:06+08:00,short,33,206,7176,5,
2025-04-21T15:37:35+08:00,long,288,1769,55162,5,
2025-04-21T15:37:54+08:00,long,296,1494,47829,5,
2025-04-21T15:38:01+08:00,long,295,1530,50380,5,
2025-04-21T15:38:13+08:00,short,38,1463,45724,5,
2025-04-21T15:38:31+08:00,short,34,906,28607,5,
2025-04-21T15:38:42+08:00,short,33,691,21916,5,
2025-04-21T15:38:12+08:00,long,0,0,60000,5,发送请求失败: timeout
2025-04-21T15:38:59+08:00,short,34,676,22665,5,
2025-04-21T15:38:51+08:00,short,34,1006,30879,5,
2025-04-21T15:38:59+08:00,short,28,1185,36817,5,
2025-04-21T15:39:04+08:00,long,281,1688,53184,5,
2025-04-21T15:39:22+08:00,short,34,1192,35771,5,
2025-04-21T15:39:37+08:00,short,35,922,29691,5,
2025-04-21T15:39:12+08:00,long,299,1814,55118,5,
2025-04-21T15:39:22+08:00,short,32,1555,46992,5,
2025-04-21T15:40:07+08:00,short,33,777,25157,5,
2025-04-21T15:39:57+08:00,short,29,1375,41815,5,
2025-04-21T15:40:09+08:00,long,292,1278,38409,5,
2025-04-21T15:40:06+08:00,long,290,1586,45621,5,
2025-04-21T15:39:59+08:00,long,301,1874,57468,5,
2025-04-21T15:40:52+08:00,long,287,1565,45984,5,
2025-04-21T15:41:39+08:00,short,34,859,25973,5,
2025-04-21T15:42:05+08:00,short,33,1031,36665,10,
2025-04-21T15:42:05+08:00,short,31,1032,38132,10,
2025-04-21T15:42:05+08:00,short,33,1124,41458,10,
2025-04-21T15:42:05+08:00,short,31,1209,42973,10,
2025-04-21T15:42:05+08:00,short,36,1260,45000,10,
2025-04-21T15:42:05+08:00,long,301,1254,45741,10,
2025-04-21T15:42:05+08:00,short,0,0,60025,10,发送请求失败: timeout
2025-04-21T15:42:05+08:00,long,0,0,60024,10,发送请求失败: timeout
2025-04-21T15:42:05+08:00,long,0,0,60024,10,发送请求失败: timeout
2025-04-21T15:42:05+08:00,long,0,0,60027,10,发送请求失败: timeout
2025-04-21T15:42:43+08:00,short,30,627,24785,10,
2025-04-21T15:42:41+08:00,short,33,880,33993,10,
2025-04-21T15:42:46+08:00,short,33,763,29495,10,
2025-04-21T15:42:48+08:00,short,32,867,34001,10,
2025-04-21T15:42:51+08:00,short,32,878,33547,10,
2025-04-21T15:43:08+08:00,short,34,959,36283,10,
2025-04-21T15:43:05+08:00,short,32,1059,40263,10,
2025-04-21T15:42:50+08:00,long,0,0,60001,10,发送请求失败: timeout
2025-04-21T15:43:05+08:00,long,302,1515,55759,10,
2025-04-21T15:43:05+08:00,short,32,1480,55895,10,
2025-04-21T15:43:05+08:00,long,0,0,60025,10,发送请求失败: timeout
2025-04-21T15:43:44+08:00,short,34,663,25933,10,
2025-04-21T15:43:16+08:00,long,302,1503,55131,10,
2025-04-21T15:43:24+08:00,short,34,1385,50030,10,
2025-04-21T15:43:16+08:00,long,290,1646,59167,10,
2025-04-21T15:43:22+08:00,short,31,1512,56955,10,
2025-04-21T15:43:45+08:00,short,28,1248,47396,10,
2025-04-21T15:44:05+08:00,short,34,945,35028,10,
2025-04-21T15:44:01+08:00,short,35,1134,43283,10,
2025-04-21T15:44:15+08:00,short,35,921,32485,10,
2025-04-21T15:43:50+08:00,short,34,1628,59025,10,
2025-04-21T15:44:11+08:00,short,31,1160,42541,10,
2025-04-21T15:44:15+08:00,short,34,1036,41375,10,
2025-04-21T15:44:01+08:00,long,286,1507,56831,10,
2025-04-21T15:44:48+08:00,short,28,345,13944,10,
2025-04-21T15:44:33+08:00,short,28,965,35746,10,
2025-04-21T15:44:10+08:00,short,34,1640,59733,10,
2025-04-21T15:44:19+08:00,short,31,1563,55769,10,
2025-04-21T15:44:54+08:00,short,37,721,27026,10,
2025-04-21T15:44:44+08:00,long,291,1268,45194,10,
2025-04-21T15:44:58+08:00,short,30,800,31400,10,
2025-04-21T15:44:40+08:00,long,287,1422,54009,10,
2025-04-21T15:44:49+08:00,short,32,1479,52350,10,
2025-04-21T15:45:15+08:00,short,31,728,27863,10,
2025-04-21T15:45:09+08:00,short,35,1127,42373,10,
2025-04-21T15:44:56+08:00,short,0,0,60001,10,发送请求失败: timeout
2025-04-21T15:45:02+08:00,short,34,1491,55425,10,
2025-04-21T15:45:10+08:00,short,0,0,60001,10,发送请求失败: timeout
2025-04-21T15:45:43+08:00,short,31,814,32256,10,
2025-04-21T15:45:21+08:00,long,0,0,60001,10,发送请求失败: timeout
2025-04-21T15:45:51+08:00,short,33,808,31357,10,
2025-04-21T15:45:30+08:00,long,302,1447,54647,10,
2025-04-21T15:45:56+08:00,short,36,761,29637,10,
2025-04-21T15:45:42+08:00,short,34,1214,45591,10,
2025-04-21T15:45:35+08:00,short,29,1465,54649,10,
2025-04-21T15:45:29+08:00,long,0,0,60000,10,发送请求失败: timeout
2025-04-21T15:45:58+08:00,short,35,874,34326,10,
2025-04-21T15:46:10+08:00,short,27,1111,41640,10,
2025-04-21T15:46:21+08:00,short,32,816,32250,10,
2025-04-21T15:46:29+08:00,short,32,709,25910,10,
2025-04-21T15:46:25+08:00,short,37,863,31820,10,
2025-04-21T15:46:32+08:00,short,31,652,24537,10,
2025-04-21T15:46:28+08:00,short,30,1034,37449,10,
2025-04-21T15:46:15+08:00,long,0,0,60001,10,发送请求失败: timeout
2025-04-21T15:46:54+08:00,short,28,607,23222,10,
2025-04-21T15:46:26+08:00,long,294,1390,51678,10,
2025-04-21T15:46:56+08:00,short,28,705,26915,10,
2025-04-21T15:46:23+08:00,long,0,0,60001,10,发送请求失败: timeout
2025-04-21T15:46:57+08:00,short,33,818,30689,10,
2025-04-21T15:46:30+08:00,short,0,0,60030,10,发送请求失败: timeout
2025-04-21T15:46:57+08:00,short,37,1139,40595,10,
2025-04-21T15:46:52+08:00,long,293,1659,57231,10,
2025-04-21T15:47:30+08:00,short,32,781,24085,10,
2025-04-21T15:47:06+08:00,long,300,1785,55860,10,
2025-04-21T15:47:49+08:00,long,297,1450,40171,10,
1 Timestamp PromptType PromptTokens ResponseTokens ResponseTime ConcurrencyStep Error
2 2025-04-21T15:29:08+08:00 long 290 1767 50692 1
3 2025-04-21T15:29:59+08:00 short 32 1240 35591 1
4 2025-04-21T15:30:35+08:00 long 299 1580 46800 1
5 2025-04-21T15:31:22+08:00 long 285 1736 50826 1
6 2025-04-21T15:32:13+08:00 long 294 1185 35257 1
7 2025-04-21T15:32:48+08:00 long 278 1118 33428 1
8 2025-04-21T15:33:22+08:00 short 30 1075 31594 1
9 2025-04-21T15:33:54+08:00 short 35 1693 48817 1
10 2025-04-21T15:34:43+08:00 short 30 1449 40108 1
11 2025-04-21T15:35:23+08:00 short 31 1267 39528 5
12 2025-04-21T15:35:23+08:00 short 34 1331 43050 5
13 2025-04-21T15:35:23+08:00 long 281 1455 47254 5
14 2025-04-21T15:35:23+08:00 short 34 1741 54033 5
15 2025-04-21T15:35:23+08:00 short 0 0 60000 5 发送请求失败: timeout
16 2025-04-21T15:36:07+08:00 short 34 1177 37663 5
17 2025-04-21T15:36:03+08:00 long 298 1479 47924 5
18 2025-04-21T15:36:18+08:00 short 36 1189 39376 5
19 2025-04-21T15:36:11+08:00 long 300 1778 56227 5
20 2025-04-21T15:36:45+08:00 short 30 662 22887 5
21 2025-04-21T15:36:23+08:00 long 295 1390 44672 5
22 2025-04-21T15:36:51+08:00 long 306 1366 44047 5
23 2025-04-21T15:37:08+08:00 long 298 1342 43062 5
24 2025-04-21T15:36:57+08:00 long 291 1743 56131 5
25 2025-04-21T15:37:08+08:00 short 31 1675 52352 5
26 2025-04-21T15:37:07+08:00 long 294 1877 58539 5
27 2025-04-21T15:37:52+08:00 short 31 625 19803 5
28 2025-04-21T15:38:06+08:00 short 33 206 7176 5
29 2025-04-21T15:37:35+08:00 long 288 1769 55162 5
30 2025-04-21T15:37:54+08:00 long 296 1494 47829 5
31 2025-04-21T15:38:01+08:00 long 295 1530 50380 5
32 2025-04-21T15:38:13+08:00 short 38 1463 45724 5
33 2025-04-21T15:38:31+08:00 short 34 906 28607 5
34 2025-04-21T15:38:42+08:00 short 33 691 21916 5
35 2025-04-21T15:38:12+08:00 long 0 0 60000 5 发送请求失败: timeout
36 2025-04-21T15:38:59+08:00 short 34 676 22665 5
37 2025-04-21T15:38:51+08:00 short 34 1006 30879 5
38 2025-04-21T15:38:59+08:00 short 28 1185 36817 5
39 2025-04-21T15:39:04+08:00 long 281 1688 53184 5
40 2025-04-21T15:39:22+08:00 short 34 1192 35771 5
41 2025-04-21T15:39:37+08:00 short 35 922 29691 5
42 2025-04-21T15:39:12+08:00 long 299 1814 55118 5
43 2025-04-21T15:39:22+08:00 short 32 1555 46992 5
44 2025-04-21T15:40:07+08:00 short 33 777 25157 5
45 2025-04-21T15:39:57+08:00 short 29 1375 41815 5
46 2025-04-21T15:40:09+08:00 long 292 1278 38409 5
47 2025-04-21T15:40:06+08:00 long 290 1586 45621 5
48 2025-04-21T15:39:59+08:00 long 301 1874 57468 5
49 2025-04-21T15:40:52+08:00 long 287 1565 45984 5
50 2025-04-21T15:41:39+08:00 short 34 859 25973 5
51 2025-04-21T15:42:05+08:00 short 33 1031 36665 10
52 2025-04-21T15:42:05+08:00 short 31 1032 38132 10
53 2025-04-21T15:42:05+08:00 short 33 1124 41458 10
54 2025-04-21T15:42:05+08:00 short 31 1209 42973 10
55 2025-04-21T15:42:05+08:00 short 36 1260 45000 10
56 2025-04-21T15:42:05+08:00 long 301 1254 45741 10
57 2025-04-21T15:42:05+08:00 short 0 0 60025 10 发送请求失败: timeout
58 2025-04-21T15:42:05+08:00 long 0 0 60024 10 发送请求失败: timeout
59 2025-04-21T15:42:05+08:00 long 0 0 60024 10 发送请求失败: timeout
60 2025-04-21T15:42:05+08:00 long 0 0 60027 10 发送请求失败: timeout
61 2025-04-21T15:42:43+08:00 short 30 627 24785 10
62 2025-04-21T15:42:41+08:00 short 33 880 33993 10
63 2025-04-21T15:42:46+08:00 short 33 763 29495 10
64 2025-04-21T15:42:48+08:00 short 32 867 34001 10
65 2025-04-21T15:42:51+08:00 short 32 878 33547 10
66 2025-04-21T15:43:08+08:00 short 34 959 36283 10
67 2025-04-21T15:43:05+08:00 short 32 1059 40263 10
68 2025-04-21T15:42:50+08:00 long 0 0 60001 10 发送请求失败: timeout
69 2025-04-21T15:43:05+08:00 long 302 1515 55759 10
70 2025-04-21T15:43:05+08:00 short 32 1480 55895 10
71 2025-04-21T15:43:05+08:00 long 0 0 60025 10 发送请求失败: timeout
72 2025-04-21T15:43:44+08:00 short 34 663 25933 10
73 2025-04-21T15:43:16+08:00 long 302 1503 55131 10
74 2025-04-21T15:43:24+08:00 short 34 1385 50030 10
75 2025-04-21T15:43:16+08:00 long 290 1646 59167 10
76 2025-04-21T15:43:22+08:00 short 31 1512 56955 10
77 2025-04-21T15:43:45+08:00 short 28 1248 47396 10
78 2025-04-21T15:44:05+08:00 short 34 945 35028 10
79 2025-04-21T15:44:01+08:00 short 35 1134 43283 10
80 2025-04-21T15:44:15+08:00 short 35 921 32485 10
81 2025-04-21T15:43:50+08:00 short 34 1628 59025 10
82 2025-04-21T15:44:11+08:00 short 31 1160 42541 10
83 2025-04-21T15:44:15+08:00 short 34 1036 41375 10
84 2025-04-21T15:44:01+08:00 long 286 1507 56831 10
85 2025-04-21T15:44:48+08:00 short 28 345 13944 10
86 2025-04-21T15:44:33+08:00 short 28 965 35746 10
87 2025-04-21T15:44:10+08:00 short 34 1640 59733 10
88 2025-04-21T15:44:19+08:00 short 31 1563 55769 10
89 2025-04-21T15:44:54+08:00 short 37 721 27026 10
90 2025-04-21T15:44:44+08:00 long 291 1268 45194 10
91 2025-04-21T15:44:58+08:00 short 30 800 31400 10
92 2025-04-21T15:44:40+08:00 long 287 1422 54009 10
93 2025-04-21T15:44:49+08:00 short 32 1479 52350 10
94 2025-04-21T15:45:15+08:00 short 31 728 27863 10
95 2025-04-21T15:45:09+08:00 short 35 1127 42373 10
96 2025-04-21T15:44:56+08:00 short 0 0 60001 10 发送请求失败: timeout
97 2025-04-21T15:45:02+08:00 short 34 1491 55425 10
98 2025-04-21T15:45:10+08:00 short 0 0 60001 10 发送请求失败: timeout
99 2025-04-21T15:45:43+08:00 short 31 814 32256 10
100 2025-04-21T15:45:21+08:00 long 0 0 60001 10 发送请求失败: timeout
101 2025-04-21T15:45:51+08:00 short 33 808 31357 10
102 2025-04-21T15:45:30+08:00 long 302 1447 54647 10
103 2025-04-21T15:45:56+08:00 short 36 761 29637 10
104 2025-04-21T15:45:42+08:00 short 34 1214 45591 10
105 2025-04-21T15:45:35+08:00 short 29 1465 54649 10
106 2025-04-21T15:45:29+08:00 long 0 0 60000 10 发送请求失败: timeout
107 2025-04-21T15:45:58+08:00 short 35 874 34326 10
108 2025-04-21T15:46:10+08:00 short 27 1111 41640 10
109 2025-04-21T15:46:21+08:00 short 32 816 32250 10
110 2025-04-21T15:46:29+08:00 short 32 709 25910 10
111 2025-04-21T15:46:25+08:00 short 37 863 31820 10
112 2025-04-21T15:46:32+08:00 short 31 652 24537 10
113 2025-04-21T15:46:28+08:00 short 30 1034 37449 10
114 2025-04-21T15:46:15+08:00 long 0 0 60001 10 发送请求失败: timeout
115 2025-04-21T15:46:54+08:00 short 28 607 23222 10
116 2025-04-21T15:46:26+08:00 long 294 1390 51678 10
117 2025-04-21T15:46:56+08:00 short 28 705 26915 10
118 2025-04-21T15:46:23+08:00 long 0 0 60001 10 发送请求失败: timeout
119 2025-04-21T15:46:57+08:00 short 33 818 30689 10
120 2025-04-21T15:46:30+08:00 short 0 0 60030 10 发送请求失败: timeout
121 2025-04-21T15:46:57+08:00 short 37 1139 40595 10
122 2025-04-21T15:46:52+08:00 long 293 1659 57231 10
123 2025-04-21T15:47:30+08:00 short 32 781 24085 10
124 2025-04-21T15:47:06+08:00 long 300 1785 55860 10
125 2025-04-21T15:47:49+08:00 long 297 1450 40171 10
@@ -1,104 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>LLM API 基准测试报告</title>
<script src="https://go-echarts.github.io/go-echarts-assets/assets/echarts.min.js"></script>
</head>
<body>
<style> .container {display: flex;justify-content: center;align-items: center;} .item {margin: auto;} </style>
<div class="container">
<div class="item" id="EHMZswDJUWqH" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_EHMZswDJUWqH = echarts.init(document.getElementById('EHMZswDJUWqH'), "white");
let option_EHMZswDJUWqH = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"响应时间","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":0},{"value":0},{"value":0},{"value":0}]}],"title":{"text":"性能指标概述"},"tooltip":{"show":false},"xAxis":[{"data":["平均响应时间 (ms)","P90响应时间 (ms)","P95响应时间 (ms)","P99响应时间 (ms)"]}],"yAxis":[{}]}
;
let action_EHMZswDJUWqH = {"areas":{},"type":""}
;
goecharts_EHMZswDJUWqH.setOption(option_EHMZswDJUWqH);
goecharts_EHMZswDJUWqH.dispatchAction(action_EHMZswDJUWqH);
</script>
<div class="container">
<div class="item" id="FLqyKjIZgeEc" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_FLqyKjIZgeEc = echarts.init(document.getElementById('FLqyKjIZgeEc'), "white");
let option_FLqyKjIZgeEc = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"请求数","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0}]}],"title":{"text":"响应时间分布"},"tooltip":{"show":false},"xAxis":[{"data":["≤100ms","100-200ms","200-500ms","500-1000ms","1000-2000ms","2000-5000ms","5000-10000ms",">10000ms"]}],"yAxis":[{}]}
;
let action_FLqyKjIZgeEc = {"areas":{},"type":""}
;
goecharts_FLqyKjIZgeEc.setOption(option_FLqyKjIZgeEc);
goecharts_FLqyKjIZgeEc.dispatchAction(action_FLqyKjIZgeEc);
</script>
<div class="container">
<div class="item" id="RXbTCdDVHsPX" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_RXbTCdDVHsPX = echarts.init(document.getElementById('RXbTCdDVHsPX'), "white");
let option_RXbTCdDVHsPX = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":null,"title":{"text":"QPS随时间变化"},"tooltip":{"show":false},"xAxis":[{}],"yAxis":[{}]}
;
let action_RXbTCdDVHsPX = {"areas":{},"type":""}
;
goecharts_RXbTCdDVHsPX.setOption(option_RXbTCdDVHsPX);
goecharts_RXbTCdDVHsPX.dispatchAction(action_RXbTCdDVHsPX);
</script>
<div class="container">
<div class="item" id="XNYfcbGGZVDe" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_XNYfcbGGZVDe = echarts.init(document.getElementById('XNYfcbGGZVDe'), "white");
let option_XNYfcbGGZVDe = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":null,"title":{"text":"Token生成速率随时间变化"},"tooltip":{"show":false},"xAxis":[{}],"yAxis":[{}]}
;
let action_XNYfcbGGZVDe = {"areas":{},"type":""}
;
goecharts_XNYfcbGGZVDe.setOption(option_XNYfcbGGZVDe);
goecharts_XNYfcbGGZVDe.dispatchAction(action_XNYfcbGGZVDe);
</script>
<div class="container">
<div class="item" id="odZNEMJQVnFD" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_odZNEMJQVnFD = echarts.init(document.getElementById('odZNEMJQVnFD'), "white");
let option_odZNEMJQVnFD = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"平均响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null},{"name":"P90响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null},{"name":"P95响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null}],"title":{"text":"并发与响应时间关系"},"tooltip":{"show":false},"xAxis":[{"data":[]}],"yAxis":[{}]}
;
let action_odZNEMJQVnFD = {"areas":{},"type":""}
;
goecharts_odZNEMJQVnFD.setOption(option_odZNEMJQVnFD);
goecharts_odZNEMJQVnFD.dispatchAction(action_odZNEMJQVnFD);
</script>
</body>
</html>
@@ -1 +0,0 @@
Timestamp,PromptType,PromptTokens,CompletionTokens,ResponseTime,Concurrency,Error
1 Timestamp PromptType PromptTokens CompletionTokens ResponseTime Concurrency Error
@@ -1,104 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>LLM API 基准测试报告</title>
<script src="https://go-echarts.github.io/go-echarts-assets/assets/echarts.min.js"></script>
</head>
<body>
<style> .container {display: flex;justify-content: center;align-items: center;} .item {margin: auto;} </style>
<div class="container">
<div class="item" id="VtXivSkVGWsI" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_VtXivSkVGWsI = echarts.init(document.getElementById('VtXivSkVGWsI'), "white");
let option_VtXivSkVGWsI = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"响应时间","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":0},{"value":0},{"value":0},{"value":0}]}],"title":{"text":"性能指标概述"},"tooltip":{"show":false},"xAxis":[{"data":["平均响应时间 (ms)","P90响应时间 (ms)","P95响应时间 (ms)","P99响应时间 (ms)"]}],"yAxis":[{}]}
;
let action_VtXivSkVGWsI = {"areas":{},"type":""}
;
goecharts_VtXivSkVGWsI.setOption(option_VtXivSkVGWsI);
goecharts_VtXivSkVGWsI.dispatchAction(action_VtXivSkVGWsI);
</script>
<div class="container">
<div class="item" id="vwmlpHKQyHVV" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_vwmlpHKQyHVV = echarts.init(document.getElementById('vwmlpHKQyHVV'), "white");
let option_vwmlpHKQyHVV = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"请求数","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0}]}],"title":{"text":"响应时间分布"},"tooltip":{"show":false},"xAxis":[{"data":["≤100ms","100-200ms","200-500ms","500-1000ms","1000-2000ms","2000-5000ms","5000-10000ms",">10000ms"]}],"yAxis":[{}]}
;
let action_vwmlpHKQyHVV = {"areas":{},"type":""}
;
goecharts_vwmlpHKQyHVV.setOption(option_vwmlpHKQyHVV);
goecharts_vwmlpHKQyHVV.dispatchAction(action_vwmlpHKQyHVV);
</script>
<div class="container">
<div class="item" id="ONXViPrQSJxX" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_ONXViPrQSJxX = echarts.init(document.getElementById('ONXViPrQSJxX'), "white");
let option_ONXViPrQSJxX = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":null,"title":{"text":"QPS随时间变化"},"tooltip":{"show":false},"xAxis":[{}],"yAxis":[{}]}
;
let action_ONXViPrQSJxX = {"areas":{},"type":""}
;
goecharts_ONXViPrQSJxX.setOption(option_ONXViPrQSJxX);
goecharts_ONXViPrQSJxX.dispatchAction(action_ONXViPrQSJxX);
</script>
<div class="container">
<div class="item" id="KFGiaxLKrHzO" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_KFGiaxLKrHzO = echarts.init(document.getElementById('KFGiaxLKrHzO'), "white");
let option_KFGiaxLKrHzO = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":null,"title":{"text":"Token生成速率随时间变化"},"tooltip":{"show":false},"xAxis":[{}],"yAxis":[{}]}
;
let action_KFGiaxLKrHzO = {"areas":{},"type":""}
;
goecharts_KFGiaxLKrHzO.setOption(option_KFGiaxLKrHzO);
goecharts_KFGiaxLKrHzO.dispatchAction(action_KFGiaxLKrHzO);
</script>
<div class="container">
<div class="item" id="eUNIZaAWZYZJ" style="width:900px;height:500px;"></div>
</div>
<script type="text/javascript">
"use strict";
let goecharts_eUNIZaAWZYZJ = echarts.init(document.getElementById('eUNIZaAWZYZJ'), "white");
let option_eUNIZaAWZYZJ = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"平均响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null},{"name":"P90响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null},{"name":"P95响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null}],"title":{"text":"并发与响应时间关系"},"tooltip":{"show":false},"xAxis":[{"data":[]}],"yAxis":[{}]}
;
let action_eUNIZaAWZYZJ = {"areas":{},"type":""}
;
goecharts_eUNIZaAWZYZJ.setOption(option_eUNIZaAWZYZJ);
goecharts_eUNIZaAWZYZJ.dispatchAction(action_eUNIZaAWZYZJ);
</script>
</body>
</html>
@@ -1 +0,0 @@
Timestamp,PromptType,PromptTokens,CompletionTokens,ResponseTime,Concurrency,Error
1 Timestamp PromptType PromptTokens CompletionTokens ResponseTime Concurrency Error
+170
View File
@@ -0,0 +1,170 @@
package test
import (
"net/http"
"net/http/httptest"
"testing"
"llm-api-benchmark-tool/pkg/client"
"github.com/stretchr/testify/assert"
"io"
"fmt"
"time"
"sync"
)
// TestHTTPClientDo tests the non-streaming Do method of the HTTPClient interface.
func TestHTTPClientDo(t *testing.T) {
// 1. Mock server setup
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Basic validation of the incoming request
assert.Equal(t, "POST", r.Method)
assert.Equal(t, "/test-endpoint", r.URL.Path)
assert.Equal(t, "application/json", r.Header.Get("Content-Type"))
assert.Equal(t, "Bearer test-key", r.Header.Get("Authorization"))
if r.ContentLength <= 0 {
t.Errorf("Server received zero or negative Content-Length: %d", r.ContentLength)
w.WriteHeader(http.StatusBadRequest)
_, _ = w.Write([]byte(`{"error":"missing or invalid content length"}`))
return
}
bodyBytes := make([]byte, r.ContentLength)
_, err := io.ReadFull(r.Body, bodyBytes)
assert.NoError(t, err, "Server failed to read expected number of body bytes")
assert.Equal(t, `{"message":"ping"}`, string(bodyBytes))
// Send mock response
w.Header().Set("Content-Type", "application/json")
w.Header().Set("X-Test-Header", "test-value")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(`{"reply":"pong"}`))
}))
defer server.Close()
// 2. Instantiate client implementation
httpClient := client.NewFastHTTPClient() // Using fasthttp client
// 3. Create mock Request
req := &client.Request{
Method: "POST",
URL: server.URL + "/test-endpoint",
Body: []byte(`{"message":"ping"}`),
Headers: map[string]string{
"Content-Type": "application/json",
"Authorization": "Bearer test-key",
},
}
// 4. Call Do method
resp, err := httpClient.Do(req)
// 5. Assert Response correctness
assert.NoError(t, err)
assert.NotNil(t, resp)
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, `{"reply":"pong"}`, string(resp.Body))
assert.Equal(t, "application/json", resp.Headers["Content-Type"])
assert.Equal(t, "test-value", resp.Headers["X-Test-Header"])
// TODO: Add test cases for error handling (e.g., server error, network error)
}
// TestHTTPClientStream tests the streaming Stream method of the HTTPClient interface.
// It should verify SSE parsing, callback invocation, TTFT measurement, and handling of [DONE] message.
func TestHTTPClientStream(t *testing.T) {
// 1. Mock SSE server setup
mockSSEData := []string{
`data: {"id":"1","choices":[{"delta":{"role":"assistant"}}]}`, // Initial metadata
`data: {"id":"1","choices":[{"delta":{"content":"Hello"}}]}`, // First content
`data: {"id":"1","choices":[{"delta":{"reasoning_content":" Thinking..."}}]}`, // Reasoning content
`data: {"id":"1","choices":[{"delta":{"content":" world"}}]}`, // More content
`data: [DONE]`,
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, "POST", r.Method)
assert.Equal(t, "/test-stream", r.URL.Path)
assert.Equal(t, "Bearer test-key-stream", r.Header.Get("Authorization"))
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
flusher, ok := w.(http.Flusher)
if !ok {
t.Fatal("Server does not support flushing")
}
for _, line := range mockSSEData {
fmt.Fprintf(w, "%s\n\n", line)
flusher.Flush() // Ensure data is sent immediately
time.Sleep(10 * time.Millisecond) // Small delay between chunks
}
}))
defer server.Close()
// 2. Instantiate client implementation
httpClient := client.NewFastHTTPClient()
// 3. Create mock Request
req := &client.Request{
Method: "POST",
URL: server.URL + "/test-stream",
Body: []byte(`{"prompt":"test prompt","stream":true}`),
Headers: map[string]string{
"Content-Type": "application/json",
"Authorization": "Bearer test-key-stream",
},
}
// 4. Define callback function
var receivedChunks []client.SSEChunk
var firstChunkTime int64
var mu sync.Mutex // Protect access to shared variables
callback := func(chunk client.SSEChunk) error {
mu.Lock()
defer mu.Unlock()
if firstChunkTime == 0 && len(chunk.Data) > 0 && !chunk.IsDone {
firstChunkTime = chunk.Timestamp // Record time of first data chunk
}
receivedChunks = append(receivedChunks, chunk)
// fmt.Printf("Callback received: IsDone=%v, Data=%s\n", chunk.IsDone, string(chunk.Data))
return nil // No error from callback in this test
}
// 5. Call Stream method (EXPECTED TO FAIL INITIALLY)
startTime := time.Now().UnixNano()
err := httpClient.Stream(req, callback)
// 6. Assert results (WILL FAIL INITIALLY)
// assert.NoError(t, err) // Initially, this will fail
if err != nil {
assert.ErrorContains(t, err, "FastHTTPClient.Stream not implemented yet")
// Since it's not implemented, we don't assert chunk content or TTFT yet
} else {
// --- Assertions for when Stream IS implemented ---
mu.Lock()
defer mu.Unlock()
assert.NotEmpty(t, receivedChunks, "Callback should have been called")
assert.Len(t, receivedChunks, len(mockSSEData), "Should receive the same number of chunks as sent")
// Verify last chunk is [DONE]
lastChunk := receivedChunks[len(receivedChunks)-1]
assert.True(t, lastChunk.IsDone, "Last chunk should be marked as IsDone")
assert.Equal(t, "[DONE]", string(lastChunk.Data), "Last chunk data should be [DONE]")
// Verify TTFT calculation (placeholder)
assert.Greater(t, firstChunkTime, startTime, "First chunk time should be after start time")
ttftMs := float64(firstChunkTime-startTime) / 1e6
t.Logf("Measured TTFT: %.2f ms", ttftMs)
assert.Less(t, ttftMs, 100.0, "TTFT should be reasonably small in mock test") // Adjust threshold as needed
// TODO: Add more detailed assertions on specific chunk content if needed
// TODO: Add assertions for handling 'content' vs 'reasoning_content'
// TODO: Add test cases for error handling (server error, invalid SSE, callback error)
}
}
// TODO: Add helper functions for mock server/client setup if needed
+124
View File
@@ -0,0 +1,124 @@
package test
import (
"context"
"fmt" // Added for createTestRequest
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"llm-api-benchmark-tool/pkg/client" // Needs client types
"llm-api-benchmark-tool/pkg/concurrency" // Target package (to be created)
"llm-api-benchmark-tool/pkg/stats" // Added for stats collector
)
// --- Mock HTTP Client ---
type MockHTTPClient struct {
DoFunc func(req *client.Request) (*client.Response, error)
StreamFunc func(req *client.Request, callback func(chunk client.SSEChunk) error) error
ReqCount atomic.Int64 // Track calls
}
func (m *MockHTTPClient) Do(req *client.Request) (*client.Response, error) {
m.ReqCount.Add(1)
if m.DoFunc != nil {
return m.DoFunc(req)
}
// Default mock response for Do
return &client.Response{
StatusCode: 200,
Body: []byte(`{"mock": "response"}`),
}, nil
}
func (m *MockHTTPClient) Stream(req *client.Request, callback func(chunk client.SSEChunk) error) error {
m.ReqCount.Add(1)
if m.StreamFunc != nil {
return m.StreamFunc(req, callback)
}
// Default mock stream behavior
go func() { // Simulate receiving chunks
time.Sleep(5 * time.Millisecond) // Simulate TTFT
// Ensure callback is not nil before calling
if callback != nil {
callback(client.SSEChunk{Data: []byte(`{"mock": "chunk1"}`), Timestamp: time.Now().UnixNano()})
time.Sleep(10 * time.Millisecond)
callback(client.SSEChunk{Data: []byte(`[DONE]`), Timestamp: time.Now().UnixNano(), IsDone: true})
}
}()
return nil
}
// --- Test Concurrency Manager ---
func TestConcurrencyManager_RunBenchmark(t *testing.T) {
mockClient := &MockHTTPClient{}
params := concurrency.BenchmarkParams{
HTTPClient: mockClient,
TargetURL: "http://mock.test/api",
TotalDuration: 100 * time.Millisecond, // Short duration for testing
Concurrency: 10,
RequestInterval: 5 * time.Millisecond, // Interval between starting new requests
IsStreaming: false, // Test non-streaming first
PromptTokens: 50, // Example prompt size
// Add other necessary params like API Key, Model later
}
// Create the manager (package and struct to be defined)
statsCollector := stats.NewStatsCollector() // Create mock stats collector
manager := concurrency.NewManager(params, statsCollector) // Assuming a constructor
// Run the benchmark
// TODO: Define the result struct if RunBenchmark returns results
err := manager.RunBenchmark(context.Background()) // Use background context for now
require.NoError(t, err, "RunBenchmark should complete without error")
// Assertions
// 1. Check if roughly the correct number of requests were made.
// Expected rate = Concurrency / RequestInterval (adjust if interval is per worker)
// Expected requests ~ TotalDuration / RequestInterval (simplified, needs refinement)
// Let's just check if *some* requests were made for now.
madeRequests := mockClient.ReqCount.Load()
assert.Greater(t, madeRequests, int64(0), "Should have made at least one request")
t.Logf("Total requests made by mock client: %d", madeRequests)
// 2. Test with Streaming
mockClientStreaming := &MockHTTPClient{}
paramsStreaming := params
paramsStreaming.HTTPClient = mockClientStreaming
paramsStreaming.IsStreaming = true
paramsStreaming.TotalDuration = 150 * time.Millisecond // Slightly longer for streaming
statsCollectorStreaming := stats.NewStatsCollector() // Create mock stats collector
managerStreaming := concurrency.NewManager(paramsStreaming, statsCollectorStreaming)
errStreaming := managerStreaming.RunBenchmark(context.Background())
require.NoError(t, errStreaming, "RunBenchmark (streaming) should complete without error")
madeRequestsStreaming := mockClientStreaming.ReqCount.Load()
assert.Greater(t, madeRequestsStreaming, int64(0), "Should have made at least one streaming request")
t.Logf("Total streaming requests made by mock client: %d", madeRequestsStreaming)
// TODO: Add test case for context cancellation/timeout
// TODO: Add test case for handling client errors
// TODO: Add more precise assertions on request count and timing if possible
}
// Helper function to create a basic request (might move to a shared test util)
func createTestRequest(url string, isStreaming bool, promptTokens int) *client.Request {
// In a real scenario, we'd use the tokenizer here
mockPrompt := fmt.Sprintf("mock prompt %d tokens", promptTokens)
body := []byte(fmt.Sprintf(`{"prompt": "%s", "stream": %v}`, mockPrompt, isStreaming))
return &client.Request{
URL: url,
Method: "POST",
Headers: map[string]string{"Content-Type": "application/json"},
Body: body,
}
}
+210
View File
@@ -0,0 +1,210 @@
package test
import (
"context"
"fmt"
"html"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"llm-api-benchmark-tool/pkg/client"
"llm-api-benchmark-tool/pkg/concurrency"
"llm-api-benchmark-tool/pkg/config"
"llm-api-benchmark-tool/pkg/report"
"llm-api-benchmark-tool/pkg/stats"
)
// Helper to create durations
func dms(ms int) time.Duration {
return time.Duration(ms) * time.Millisecond
}
// TestBenchmarkIntegration performs an end-to-end test of the benchmark tool.
func TestBenchmarkIntegration(t *testing.T) {
// --- 1. Set up Mock HTTP Server ---
mux := http.NewServeMux()
mux.HandleFunc("/v1/chat/completions", func(w http.ResponseWriter, r *http.Request) {
// Simulate some processing time
time.Sleep(dms(50 + int(time.Now().UnixMilli()%50))) // 50-100ms latency
// Check if it's a streaming request based on header or request body field
// For simplicity, let's assume non-streaming for now
isStreaming := false // TODO: Detect streaming requests properly
if isStreaming {
// Simulate SSE streaming
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
flusher, ok := w.(http.Flusher)
if !ok {
http.Error(w, "Streaming unsupported!", http.StatusInternalServerError)
return
}
// Send TTFT event quickly
time.Sleep(dms(10 + int(time.Now().UnixMilli()%10))) // 10-20ms TTFT
fmt.Fprintf(w, "id: msg1\nevent: content_block_delta\ndata: {\"type\": \"content_block_delta\", \"index\": 0, \"delta\": {\"type\": \"text_delta\", \"text\": \"Hello\"}}\n\n")
flusher.Flush()
// Send subsequent events
time.Sleep(dms(30))
fmt.Fprintf(w, "id: msg2\nevent: content_block_delta\ndata: {\"type\": \"content_block_delta\", \"index\": 0, \"delta\": {\"type\": \"text_delta\", \"text\": \" world!\"}}\n\n")
flusher.Flush()
// Send stop event
time.Sleep(dms(10))
fmt.Fprintf(w, "id: msg3\nevent: message_stop\ndata: {\"type\": \"message_stop\"}\n\n")
flusher.Flush()
} else {
// Simulate non-streaming response
w.Header().Set("Content-Type", "application/json")
// A simple, valid-looking JSON response
responseBody := `{
"id": "chatcmpl-mock123",
"object": "chat.completion",
"created": 1677652288,
"model": "gpt-mock-3.5",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "This is a mock response."
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 9,
"completion_tokens": 12,
"total_tokens": 21
}
}`
fmt.Fprintln(w, responseBody)
}
})
server := httptest.NewServer(mux)
defer server.Close()
// --- 2. Set up Configuration ---
testDuration := 100 * time.Millisecond // Short duration for testing
cfg := config.Config{
APIEndpoint: server.URL + "/v1/chat/completions", // Use mock server URL
Model: "gpt-mock-3.5",
Client: "fasthttp", // Use fasthttp for testing
Timeout: 5 * time.Second,
Concurrency: 2, // Low concurrency for test
Duration: testDuration,
Requests: 10, // Limit requests for test (Note: Duration takes precedence)
RateLimit: 0, // No rate limit for test
Streaming: false, // Test non-streaming first
PromptTokens: 10,
MaxTokens: 20,
OutputReport: filepath.Join(t.TempDir(), "integration_report.html"),
Headers: make(map[string]string),
Payload: make(map[string]interface{}),
}
// --- 3. Initialize Components ---
httpClient := client.NewFastHTTPClient()
require.NotNil(t, httpClient, "Failed to create HTTP client")
statsCollector := stats.NewStatsCollector()
params := concurrency.BenchmarkParams{
TargetURL: cfg.APIEndpoint, // Use correct field name
HTTPClient: httpClient,
TotalDuration: cfg.Duration,
Concurrency: cfg.Concurrency,
RequestInterval: 0, // No artificial delay between requests
IsStreaming: cfg.Streaming,
PromptTokens: cfg.PromptTokens,
}
manager := concurrency.NewManager(params, statsCollector)
require.NotNil(t, manager, "Manager should not be nil")
// --- 4. Run Benchmark ---
t.Logf("Running benchmark against mock server: %s", cfg.APIEndpoint)
startTime := time.Now()
err := manager.RunBenchmark(context.Background()) // Pass context
actualDuration := time.Since(startTime)
require.NoError(t, err, "Benchmark run failed")
// --- 5. Get Final Stats ---
finalStats := statsCollector.CalculateStats(actualDuration)
t.Logf("Benchmark finished. Final stats: %+v", finalStats)
// Explicit check before assertion
if finalStats.TotalRequests > 0 {
t.Logf("Check PASSED: finalStats.TotalRequests (%d) > 0", finalStats.TotalRequests)
} else {
t.Errorf("Check FAILED: finalStats.TotalRequests (%d) is not > 0", finalStats.TotalRequests)
}
// Replace assert.Greater with standard check due to potential testify issue
if !(finalStats.TotalRequests > 0) { // Explicitly check the condition
t.Errorf("Test FAILED: Expected TotalRequests (%d) to be greater than 0", finalStats.TotalRequests)
}
// Replace assert.LessOrEqual with standard check due to potential testify issue
if !(finalStats.FailedRequests <= 1) { // Explicitly check the condition
t.Errorf("Test FAILED: Expected FailedRequests (%d) to be less than or equal to 1", finalStats.FailedRequests)
}
// --- 6. Generate Report ---
err = report.GenerateHTMLReport(finalStats, cfg.OutputReport) // Pass report path string
require.NoError(t, err, "Failed to generate HTML report")
t.Logf("Report generated at: %s", cfg.OutputReport)
// --- 7. Verify Report Content ---
reportBytes, err := os.ReadFile(cfg.OutputReport)
require.NoError(t, err, "Failed to read generated report file")
reportContent := string(reportBytes)
// Check for basic summary stats in the report (adjust format as needed)
assert.Contains(t, reportContent, "Total Requests:", "Report missing actual summary data (e.g., 'Total Requests:')")
assert.Contains(t, reportContent, fmt.Sprintf("Total Requests:</td><td>%d", finalStats.TotalRequests), "Report missing or incorrect total requests")
assert.Contains(t, reportContent, fmt.Sprintf("Successful Requests:</td><td>%d", finalStats.SuccessfulRequests), "Report missing or incorrect successful requests")
assert.Contains(t, reportContent, "Avg Latency:", "Report missing avg latency")
// Check for chart divs
assert.Contains(t, reportContent, "<div id=\"latencyHistogram\"", "Report missing latency histogram div")
assert.Contains(t, reportContent, "<div id=\"ttftHistogram\"", "Report missing TTFT histogram div")
// Check for chart initialization JS (presence, not exact content)
assert.Contains(t, reportContent, "echarts.init(document.getElementById('latencyHistogram')", "Report missing latency chart JS init")
assert.Contains(t, reportContent, "echarts.init(document.getElementById('ttftHistogram')", "Report missing TTFT chart JS init")
// Check that histogram data is not just the default empty/"No Data" case
// We need to find the data part of the JS options
latencyOptionStart := strings.Index(reportContent, "let option_latencyHistogram = {")
require.True(t, latencyOptionStart > 0, "Latency chart options not found in JS")
latencyOptionEnd := strings.Index(reportContent[latencyOptionStart:], "};")
require.True(t, latencyOptionEnd > 0, "Latency chart options end not found")
latencyOptionJS := reportContent[latencyOptionStart : latencyOptionStart+latencyOptionEnd]
// Check that the default "No Data" category isn't present if we have data
if len(finalStats.LatencyData) > 0 {
assert.NotContains(t, latencyOptionJS, `"name":"Latency","type":"bar","data":[{"value":0}]`, "Latency chart shows default empty data when actual data exists")
assert.NotContains(t, latencyOptionJS, `xaxis":[{"data":["No Data"]}]`, "Latency chart shows 'No Data' category when actual data exists") // Check category
}
// Check HTML escaping for summary values (find an example value)
avgLatencyStr := html.EscapeString(finalStats.AvgLatency.String())
assert.Contains(t, reportContent, avgLatencyStr, "AvgLatency string not found or not escaped in HTML")
t.Log("Integration test completed successfully.")
}
// TODO: Add another integration test case for streaming requests
+86
View File
@@ -0,0 +1,86 @@
package test
import (
"fmt"
"os"
"strings"
"testing"
"time"
"llm-api-benchmark-tool/pkg/report"
"llm-api-benchmark-tool/pkg/stats"
"github.com/stretchr/testify/assert"
)
// createSampleStats creates a sample FinalStats struct for testing.
func createSampleStats() stats.FinalStats {
return stats.FinalStats{
TotalRequests: 1000,
SuccessfulRequests: 950,
FailedRequests: 50,
TotalDuration: 30 * time.Second,
AvgLatency: 150 * time.Millisecond,
MinLatency: 50 * time.Millisecond,
MaxLatency: 500 * time.Millisecond,
P90Latency: 250 * time.Millisecond,
P95Latency: 300 * time.Millisecond,
P99Latency: 400 * time.Millisecond,
AvgTimeToFirstToken: 80 * time.Millisecond,
MinTimeToFirstToken: 30 * time.Millisecond,
MaxTimeToFirstToken: 200 * time.Millisecond,
P90TimeToFirstToken: 120 * time.Millisecond,
P95TimeToFirstToken: 150 * time.Millisecond,
P99TimeToFirstToken: 180 * time.Millisecond,
AvgQPS: 33.33,
AvgTokensPerSecond: 500.5,
// Add more fields if necessary for report testing
}
}
// TestGenerateHTMLReport tests the basic HTML report generation.
func TestGenerateHTMLReport(t *testing.T) {
sampleStats := createSampleStats()
outputDir := t.TempDir() // Create a temporary directory for the report
outputFilePath := outputDir + "/report.html"
// Define expected chart IDs (must match IDs set in generator.go)
latencyChartID := "latencyHistogram"
ttftChartID := "ttftHistogram"
// Generate the report
err := report.GenerateHTMLReport(sampleStats, outputFilePath)
assert.NoError(t, err, "GenerateHTMLReport should not return an error")
// Check if the file was created
_, err = os.Stat(outputFilePath)
assert.NoError(t, err, "HTML report file should be created")
// Read the generated file content
contentBytes, err := os.ReadFile(outputFilePath)
assert.NoError(t, err, "Should be able to read the generated report file")
content := string(contentBytes)
// Basic checks on the HTML content
assert.True(t, strings.HasPrefix(content, "<!DOCTYPE html>"), "Output should start with DOCTYPE html")
assert.Contains(t, content, "<title>LLM API Benchmark Report</title>", "HTML should contain the correct title")
assert.Contains(t, content, "<h1>LLM API Benchmark Report</h1>", "HTML should contain the main heading")
assert.Contains(t, content, "<h2>Summary</h2>", "HTML should contain Summary section")
assert.Contains(t, content, "<h2>Charts</h2>", "HTML should contain Charts section")
// Check if some key stats are present (as simple text for now)
assert.Contains(t, content, "Total Requests:</td><td>1000</td>", "HTML should contain Total Requests stat")
assert.Contains(t, content, "Avg Latency:</td><td>150ms</td>", "HTML should contain Avg Latency stat")
assert.Contains(t, content, "Avg TTFT:</td><td>80ms</td>", "HTML should contain Avg TTFT stat")
assert.Contains(t, content, "Avg QPS:</td><td>33.33</td>", "HTML should contain Avg QPS stat")
// Check if the ECharts initialization script snippet is present for each chart
// latencyInitScript := fmt.Sprintf("echarts.init(document.getElementById('%s'))", latencyChartID)
// assert.Contains(t, content, latencyInitScript, "HTML should contain JS init for latency chart")
// Check if the chart divs are present (the ones generated by go-echarts render)
assert.Contains(t, content, fmt.Sprintf(`<div id="%s"`, latencyChartID), "HTML should contain div for latency chart")
assert.Contains(t, content, fmt.Sprintf(`<div id="%s"`, ttftChartID), "HTML should contain div for TTFT chart")
// TODO: Add more sophisticated checks? (e.g., parse HTML, check chart script data)
}
+127
View File
@@ -0,0 +1,127 @@
package test
import (
"sync"
"testing"
"time"
"llm-api-benchmark-tool/pkg/stats"
"github.com/stretchr/testify/assert"
// "gonum.org/v1/gonum/stat" // Might need later for percentile verification
)
// TestStatsCollector_BasicRecording tests basic recording and aggregation.
func TestStatsCollector_BasicRecording(t *testing.T) {
collector := stats.NewStatsCollector()
// Record some sample results
collector.RecordResult(stats.RequestResult{
IsSuccess: true,
Latency: 100 * time.Millisecond,
TimeToFirstToken: 50 * time.Millisecond, // Example TTFT
TotalTokens: 150, // Example token count
StreamLatency: nil, // Not a streaming result for this example
})
collector.RecordResult(stats.RequestResult{
IsSuccess: true,
Latency: 120 * time.Millisecond,
TimeToFirstToken: 60 * time.Millisecond,
TotalTokens: 160,
StreamLatency: nil,
})
collector.RecordResult(stats.RequestResult{
IsSuccess: false, // Failed request
Latency: 50 * time.Millisecond, // Failed quickly
TimeToFirstToken: 0, // No TTFT for failed requests
TotalTokens: 0, // No tokens for failed requests
StreamLatency: nil,
})
// Calculate final stats
finalStats := collector.CalculateStats(2 * time.Second) // Example total duration
assert.Equal(t, int64(3), finalStats.TotalRequests, "Total requests should be 3")
assert.Equal(t, int64(2), finalStats.SuccessfulRequests, "Successful requests should be 2")
assert.Equal(t, int64(1), finalStats.FailedRequests, "Failed requests should be 1")
// Basic average latency check (only successful)
// Expected avg = (100 + 120) / 2 = 110 ms
// Compare float64 seconds
assert.InDelta(t, 110*time.Millisecond.Seconds(), finalStats.AvgLatency.Seconds(), 0.01, "Average latency mismatch")
// Basic average TTFT check (only successful)
// Expected avg = (50 + 60) / 2 = 55 ms
// Compare float64 seconds
assert.InDelta(t, 55*time.Millisecond.Seconds(), finalStats.AvgTimeToFirstToken.Seconds(), 0.01, "Average TTFT mismatch")
// Check Min/Max Latency (successful only)
assert.Equal(t, 100*time.Millisecond, finalStats.MinLatency, "Min latency mismatch")
assert.Equal(t, 120*time.Millisecond, finalStats.MaxLatency, "Max latency mismatch")
// Check Min/Max TTFT (successful only)
assert.Equal(t, 50*time.Millisecond, finalStats.MinTimeToFirstToken, "Min TTFT mismatch")
assert.Equal(t, 60*time.Millisecond, finalStats.MaxTimeToFirstToken, "Max TTFT mismatch")
// Basic QPS check
// Expected QPS = 3 requests / 2 seconds = 1.5
assert.InDelta(t, 1.5, finalStats.AvgQPS, 0.01, "Average QPS mismatch")
// Basic Token Rate check (only successful)
// Total tokens = 150 + 160 = 310
// Total successful time = 100ms + 120ms = 220ms = 0.22s
// Expected rate = 310 / 0.22 = ~1409 tokens/sec
assert.InDelta(t, 310/0.22, finalStats.AvgTokensPerSecond, 1, "Average tokens/sec mismatch") // Larger delta due to potential floating point issues
// Check Percentiles (calculated manually for [100, 120] ms latency)
// P90 = 100*(1-0.9) + 120*0.9 = 10 + 108 = 118
// P95 = 100*(1-0.95) + 120*0.95 = 5 + 114 = 119
// P99 = 100*(1-0.99) + 120*0.99 = 1 + 118.8 = 119.8
assert.Equal(t, 118*time.Millisecond, finalStats.P90Latency, "P90 Latency mismatch")
assert.Equal(t, 119*time.Millisecond, finalStats.P95Latency, "P95 Latency mismatch")
assert.InDelta(t, 119.8*float64(time.Millisecond), float64(finalStats.P99Latency.Nanoseconds()), 0.1*float64(time.Millisecond), "P99 Latency mismatch")
// Check Percentiles (calculated manually for [50, 60] ms TTFT)
// P90 = 50*0.1 + 60*0.9 = 5 + 54 = 59
// P95 = 50*0.05 + 60*0.95 = 2.5 + 57 = 59.5
// P99 = 50*0.01 + 60*0.99 = 0.5 + 59.4 = 59.9
assert.Equal(t, 59*time.Millisecond, finalStats.P90TimeToFirstToken, "P90 TTFT mismatch")
assert.InDelta(t, 59.5*float64(time.Millisecond), float64(finalStats.P95TimeToFirstToken.Nanoseconds()), 0.1*float64(time.Millisecond), "P95 TTFT mismatch")
assert.InDelta(t, 59.9*float64(time.Millisecond), float64(finalStats.P99TimeToFirstToken.Nanoseconds()), 0.1*float64(time.Millisecond), "P99 TTFT mismatch")
}
// TestStatsCollector_ConcurrentRecording tests thread-safety.
func TestStatsCollector_ConcurrentRecording(t *testing.T) {
collector := stats.NewStatsCollector()
numGoroutines := 50
numRequestsPerGoroutine := 100
totalRequests := numGoroutines * numRequestsPerGoroutine
var wg sync.WaitGroup
wg.Add(numGoroutines)
for i := 0; i < numGoroutines; i++ {
go func(routineID int) {
defer wg.Done()
for j := 0; j < numRequestsPerGoroutine; j++ {
latency := time.Duration(100+routineID+j) * time.Millisecond
ttft := time.Duration(50+routineID+j) * time.Millisecond
collector.RecordResult(stats.RequestResult{
IsSuccess: true,
Latency: latency,
TimeToFirstToken: ttft,
TotalTokens: 100,
StreamLatency: nil,
})
}
}(i)
}
wg.Wait()
finalStats := collector.CalculateStats(10 * time.Second) // Arbitrary duration
assert.Equal(t, int64(totalRequests), finalStats.TotalRequests, "Total requests should match after concurrent writes")
assert.Equal(t, int64(totalRequests), finalStats.SuccessfulRequests, "All requests should be successful")
assert.Equal(t, int64(0), finalStats.FailedRequests, "No requests should fail")
// Add more assertions for concurrent results if necessary
}
+62
View File
@@ -0,0 +1,62 @@
package test
import (
"fmt"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/tiktoken-go/tokenizer" // Assuming this is the correct import path
token "llm-api-benchmark-tool/pkg/tokenizer" // Target package (to be created)
)
const (
tolerance = 0.05 // 5%
modelName = "gpt-4" // Or another common model for tiktoken
)
// Helper function to count tokens
func countTokens(t *testing.T, text string, model string) int {
codec, err := tokenizer.Get(tokenizer.Cl100kBase) // Or appropriate codec based on model
require.NoError(t, err, "Failed to get tokenizer codec")
ids, _, err := codec.Encode(text)
require.NoError(t, err, "Failed to encode text")
return len(ids)
}
func TestGeneratePrompt50Tokens(t *testing.T) {
targetTokens := 50
minTokens := int(float64(targetTokens) * (1 - tolerance))
maxTokens := int(float64(targetTokens) * (1 + tolerance))
// Assuming GeneratePrompt takes the target count and maybe a model/codec hint
prompt, err := token.GeneratePrompt(targetTokens)
require.NoError(t, err, "GeneratePrompt should not return an error for 50 tokens")
require.NotEmpty(t, prompt, "Generated prompt should not be empty")
actualTokens := countTokens(t, prompt, modelName)
assert.GreaterOrEqual(t, actualTokens, minTokens, fmt.Sprintf("Token count %d should be >= %d (target %d)", actualTokens, minTokens, targetTokens))
assert.LessOrEqual(t, actualTokens, maxTokens, fmt.Sprintf("Token count %d should be <= %d (target %d)", actualTokens, maxTokens, targetTokens))
t.Logf("Generated 50-token prompt (target: %d±%.0f%%): %d tokens", targetTokens, tolerance*100, actualTokens)
}
func TestGeneratePrompt1000Tokens(t *testing.T) {
targetTokens := 1000
minTokens := int(float64(targetTokens) * (1 - tolerance))
maxTokens := int(float64(targetTokens) * (1 + tolerance))
// Assuming GeneratePrompt takes the target count
prompt, err := token.GeneratePrompt(targetTokens)
require.NoError(t, err, "GeneratePrompt should not return an error for 1000 tokens")
require.NotEmpty(t, prompt, "Generated prompt should not be empty")
actualTokens := countTokens(t, prompt, modelName)
assert.GreaterOrEqual(t, actualTokens, minTokens, fmt.Sprintf("Token count %d should be >= %d (target %d)", actualTokens, minTokens, targetTokens))
assert.LessOrEqual(t, actualTokens, maxTokens, fmt.Sprintf("Token count %d should be <= %d (target %d)", actualTokens, maxTokens, targetTokens))
t.Logf("Generated 1000-token prompt (target: %d±%.0f%%): %d tokens", targetTokens, tolerance*100, actualTokens)
}
// TODO: Add test for invalid target token count (e.g., 0 or negative) if applicable
+19
View File
@@ -0,0 +1,19 @@
Copyright (c) 2009, 2010, 2013-2016 by the Brotli Authors.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+14
View File
@@ -0,0 +1,14 @@
This package is a brotli compressor and decompressor implemented in Go.
It was translated from the reference implementation (https://github.com/google/brotli)
with the `c2go` tool at https://github.com/andybalholm/c2go.
I have been working on new compression algorithms (not translated from C)
in the matchfinder package.
You can use them with the NewWriterV2 function.
Currently they give better results than the old implementation
(at least for compressing my test file, Newton’s *Opticks*)
on levels 2 to 6.
I am using it in production with https://github.com/andybalholm/redwood.
API documentation is found at https://pkg.go.dev/github.com/andybalholm/brotli?tab=doc.
+185
View File
@@ -0,0 +1,185 @@
package brotli
import (
"sync"
)
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Function to find backward reference copies. */
func computeDistanceCode(distance uint, max_distance uint, dist_cache []int) uint {
if distance <= max_distance {
var distance_plus_3 uint = distance + 3
var offset0 uint = distance_plus_3 - uint(dist_cache[0])
var offset1 uint = distance_plus_3 - uint(dist_cache[1])
if distance == uint(dist_cache[0]) {
return 0
} else if distance == uint(dist_cache[1]) {
return 1
} else if offset0 < 7 {
return (0x9750468 >> (4 * offset0)) & 0xF
} else if offset1 < 7 {
return (0xFDB1ACE >> (4 * offset1)) & 0xF
} else if distance == uint(dist_cache[2]) {
return 2
} else if distance == uint(dist_cache[3]) {
return 3
}
}
return distance + numDistanceShortCodes - 1
}
var hasherSearchResultPool sync.Pool
func createBackwardReferences(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, hasher hasherHandle, dist_cache []int, last_insert_len *uint, commands *[]command, num_literals *uint) {
var max_backward_limit uint = maxBackwardLimit(params.lgwin)
var insert_length uint = *last_insert_len
var pos_end uint = position + num_bytes
var store_end uint
if num_bytes >= hasher.StoreLookahead() {
store_end = position + num_bytes - hasher.StoreLookahead() + 1
} else {
store_end = position
}
var random_heuristics_window_size uint = literalSpreeLengthForSparseSearch(params)
var apply_random_heuristics uint = position + random_heuristics_window_size
var gap uint = 0
/* Set maximum distance, see section 9.1. of the spec. */
const kMinScore uint = scoreBase + 100
/* For speed up heuristics for random data. */
/* Minimum score to accept a backward reference. */
hasher.PrepareDistanceCache(dist_cache)
sr2, _ := hasherSearchResultPool.Get().(*hasherSearchResult)
if sr2 == nil {
sr2 = &hasherSearchResult{}
}
sr, _ := hasherSearchResultPool.Get().(*hasherSearchResult)
if sr == nil {
sr = &hasherSearchResult{}
}
for position+hasher.HashTypeLength() < pos_end {
var max_length uint = pos_end - position
var max_distance uint = brotli_min_size_t(position, max_backward_limit)
sr.len = 0
sr.len_code_delta = 0
sr.distance = 0
sr.score = kMinScore
hasher.FindLongestMatch(&params.dictionary, ringbuffer, ringbuffer_mask, dist_cache, position, max_length, max_distance, gap, params.dist.max_distance, sr)
if sr.score > kMinScore {
/* Found a match. Let's look for something even better ahead. */
var delayed_backward_references_in_row int = 0
max_length--
for ; ; max_length-- {
var cost_diff_lazy uint = 175
if params.quality < minQualityForExtensiveReferenceSearch {
sr2.len = brotli_min_size_t(sr.len-1, max_length)
} else {
sr2.len = 0
}
sr2.len_code_delta = 0
sr2.distance = 0
sr2.score = kMinScore
max_distance = brotli_min_size_t(position+1, max_backward_limit)
hasher.FindLongestMatch(&params.dictionary, ringbuffer, ringbuffer_mask, dist_cache, position+1, max_length, max_distance, gap, params.dist.max_distance, sr2)
if sr2.score >= sr.score+cost_diff_lazy {
/* Ok, let's just write one byte for now and start a match from the
next byte. */
position++
insert_length++
*sr = *sr2
delayed_backward_references_in_row++
if delayed_backward_references_in_row < 4 && position+hasher.HashTypeLength() < pos_end {
continue
}
}
break
}
apply_random_heuristics = position + 2*sr.len + random_heuristics_window_size
max_distance = brotli_min_size_t(position, max_backward_limit)
{
/* The first 16 codes are special short-codes,
and the minimum offset is 1. */
var distance_code uint = computeDistanceCode(sr.distance, max_distance+gap, dist_cache)
if (sr.distance <= (max_distance + gap)) && distance_code > 0 {
dist_cache[3] = dist_cache[2]
dist_cache[2] = dist_cache[1]
dist_cache[1] = dist_cache[0]
dist_cache[0] = int(sr.distance)
hasher.PrepareDistanceCache(dist_cache)
}
*commands = append(*commands, makeCommand(&params.dist, insert_length, sr.len, sr.len_code_delta, distance_code))
}
*num_literals += insert_length
insert_length = 0
/* Put the hash keys into the table, if there are enough bytes left.
Depending on the hasher implementation, it can push all positions
in the given range or only a subset of them.
Avoid hash poisoning with RLE data. */
{
var range_start uint = position + 2
var range_end uint = brotli_min_size_t(position+sr.len, store_end)
if sr.distance < sr.len>>2 {
range_start = brotli_min_size_t(range_end, brotli_max_size_t(range_start, position+sr.len-(sr.distance<<2)))
}
hasher.StoreRange(ringbuffer, ringbuffer_mask, range_start, range_end)
}
position += sr.len
} else {
insert_length++
position++
/* If we have not seen matches for a long time, we can skip some
match lookups. Unsuccessful match lookups are very very expensive
and this kind of a heuristic speeds up compression quite
a lot. */
if position > apply_random_heuristics {
/* Going through uncompressible data, jump. */
if position > apply_random_heuristics+4*random_heuristics_window_size {
var kMargin uint = brotli_max_size_t(hasher.StoreLookahead()-1, 4)
/* It is quite a long time since we saw a copy, so we assume
that this data is not compressible, and store hashes less
often. Hashes of non compressible data are less likely to
turn out to be useful in the future, too, so we store less of
them to not to flood out the hash table of good compressible
data. */
var pos_jump uint = brotli_min_size_t(position+16, pos_end-kMargin)
for ; position < pos_jump; position += 4 {
hasher.Store(ringbuffer, ringbuffer_mask, position)
insert_length += 4
}
} else {
var kMargin uint = brotli_max_size_t(hasher.StoreLookahead()-1, 2)
var pos_jump uint = brotli_min_size_t(position+8, pos_end-kMargin)
for ; position < pos_jump; position += 2 {
hasher.Store(ringbuffer, ringbuffer_mask, position)
insert_length += 2
}
}
}
}
}
insert_length += pos_end - position
*last_insert_len = insert_length
hasherSearchResultPool.Put(sr)
hasherSearchResultPool.Put(sr2)
}
+796
View File
@@ -0,0 +1,796 @@
package brotli
import "math"
type zopfliNode struct {
length uint32
distance uint32
dcode_insert_length uint32
u struct {
cost float32
next uint32
shortcut uint32
}
}
const maxEffectiveDistanceAlphabetSize = 544
const kInfinity float32 = 1.7e38 /* ~= 2 ^ 127 */
var kDistanceCacheIndex = []uint32{0, 1, 2, 3, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1}
var kDistanceCacheOffset = []int{0, 0, 0, 0, -1, 1, -2, 2, -3, 3, -1, 1, -2, 2, -3, 3}
func initZopfliNodes(array []zopfliNode, length uint) {
var stub zopfliNode
var i uint
stub.length = 1
stub.distance = 0
stub.dcode_insert_length = 0
stub.u.cost = kInfinity
for i = 0; i < length; i++ {
array[i] = stub
}
}
func zopfliNodeCopyLength(self *zopfliNode) uint32 {
return self.length & 0x1FFFFFF
}
func zopfliNodeLengthCode(self *zopfliNode) uint32 {
var modifier uint32 = self.length >> 25
return zopfliNodeCopyLength(self) + 9 - modifier
}
func zopfliNodeCopyDistance(self *zopfliNode) uint32 {
return self.distance
}
func zopfliNodeDistanceCode(self *zopfliNode) uint32 {
var short_code uint32 = self.dcode_insert_length >> 27
if short_code == 0 {
return zopfliNodeCopyDistance(self) + numDistanceShortCodes - 1
} else {
return short_code - 1
}
}
func zopfliNodeCommandLength(self *zopfliNode) uint32 {
return zopfliNodeCopyLength(self) + (self.dcode_insert_length & 0x7FFFFFF)
}
/* Histogram based cost model for zopflification. */
type zopfliCostModel struct {
cost_cmd_ [numCommandSymbols]float32
cost_dist_ []float32
distance_histogram_size uint32
literal_costs_ []float32
min_cost_cmd_ float32
num_bytes_ uint
}
func initZopfliCostModel(self *zopfliCostModel, dist *distanceParams, num_bytes uint) {
var distance_histogram_size uint32 = dist.alphabet_size
if distance_histogram_size > maxEffectiveDistanceAlphabetSize {
distance_histogram_size = maxEffectiveDistanceAlphabetSize
}
self.num_bytes_ = num_bytes
self.literal_costs_ = make([]float32, (num_bytes + 2))
self.cost_dist_ = make([]float32, (dist.alphabet_size))
self.distance_histogram_size = distance_histogram_size
}
func cleanupZopfliCostModel(self *zopfliCostModel) {
self.literal_costs_ = nil
self.cost_dist_ = nil
}
func setCost(histogram []uint32, histogram_size uint, literal_histogram bool, cost []float32) {
var sum uint = 0
var missing_symbol_sum uint
var log2sum float32
var missing_symbol_cost float32
var i uint
for i = 0; i < histogram_size; i++ {
sum += uint(histogram[i])
}
log2sum = float32(fastLog2(sum))
missing_symbol_sum = sum
if !literal_histogram {
for i = 0; i < histogram_size; i++ {
if histogram[i] == 0 {
missing_symbol_sum++
}
}
}
missing_symbol_cost = float32(fastLog2(missing_symbol_sum)) + 2
for i = 0; i < histogram_size; i++ {
if histogram[i] == 0 {
cost[i] = missing_symbol_cost
continue
}
/* Shannon bits for this symbol. */
cost[i] = log2sum - float32(fastLog2(uint(histogram[i])))
/* Cannot be coded with less than 1 bit */
if cost[i] < 1 {
cost[i] = 1
}
}
}
func zopfliCostModelSetFromCommands(self *zopfliCostModel, position uint, ringbuffer []byte, ringbuffer_mask uint, commands []command, last_insert_len uint) {
var histogram_literal [numLiteralSymbols]uint32
var histogram_cmd [numCommandSymbols]uint32
var histogram_dist [maxEffectiveDistanceAlphabetSize]uint32
var cost_literal [numLiteralSymbols]float32
var pos uint = position - last_insert_len
var min_cost_cmd float32 = kInfinity
var cost_cmd []float32 = self.cost_cmd_[:]
var literal_costs []float32
histogram_literal = [numLiteralSymbols]uint32{}
histogram_cmd = [numCommandSymbols]uint32{}
histogram_dist = [maxEffectiveDistanceAlphabetSize]uint32{}
for i := range commands {
var inslength uint = uint(commands[i].insert_len_)
var copylength uint = uint(commandCopyLen(&commands[i]))
var distcode uint = uint(commands[i].dist_prefix_) & 0x3FF
var cmdcode uint = uint(commands[i].cmd_prefix_)
var j uint
histogram_cmd[cmdcode]++
if cmdcode >= 128 {
histogram_dist[distcode]++
}
for j = 0; j < inslength; j++ {
histogram_literal[ringbuffer[(pos+j)&ringbuffer_mask]]++
}
pos += inslength + copylength
}
setCost(histogram_literal[:], numLiteralSymbols, true, cost_literal[:])
setCost(histogram_cmd[:], numCommandSymbols, false, cost_cmd)
setCost(histogram_dist[:], uint(self.distance_histogram_size), false, self.cost_dist_)
for i := 0; i < numCommandSymbols; i++ {
min_cost_cmd = brotli_min_float(min_cost_cmd, cost_cmd[i])
}
self.min_cost_cmd_ = min_cost_cmd
{
literal_costs = self.literal_costs_
var literal_carry float32 = 0.0
num_bytes := int(self.num_bytes_)
literal_costs[0] = 0.0
for i := 0; i < num_bytes; i++ {
literal_carry += cost_literal[ringbuffer[(position+uint(i))&ringbuffer_mask]]
literal_costs[i+1] = literal_costs[i] + literal_carry
literal_carry -= literal_costs[i+1] - literal_costs[i]
}
}
}
func zopfliCostModelSetFromLiteralCosts(self *zopfliCostModel, position uint, ringbuffer []byte, ringbuffer_mask uint) {
var literal_costs []float32 = self.literal_costs_
var literal_carry float32 = 0.0
var cost_dist []float32 = self.cost_dist_
var cost_cmd []float32 = self.cost_cmd_[:]
var num_bytes uint = self.num_bytes_
var i uint
estimateBitCostsForLiterals(position, num_bytes, ringbuffer_mask, ringbuffer, literal_costs[1:])
literal_costs[0] = 0.0
for i = 0; i < num_bytes; i++ {
literal_carry += literal_costs[i+1]
literal_costs[i+1] = literal_costs[i] + literal_carry
literal_carry -= literal_costs[i+1] - literal_costs[i]
}
for i = 0; i < numCommandSymbols; i++ {
cost_cmd[i] = float32(fastLog2(uint(11 + uint32(i))))
}
for i = 0; uint32(i) < self.distance_histogram_size; i++ {
cost_dist[i] = float32(fastLog2(uint(20 + uint32(i))))
}
self.min_cost_cmd_ = float32(fastLog2(11))
}
func zopfliCostModelGetCommandCost(self *zopfliCostModel, cmdcode uint16) float32 {
return self.cost_cmd_[cmdcode]
}
func zopfliCostModelGetDistanceCost(self *zopfliCostModel, distcode uint) float32 {
return self.cost_dist_[distcode]
}
func zopfliCostModelGetLiteralCosts(self *zopfliCostModel, from uint, to uint) float32 {
return self.literal_costs_[to] - self.literal_costs_[from]
}
func zopfliCostModelGetMinCostCmd(self *zopfliCostModel) float32 {
return self.min_cost_cmd_
}
/* REQUIRES: len >= 2, start_pos <= pos */
/* REQUIRES: cost < kInfinity, nodes[start_pos].cost < kInfinity */
/* Maintains the "ZopfliNode array invariant". */
func updateZopfliNode(nodes []zopfliNode, pos uint, start_pos uint, len uint, len_code uint, dist uint, short_code uint, cost float32) {
var next *zopfliNode = &nodes[pos+len]
next.length = uint32(len | (len+9-len_code)<<25)
next.distance = uint32(dist)
next.dcode_insert_length = uint32(short_code<<27 | (pos - start_pos))
next.u.cost = cost
}
type posData struct {
pos uint
distance_cache [4]int
costdiff float32
cost float32
}
/* Maintains the smallest 8 cost difference together with their positions */
type startPosQueue struct {
q_ [8]posData
idx_ uint
}
func initStartPosQueue(self *startPosQueue) {
self.idx_ = 0
}
func startPosQueueSize(self *startPosQueue) uint {
return brotli_min_size_t(self.idx_, 8)
}
func startPosQueuePush(self *startPosQueue, posdata *posData) {
var offset uint = ^(self.idx_) & 7
self.idx_++
var len uint = startPosQueueSize(self)
var i uint
var q []posData = self.q_[:]
q[offset] = *posdata
/* Restore the sorted order. In the list of |len| items at most |len - 1|
adjacent element comparisons / swaps are required. */
for i = 1; i < len; i++ {
if q[offset&7].costdiff > q[(offset+1)&7].costdiff {
var tmp posData = q[offset&7]
q[offset&7] = q[(offset+1)&7]
q[(offset+1)&7] = tmp
}
offset++
}
}
func startPosQueueAt(self *startPosQueue, k uint) *posData {
return &self.q_[(k-self.idx_)&7]
}
/* Returns the minimum possible copy length that can improve the cost of any */
/* future position. */
func computeMinimumCopyLength(start_cost float32, nodes []zopfliNode, num_bytes uint, pos uint) uint {
var min_cost float32 = start_cost
var len uint = 2
var next_len_bucket uint = 4
/* Compute the minimum possible cost of reaching any future position. */
var next_len_offset uint = 10
for pos+len <= num_bytes && nodes[pos+len].u.cost <= min_cost {
/* We already reached (pos + len) with no more cost than the minimum
possible cost of reaching anything from this pos, so there is no point in
looking for lengths <= len. */
len++
if len == next_len_offset {
/* We reached the next copy length code bucket, so we add one more
extra bit to the minimum cost. */
min_cost += 1.0
next_len_offset += next_len_bucket
next_len_bucket *= 2
}
}
return uint(len)
}
/* REQUIRES: nodes[pos].cost < kInfinity
REQUIRES: nodes[0..pos] satisfies that "ZopfliNode array invariant". */
func computeDistanceShortcut(block_start uint, pos uint, max_backward_limit uint, gap uint, nodes []zopfliNode) uint32 {
var clen uint = uint(zopfliNodeCopyLength(&nodes[pos]))
var ilen uint = uint(nodes[pos].dcode_insert_length & 0x7FFFFFF)
var dist uint = uint(zopfliNodeCopyDistance(&nodes[pos]))
/* Since |block_start + pos| is the end position of the command, the copy part
starts from |block_start + pos - clen|. Distances that are greater than
this or greater than |max_backward_limit| + |gap| are static dictionary
references, and do not update the last distances.
Also distance code 0 (last distance) does not update the last distances. */
if pos == 0 {
return 0
} else if dist+clen <= block_start+pos+gap && dist <= max_backward_limit+gap && zopfliNodeDistanceCode(&nodes[pos]) > 0 {
return uint32(pos)
} else {
return nodes[pos-clen-ilen].u.shortcut
}
}
/* Fills in dist_cache[0..3] with the last four distances (as defined by
Section 4. of the Spec) that would be used at (block_start + pos) if we
used the shortest path of commands from block_start, computed from
nodes[0..pos]. The last four distances at block_start are in
starting_dist_cache[0..3].
REQUIRES: nodes[pos].cost < kInfinity
REQUIRES: nodes[0..pos] satisfies that "ZopfliNode array invariant". */
func computeDistanceCache(pos uint, starting_dist_cache []int, nodes []zopfliNode, dist_cache []int) {
var idx int = 0
var p uint = uint(nodes[pos].u.shortcut)
for idx < 4 && p > 0 {
var ilen uint = uint(nodes[p].dcode_insert_length & 0x7FFFFFF)
var clen uint = uint(zopfliNodeCopyLength(&nodes[p]))
var dist uint = uint(zopfliNodeCopyDistance(&nodes[p]))
dist_cache[idx] = int(dist)
idx++
/* Because of prerequisite, p >= clen + ilen >= 2. */
p = uint(nodes[p-clen-ilen].u.shortcut)
}
for ; idx < 4; idx++ {
dist_cache[idx] = starting_dist_cache[0]
starting_dist_cache = starting_dist_cache[1:]
}
}
/* Maintains "ZopfliNode array invariant" and pushes node to the queue, if it
is eligible. */
func evaluateNode(block_start uint, pos uint, max_backward_limit uint, gap uint, starting_dist_cache []int, model *zopfliCostModel, queue *startPosQueue, nodes []zopfliNode) {
/* Save cost, because ComputeDistanceCache invalidates it. */
var node_cost float32 = nodes[pos].u.cost
nodes[pos].u.shortcut = computeDistanceShortcut(block_start, pos, max_backward_limit, gap, nodes)
if node_cost <= zopfliCostModelGetLiteralCosts(model, 0, pos) {
var posdata posData
posdata.pos = pos
posdata.cost = node_cost
posdata.costdiff = node_cost - zopfliCostModelGetLiteralCosts(model, 0, pos)
computeDistanceCache(pos, starting_dist_cache, nodes, posdata.distance_cache[:])
startPosQueuePush(queue, &posdata)
}
}
/* Returns longest copy length. */
func updateNodes(num_bytes uint, block_start uint, pos uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, max_backward_limit uint, starting_dist_cache []int, num_matches uint, matches []backwardMatch, model *zopfliCostModel, queue *startPosQueue, nodes []zopfliNode) uint {
var cur_ix uint = block_start + pos
var cur_ix_masked uint = cur_ix & ringbuffer_mask
var max_distance uint = brotli_min_size_t(cur_ix, max_backward_limit)
var max_len uint = num_bytes - pos
var max_zopfli_len uint = maxZopfliLen(params)
var max_iters uint = maxZopfliCandidates(params)
var min_len uint
var result uint = 0
var k uint
var gap uint = 0
evaluateNode(block_start, pos, max_backward_limit, gap, starting_dist_cache, model, queue, nodes)
{
var posdata *posData = startPosQueueAt(queue, 0)
var min_cost float32 = (posdata.cost + zopfliCostModelGetMinCostCmd(model) + zopfliCostModelGetLiteralCosts(model, posdata.pos, pos))
min_len = computeMinimumCopyLength(min_cost, nodes, num_bytes, pos)
}
/* Go over the command starting positions in order of increasing cost
difference. */
for k = 0; k < max_iters && k < startPosQueueSize(queue); k++ {
var posdata *posData = startPosQueueAt(queue, k)
var start uint = posdata.pos
var inscode uint16 = getInsertLengthCode(pos - start)
var start_costdiff float32 = posdata.costdiff
var base_cost float32 = start_costdiff + float32(getInsertExtra(inscode)) + zopfliCostModelGetLiteralCosts(model, 0, pos)
var best_len uint = min_len - 1
var j uint = 0
/* Look for last distance matches using the distance cache from this
starting position. */
for ; j < numDistanceShortCodes && best_len < max_len; j++ {
var idx uint = uint(kDistanceCacheIndex[j])
var backward uint = uint(posdata.distance_cache[idx] + kDistanceCacheOffset[j])
var prev_ix uint = cur_ix - backward
var len uint = 0
var continuation byte = ringbuffer[cur_ix_masked+best_len]
if cur_ix_masked+best_len > ringbuffer_mask {
break
}
if backward > max_distance+gap {
/* Word dictionary -> ignore. */
continue
}
if backward <= max_distance {
/* Regular backward reference. */
if prev_ix >= cur_ix {
continue
}
prev_ix &= ringbuffer_mask
if prev_ix+best_len > ringbuffer_mask || continuation != ringbuffer[prev_ix+best_len] {
continue
}
len = findMatchLengthWithLimit(ringbuffer[prev_ix:], ringbuffer[cur_ix_masked:], max_len)
} else {
continue
}
{
var dist_cost float32 = base_cost + zopfliCostModelGetDistanceCost(model, j)
var l uint
for l = best_len + 1; l <= len; l++ {
var copycode uint16 = getCopyLengthCode(l)
var cmdcode uint16 = combineLengthCodes(inscode, copycode, j == 0)
var tmp float32
if cmdcode < 128 {
tmp = base_cost
} else {
tmp = dist_cost
}
var cost float32 = tmp + float32(getCopyExtra(copycode)) + zopfliCostModelGetCommandCost(model, cmdcode)
if cost < nodes[pos+l].u.cost {
updateZopfliNode(nodes, pos, start, l, l, backward, j+1, cost)
result = brotli_max_size_t(result, l)
}
best_len = l
}
}
}
/* At higher iterations look only for new last distance matches, since
looking only for new command start positions with the same distances
does not help much. */
if k >= 2 {
continue
}
{
/* Loop through all possible copy lengths at this position. */
var len uint = min_len
for j = 0; j < num_matches; j++ {
var match backwardMatch = matches[j]
var dist uint = uint(match.distance)
var is_dictionary_match bool = (dist > max_distance+gap)
var dist_code uint = dist + numDistanceShortCodes - 1
var dist_symbol uint16
var distextra uint32
var distnumextra uint32
var dist_cost float32
var max_match_len uint
/* We already tried all possible last distance matches, so we can use
normal distance code here. */
prefixEncodeCopyDistance(dist_code, uint(params.dist.num_direct_distance_codes), uint(params.dist.distance_postfix_bits), &dist_symbol, &distextra)
distnumextra = uint32(dist_symbol) >> 10
dist_cost = base_cost + float32(distnumextra) + zopfliCostModelGetDistanceCost(model, uint(dist_symbol)&0x3FF)
/* Try all copy lengths up until the maximum copy length corresponding
to this distance. If the distance refers to the static dictionary, or
the maximum length is long enough, try only one maximum length. */
max_match_len = backwardMatchLength(&match)
if len < max_match_len && (is_dictionary_match || max_match_len > max_zopfli_len) {
len = max_match_len
}
for ; len <= max_match_len; len++ {
var len_code uint
if is_dictionary_match {
len_code = backwardMatchLengthCode(&match)
} else {
len_code = len
}
var copycode uint16 = getCopyLengthCode(len_code)
var cmdcode uint16 = combineLengthCodes(inscode, copycode, false)
var cost float32 = dist_cost + float32(getCopyExtra(copycode)) + zopfliCostModelGetCommandCost(model, cmdcode)
if cost < nodes[pos+len].u.cost {
updateZopfliNode(nodes, pos, start, uint(len), len_code, dist, 0, cost)
if len > result {
result = len
}
}
}
}
}
}
return result
}
func computeShortestPathFromNodes(num_bytes uint, nodes []zopfliNode) uint {
var index uint = num_bytes
var num_commands uint = 0
for nodes[index].dcode_insert_length&0x7FFFFFF == 0 && nodes[index].length == 1 {
index--
}
nodes[index].u.next = math.MaxUint32
for index != 0 {
var len uint = uint(zopfliNodeCommandLength(&nodes[index]))
index -= uint(len)
nodes[index].u.next = uint32(len)
num_commands++
}
return num_commands
}
/* REQUIRES: nodes != NULL and len(nodes) >= num_bytes + 1 */
func zopfliCreateCommands(num_bytes uint, block_start uint, nodes []zopfliNode, dist_cache []int, last_insert_len *uint, params *encoderParams, commands *[]command, num_literals *uint) {
var max_backward_limit uint = maxBackwardLimit(params.lgwin)
var pos uint = 0
var offset uint32 = nodes[0].u.next
var i uint
var gap uint = 0
for i = 0; offset != math.MaxUint32; i++ {
var next *zopfliNode = &nodes[uint32(pos)+offset]
var copy_length uint = uint(zopfliNodeCopyLength(next))
var insert_length uint = uint(next.dcode_insert_length & 0x7FFFFFF)
pos += insert_length
offset = next.u.next
if i == 0 {
insert_length += *last_insert_len
*last_insert_len = 0
}
{
var distance uint = uint(zopfliNodeCopyDistance(next))
var len_code uint = uint(zopfliNodeLengthCode(next))
var max_distance uint = brotli_min_size_t(block_start+pos, max_backward_limit)
var is_dictionary bool = (distance > max_distance+gap)
var dist_code uint = uint(zopfliNodeDistanceCode(next))
*commands = append(*commands, makeCommand(&params.dist, insert_length, copy_length, int(len_code)-int(copy_length), dist_code))
if !is_dictionary && dist_code > 0 {
dist_cache[3] = dist_cache[2]
dist_cache[2] = dist_cache[1]
dist_cache[1] = dist_cache[0]
dist_cache[0] = int(distance)
}
}
*num_literals += insert_length
pos += copy_length
}
*last_insert_len += num_bytes - pos
}
func zopfliIterate(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, gap uint, dist_cache []int, model *zopfliCostModel, num_matches []uint32, matches []backwardMatch, nodes []zopfliNode) uint {
var max_backward_limit uint = maxBackwardLimit(params.lgwin)
var max_zopfli_len uint = maxZopfliLen(params)
var queue startPosQueue
var cur_match_pos uint = 0
var i uint
nodes[0].length = 0
nodes[0].u.cost = 0
initStartPosQueue(&queue)
for i = 0; i+3 < num_bytes; i++ {
var skip uint = updateNodes(num_bytes, position, i, ringbuffer, ringbuffer_mask, params, max_backward_limit, dist_cache, uint(num_matches[i]), matches[cur_match_pos:], model, &queue, nodes)
if skip < longCopyQuickStep {
skip = 0
}
cur_match_pos += uint(num_matches[i])
if num_matches[i] == 1 && backwardMatchLength(&matches[cur_match_pos-1]) > max_zopfli_len {
skip = brotli_max_size_t(backwardMatchLength(&matches[cur_match_pos-1]), skip)
}
if skip > 1 {
skip--
for skip != 0 {
i++
if i+3 >= num_bytes {
break
}
evaluateNode(position, i, max_backward_limit, gap, dist_cache, model, &queue, nodes)
cur_match_pos += uint(num_matches[i])
skip--
}
}
}
return computeShortestPathFromNodes(num_bytes, nodes)
}
/* Computes the shortest path of commands from position to at most
position + num_bytes.
On return, path->size() is the number of commands found and path[i] is the
length of the i-th command (copy length plus insert length).
Note that the sum of the lengths of all commands can be less than num_bytes.
On return, the nodes[0..num_bytes] array will have the following
"ZopfliNode array invariant":
For each i in [1..num_bytes], if nodes[i].cost < kInfinity, then
(1) nodes[i].copy_length() >= 2
(2) nodes[i].command_length() <= i and
(3) nodes[i - nodes[i].command_length()].cost < kInfinity
REQUIRES: nodes != nil and len(nodes) >= num_bytes + 1 */
func zopfliComputeShortestPath(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, dist_cache []int, hasher *h10, nodes []zopfliNode) uint {
var max_backward_limit uint = maxBackwardLimit(params.lgwin)
var max_zopfli_len uint = maxZopfliLen(params)
var model zopfliCostModel
var queue startPosQueue
var matches [2 * (maxNumMatchesH10 + 64)]backwardMatch
var store_end uint
if num_bytes >= hasher.StoreLookahead() {
store_end = position + num_bytes - hasher.StoreLookahead() + 1
} else {
store_end = position
}
var i uint
var gap uint = 0
var lz_matches_offset uint = 0
nodes[0].length = 0
nodes[0].u.cost = 0
initZopfliCostModel(&model, &params.dist, num_bytes)
zopfliCostModelSetFromLiteralCosts(&model, position, ringbuffer, ringbuffer_mask)
initStartPosQueue(&queue)
for i = 0; i+hasher.HashTypeLength()-1 < num_bytes; i++ {
var pos uint = position + i
var max_distance uint = brotli_min_size_t(pos, max_backward_limit)
var skip uint
var num_matches uint
num_matches = findAllMatchesH10(hasher, &params.dictionary, ringbuffer, ringbuffer_mask, pos, num_bytes-i, max_distance, gap, params, matches[lz_matches_offset:])
if num_matches > 0 && backwardMatchLength(&matches[num_matches-1]) > max_zopfli_len {
matches[0] = matches[num_matches-1]
num_matches = 1
}
skip = updateNodes(num_bytes, position, i, ringbuffer, ringbuffer_mask, params, max_backward_limit, dist_cache, num_matches, matches[:], &model, &queue, nodes)
if skip < longCopyQuickStep {
skip = 0
}
if num_matches == 1 && backwardMatchLength(&matches[0]) > max_zopfli_len {
skip = brotli_max_size_t(backwardMatchLength(&matches[0]), skip)
}
if skip > 1 {
/* Add the tail of the copy to the hasher. */
hasher.StoreRange(ringbuffer, ringbuffer_mask, pos+1, brotli_min_size_t(pos+skip, store_end))
skip--
for skip != 0 {
i++
if i+hasher.HashTypeLength()-1 >= num_bytes {
break
}
evaluateNode(position, i, max_backward_limit, gap, dist_cache, &model, &queue, nodes)
skip--
}
}
}
cleanupZopfliCostModel(&model)
return computeShortestPathFromNodes(num_bytes, nodes)
}
func createZopfliBackwardReferences(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, hasher *h10, dist_cache []int, last_insert_len *uint, commands *[]command, num_literals *uint) {
var nodes []zopfliNode
nodes = make([]zopfliNode, (num_bytes + 1))
initZopfliNodes(nodes, num_bytes+1)
zopfliComputeShortestPath(num_bytes, position, ringbuffer, ringbuffer_mask, params, dist_cache, hasher, nodes)
zopfliCreateCommands(num_bytes, position, nodes, dist_cache, last_insert_len, params, commands, num_literals)
nodes = nil
}
func createHqZopfliBackwardReferences(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, hasher hasherHandle, dist_cache []int, last_insert_len *uint, commands *[]command, num_literals *uint) {
var max_backward_limit uint = maxBackwardLimit(params.lgwin)
var num_matches []uint32 = make([]uint32, num_bytes)
var matches_size uint = 4 * num_bytes
var store_end uint
if num_bytes >= hasher.StoreLookahead() {
store_end = position + num_bytes - hasher.StoreLookahead() + 1
} else {
store_end = position
}
var cur_match_pos uint = 0
var i uint
var orig_num_literals uint
var orig_last_insert_len uint
var orig_dist_cache [4]int
var orig_num_commands int
var model zopfliCostModel
var nodes []zopfliNode
var matches []backwardMatch = make([]backwardMatch, matches_size)
var gap uint = 0
var shadow_matches uint = 0
var new_array []backwardMatch
for i = 0; i+hasher.HashTypeLength()-1 < num_bytes; i++ {
var pos uint = position + i
var max_distance uint = brotli_min_size_t(pos, max_backward_limit)
var max_length uint = num_bytes - i
var num_found_matches uint
var cur_match_end uint
var j uint
/* Ensure that we have enough free slots. */
if matches_size < cur_match_pos+maxNumMatchesH10+shadow_matches {
var new_size uint = matches_size
if new_size == 0 {
new_size = cur_match_pos + maxNumMatchesH10 + shadow_matches
}
for new_size < cur_match_pos+maxNumMatchesH10+shadow_matches {
new_size *= 2
}
new_array = make([]backwardMatch, new_size)
if matches_size != 0 {
copy(new_array, matches[:matches_size])
}
matches = new_array
matches_size = new_size
}
num_found_matches = findAllMatchesH10(hasher.(*h10), &params.dictionary, ringbuffer, ringbuffer_mask, pos, max_length, max_distance, gap, params, matches[cur_match_pos+shadow_matches:])
cur_match_end = cur_match_pos + num_found_matches
for j = cur_match_pos; j+1 < cur_match_end; j++ {
assert(backwardMatchLength(&matches[j]) <= backwardMatchLength(&matches[j+1]))
}
num_matches[i] = uint32(num_found_matches)
if num_found_matches > 0 {
var match_len uint = backwardMatchLength(&matches[cur_match_end-1])
if match_len > maxZopfliLenQuality11 {
var skip uint = match_len - 1
matches[cur_match_pos] = matches[cur_match_end-1]
cur_match_pos++
num_matches[i] = 1
/* Add the tail of the copy to the hasher. */
hasher.StoreRange(ringbuffer, ringbuffer_mask, pos+1, brotli_min_size_t(pos+match_len, store_end))
var pos uint = i
for i := 0; i < int(skip); i++ {
num_matches[pos+1:][i] = 0
}
i += skip
} else {
cur_match_pos = cur_match_end
}
}
}
orig_num_literals = *num_literals
orig_last_insert_len = *last_insert_len
copy(orig_dist_cache[:], dist_cache[:4])
orig_num_commands = len(*commands)
nodes = make([]zopfliNode, (num_bytes + 1))
initZopfliCostModel(&model, &params.dist, num_bytes)
for i = 0; i < 2; i++ {
initZopfliNodes(nodes, num_bytes+1)
if i == 0 {
zopfliCostModelSetFromLiteralCosts(&model, position, ringbuffer, ringbuffer_mask)
} else {
zopfliCostModelSetFromCommands(&model, position, ringbuffer, ringbuffer_mask, (*commands)[orig_num_commands:], orig_last_insert_len)
}
*commands = (*commands)[:orig_num_commands]
*num_literals = orig_num_literals
*last_insert_len = orig_last_insert_len
copy(dist_cache, orig_dist_cache[:4])
zopfliIterate(num_bytes, position, ringbuffer, ringbuffer_mask, params, gap, dist_cache, &model, num_matches, matches, nodes)
zopfliCreateCommands(num_bytes, position, nodes, dist_cache, last_insert_len, params, commands, num_literals)
}
cleanupZopfliCostModel(&model)
nodes = nil
matches = nil
num_matches = nil
}
+436
View File
@@ -0,0 +1,436 @@
package brotli
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Functions to estimate the bit cost of Huffman trees. */
func shannonEntropy(population []uint32, size uint, total *uint) float64 {
var sum uint = 0
var retval float64 = 0
var population_end []uint32 = population[size:]
var p uint
for -cap(population) < -cap(population_end) {
p = uint(population[0])
population = population[1:]
sum += p
retval -= float64(p) * fastLog2(p)
}
if sum != 0 {
retval += float64(sum) * fastLog2(sum)
}
*total = sum
return retval
}
func bitsEntropy(population []uint32, size uint) float64 {
var sum uint
var retval float64 = shannonEntropy(population, size, &sum)
if retval < float64(sum) {
/* At least one bit per literal is needed. */
retval = float64(sum)
}
return retval
}
const kOneSymbolHistogramCost float64 = 12
const kTwoSymbolHistogramCost float64 = 20
const kThreeSymbolHistogramCost float64 = 28
const kFourSymbolHistogramCost float64 = 37
func populationCostLiteral(histogram *histogramLiteral) float64 {
var data_size uint = histogramDataSizeLiteral()
var count int = 0
var s [5]uint
var bits float64 = 0.0
var i uint
if histogram.total_count_ == 0 {
return kOneSymbolHistogramCost
}
for i = 0; i < data_size; i++ {
if histogram.data_[i] > 0 {
s[count] = i
count++
if count > 4 {
break
}
}
}
if count == 1 {
return kOneSymbolHistogramCost
}
if count == 2 {
return kTwoSymbolHistogramCost + float64(histogram.total_count_)
}
if count == 3 {
var histo0 uint32 = histogram.data_[s[0]]
var histo1 uint32 = histogram.data_[s[1]]
var histo2 uint32 = histogram.data_[s[2]]
var histomax uint32 = brotli_max_uint32_t(histo0, brotli_max_uint32_t(histo1, histo2))
return kThreeSymbolHistogramCost + 2*(float64(histo0)+float64(histo1)+float64(histo2)) - float64(histomax)
}
if count == 4 {
var histo [4]uint32
var h23 uint32
var histomax uint32
for i = 0; i < 4; i++ {
histo[i] = histogram.data_[s[i]]
}
/* Sort */
for i = 0; i < 4; i++ {
var j uint
for j = i + 1; j < 4; j++ {
if histo[j] > histo[i] {
var tmp uint32 = histo[j]
histo[j] = histo[i]
histo[i] = tmp
}
}
}
h23 = histo[2] + histo[3]
histomax = brotli_max_uint32_t(h23, histo[0])
return kFourSymbolHistogramCost + 3*float64(h23) + 2*(float64(histo[0])+float64(histo[1])) - float64(histomax)
}
{
var max_depth uint = 1
var depth_histo = [codeLengthCodes]uint32{0}
/* In this loop we compute the entropy of the histogram and simultaneously
build a simplified histogram of the code length codes where we use the
zero repeat code 17, but we don't use the non-zero repeat code 16. */
var log2total float64 = fastLog2(histogram.total_count_)
for i = 0; i < data_size; {
if histogram.data_[i] > 0 {
var log2p float64 = log2total - fastLog2(uint(histogram.data_[i]))
/* Compute -log2(P(symbol)) = -log2(count(symbol)/total_count) =
= log2(total_count) - log2(count(symbol)) */
var depth uint = uint(log2p + 0.5)
/* Approximate the bit depth by round(-log2(P(symbol))) */
bits += float64(histogram.data_[i]) * log2p
if depth > 15 {
depth = 15
}
if depth > max_depth {
max_depth = depth
}
depth_histo[depth]++
i++
} else {
var reps uint32 = 1
/* Compute the run length of zeros and add the appropriate number of 0
and 17 code length codes to the code length code histogram. */
var k uint
for k = i + 1; k < data_size && histogram.data_[k] == 0; k++ {
reps++
}
i += uint(reps)
if i == data_size {
/* Don't add any cost for the last zero run, since these are encoded
only implicitly. */
break
}
if reps < 3 {
depth_histo[0] += reps
} else {
reps -= 2
for reps > 0 {
depth_histo[repeatZeroCodeLength]++
/* Add the 3 extra bits for the 17 code length code. */
bits += 3
reps >>= 3
}
}
}
}
/* Add the estimated encoding cost of the code length code histogram. */
bits += float64(18 + 2*max_depth)
/* Add the entropy of the code length code histogram. */
bits += bitsEntropy(depth_histo[:], codeLengthCodes)
}
return bits
}
func populationCostCommand(histogram *histogramCommand) float64 {
var data_size uint = histogramDataSizeCommand()
var count int = 0
var s [5]uint
var bits float64 = 0.0
var i uint
if histogram.total_count_ == 0 {
return kOneSymbolHistogramCost
}
for i = 0; i < data_size; i++ {
if histogram.data_[i] > 0 {
s[count] = i
count++
if count > 4 {
break
}
}
}
if count == 1 {
return kOneSymbolHistogramCost
}
if count == 2 {
return kTwoSymbolHistogramCost + float64(histogram.total_count_)
}
if count == 3 {
var histo0 uint32 = histogram.data_[s[0]]
var histo1 uint32 = histogram.data_[s[1]]
var histo2 uint32 = histogram.data_[s[2]]
var histomax uint32 = brotli_max_uint32_t(histo0, brotli_max_uint32_t(histo1, histo2))
return kThreeSymbolHistogramCost + 2*(float64(histo0)+float64(histo1)+float64(histo2)) - float64(histomax)
}
if count == 4 {
var histo [4]uint32
var h23 uint32
var histomax uint32
for i = 0; i < 4; i++ {
histo[i] = histogram.data_[s[i]]
}
/* Sort */
for i = 0; i < 4; i++ {
var j uint
for j = i + 1; j < 4; j++ {
if histo[j] > histo[i] {
var tmp uint32 = histo[j]
histo[j] = histo[i]
histo[i] = tmp
}
}
}
h23 = histo[2] + histo[3]
histomax = brotli_max_uint32_t(h23, histo[0])
return kFourSymbolHistogramCost + 3*float64(h23) + 2*(float64(histo[0])+float64(histo[1])) - float64(histomax)
}
{
var max_depth uint = 1
var depth_histo = [codeLengthCodes]uint32{0}
/* In this loop we compute the entropy of the histogram and simultaneously
build a simplified histogram of the code length codes where we use the
zero repeat code 17, but we don't use the non-zero repeat code 16. */
var log2total float64 = fastLog2(histogram.total_count_)
for i = 0; i < data_size; {
if histogram.data_[i] > 0 {
var log2p float64 = log2total - fastLog2(uint(histogram.data_[i]))
/* Compute -log2(P(symbol)) = -log2(count(symbol)/total_count) =
= log2(total_count) - log2(count(symbol)) */
var depth uint = uint(log2p + 0.5)
/* Approximate the bit depth by round(-log2(P(symbol))) */
bits += float64(histogram.data_[i]) * log2p
if depth > 15 {
depth = 15
}
if depth > max_depth {
max_depth = depth
}
depth_histo[depth]++
i++
} else {
var reps uint32 = 1
/* Compute the run length of zeros and add the appropriate number of 0
and 17 code length codes to the code length code histogram. */
var k uint
for k = i + 1; k < data_size && histogram.data_[k] == 0; k++ {
reps++
}
i += uint(reps)
if i == data_size {
/* Don't add any cost for the last zero run, since these are encoded
only implicitly. */
break
}
if reps < 3 {
depth_histo[0] += reps
} else {
reps -= 2
for reps > 0 {
depth_histo[repeatZeroCodeLength]++
/* Add the 3 extra bits for the 17 code length code. */
bits += 3
reps >>= 3
}
}
}
}
/* Add the estimated encoding cost of the code length code histogram. */
bits += float64(18 + 2*max_depth)
/* Add the entropy of the code length code histogram. */
bits += bitsEntropy(depth_histo[:], codeLengthCodes)
}
return bits
}
func populationCostDistance(histogram *histogramDistance) float64 {
var data_size uint = histogramDataSizeDistance()
var count int = 0
var s [5]uint
var bits float64 = 0.0
var i uint
if histogram.total_count_ == 0 {
return kOneSymbolHistogramCost
}
for i = 0; i < data_size; i++ {
if histogram.data_[i] > 0 {
s[count] = i
count++
if count > 4 {
break
}
}
}
if count == 1 {
return kOneSymbolHistogramCost
}
if count == 2 {
return kTwoSymbolHistogramCost + float64(histogram.total_count_)
}
if count == 3 {
var histo0 uint32 = histogram.data_[s[0]]
var histo1 uint32 = histogram.data_[s[1]]
var histo2 uint32 = histogram.data_[s[2]]
var histomax uint32 = brotli_max_uint32_t(histo0, brotli_max_uint32_t(histo1, histo2))
return kThreeSymbolHistogramCost + 2*(float64(histo0)+float64(histo1)+float64(histo2)) - float64(histomax)
}
if count == 4 {
var histo [4]uint32
var h23 uint32
var histomax uint32
for i = 0; i < 4; i++ {
histo[i] = histogram.data_[s[i]]
}
/* Sort */
for i = 0; i < 4; i++ {
var j uint
for j = i + 1; j < 4; j++ {
if histo[j] > histo[i] {
var tmp uint32 = histo[j]
histo[j] = histo[i]
histo[i] = tmp
}
}
}
h23 = histo[2] + histo[3]
histomax = brotli_max_uint32_t(h23, histo[0])
return kFourSymbolHistogramCost + 3*float64(h23) + 2*(float64(histo[0])+float64(histo[1])) - float64(histomax)
}
{
var max_depth uint = 1
var depth_histo = [codeLengthCodes]uint32{0}
/* In this loop we compute the entropy of the histogram and simultaneously
build a simplified histogram of the code length codes where we use the
zero repeat code 17, but we don't use the non-zero repeat code 16. */
var log2total float64 = fastLog2(histogram.total_count_)
for i = 0; i < data_size; {
if histogram.data_[i] > 0 {
var log2p float64 = log2total - fastLog2(uint(histogram.data_[i]))
/* Compute -log2(P(symbol)) = -log2(count(symbol)/total_count) =
= log2(total_count) - log2(count(symbol)) */
var depth uint = uint(log2p + 0.5)
/* Approximate the bit depth by round(-log2(P(symbol))) */
bits += float64(histogram.data_[i]) * log2p
if depth > 15 {
depth = 15
}
if depth > max_depth {
max_depth = depth
}
depth_histo[depth]++
i++
} else {
var reps uint32 = 1
/* Compute the run length of zeros and add the appropriate number of 0
and 17 code length codes to the code length code histogram. */
var k uint
for k = i + 1; k < data_size && histogram.data_[k] == 0; k++ {
reps++
}
i += uint(reps)
if i == data_size {
/* Don't add any cost for the last zero run, since these are encoded
only implicitly. */
break
}
if reps < 3 {
depth_histo[0] += reps
} else {
reps -= 2
for reps > 0 {
depth_histo[repeatZeroCodeLength]++
/* Add the 3 extra bits for the 17 code length code. */
bits += 3
reps >>= 3
}
}
}
}
/* Add the estimated encoding cost of the code length code histogram. */
bits += float64(18 + 2*max_depth)
/* Add the entropy of the code length code histogram. */
bits += bitsEntropy(depth_histo[:], codeLengthCodes)
}
return bits
}
+266
View File
@@ -0,0 +1,266 @@
package brotli
import "encoding/binary"
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Bit reading helpers */
const shortFillBitWindowRead = (8 >> 1)
var kBitMask = [33]uint32{
0x00000000,
0x00000001,
0x00000003,
0x00000007,
0x0000000F,
0x0000001F,
0x0000003F,
0x0000007F,
0x000000FF,
0x000001FF,
0x000003FF,
0x000007FF,
0x00000FFF,
0x00001FFF,
0x00003FFF,
0x00007FFF,
0x0000FFFF,
0x0001FFFF,
0x0003FFFF,
0x0007FFFF,
0x000FFFFF,
0x001FFFFF,
0x003FFFFF,
0x007FFFFF,
0x00FFFFFF,
0x01FFFFFF,
0x03FFFFFF,
0x07FFFFFF,
0x0FFFFFFF,
0x1FFFFFFF,
0x3FFFFFFF,
0x7FFFFFFF,
0xFFFFFFFF,
}
func bitMask(n uint32) uint32 {
return kBitMask[n]
}
type bitReader struct {
val_ uint64
bit_pos_ uint32
input []byte
input_len uint
byte_pos uint
}
type bitReaderState struct {
val_ uint64
bit_pos_ uint32
input []byte
input_len uint
byte_pos uint
}
/* Initializes the BrotliBitReader fields. */
/* Ensures that accumulator is not empty.
May consume up to sizeof(brotli_reg_t) - 1 bytes of input.
Returns false if data is required but there is no input available.
For BROTLI_ALIGNED_READ this function also prepares bit reader for aligned
reading. */
func bitReaderSaveState(from *bitReader, to *bitReaderState) {
to.val_ = from.val_
to.bit_pos_ = from.bit_pos_
to.input = from.input
to.input_len = from.input_len
to.byte_pos = from.byte_pos
}
func bitReaderRestoreState(to *bitReader, from *bitReaderState) {
to.val_ = from.val_
to.bit_pos_ = from.bit_pos_
to.input = from.input
to.input_len = from.input_len
to.byte_pos = from.byte_pos
}
func getAvailableBits(br *bitReader) uint32 {
return 64 - br.bit_pos_
}
/* Returns amount of unread bytes the bit reader still has buffered from the
BrotliInput, including whole bytes in br->val_. */
func getRemainingBytes(br *bitReader) uint {
return uint(uint32(br.input_len-br.byte_pos) + (getAvailableBits(br) >> 3))
}
/* Checks if there is at least |num| bytes left in the input ring-buffer
(excluding the bits remaining in br->val_). */
func checkInputAmount(br *bitReader, num uint) bool {
return br.input_len-br.byte_pos >= num
}
/* Guarantees that there are at least |n_bits| + 1 bits in accumulator.
Precondition: accumulator contains at least 1 bit.
|n_bits| should be in the range [1..24] for regular build. For portable
non-64-bit little-endian build only 16 bits are safe to request. */
func fillBitWindow(br *bitReader, n_bits uint32) {
if br.bit_pos_ >= 32 {
br.val_ >>= 32
br.bit_pos_ ^= 32 /* here same as -= 32 because of the if condition */
br.val_ |= (uint64(binary.LittleEndian.Uint32(br.input[br.byte_pos:]))) << 32
br.byte_pos += 4
}
}
/* Mostly like BrotliFillBitWindow, but guarantees only 16 bits and reads no
more than BROTLI_SHORT_FILL_BIT_WINDOW_READ bytes of input. */
func fillBitWindow16(br *bitReader) {
fillBitWindow(br, 17)
}
/* Tries to pull one byte of input to accumulator.
Returns false if there is no input available. */
func pullByte(br *bitReader) bool {
if br.byte_pos == br.input_len {
return false
}
br.val_ >>= 8
br.val_ |= (uint64(br.input[br.byte_pos])) << 56
br.bit_pos_ -= 8
br.byte_pos++
return true
}
/* Returns currently available bits.
The number of valid bits could be calculated by BrotliGetAvailableBits. */
func getBitsUnmasked(br *bitReader) uint64 {
return br.val_ >> br.bit_pos_
}
/* Like BrotliGetBits, but does not mask the result.
The result contains at least 16 valid bits. */
func get16BitsUnmasked(br *bitReader) uint32 {
fillBitWindow(br, 16)
return uint32(getBitsUnmasked(br))
}
/* Returns the specified number of bits from |br| without advancing bit
position. */
func getBits(br *bitReader, n_bits uint32) uint32 {
fillBitWindow(br, n_bits)
return uint32(getBitsUnmasked(br)) & bitMask(n_bits)
}
/* Tries to peek the specified amount of bits. Returns false, if there
is not enough input. */
func safeGetBits(br *bitReader, n_bits uint32, val *uint32) bool {
for getAvailableBits(br) < n_bits {
if !pullByte(br) {
return false
}
}
*val = uint32(getBitsUnmasked(br)) & bitMask(n_bits)
return true
}
/* Advances the bit pos by |n_bits|. */
func dropBits(br *bitReader, n_bits uint32) {
br.bit_pos_ += n_bits
}
func bitReaderUnload(br *bitReader) {
var unused_bytes uint32 = getAvailableBits(br) >> 3
var unused_bits uint32 = unused_bytes << 3
br.byte_pos -= uint(unused_bytes)
if unused_bits == 64 {
br.val_ = 0
} else {
br.val_ <<= unused_bits
}
br.bit_pos_ += unused_bits
}
/* Reads the specified number of bits from |br| and advances the bit pos.
Precondition: accumulator MUST contain at least |n_bits|. */
func takeBits(br *bitReader, n_bits uint32, val *uint32) {
*val = uint32(getBitsUnmasked(br)) & bitMask(n_bits)
dropBits(br, n_bits)
}
/* Reads the specified number of bits from |br| and advances the bit pos.
Assumes that there is enough input to perform BrotliFillBitWindow. */
func readBits(br *bitReader, n_bits uint32) uint32 {
var val uint32
fillBitWindow(br, n_bits)
takeBits(br, n_bits, &val)
return val
}
/* Tries to read the specified amount of bits. Returns false, if there
is not enough input. |n_bits| MUST be positive. */
func safeReadBits(br *bitReader, n_bits uint32, val *uint32) bool {
for getAvailableBits(br) < n_bits {
if !pullByte(br) {
return false
}
}
takeBits(br, n_bits, val)
return true
}
/* Advances the bit reader position to the next byte boundary and verifies
that any skipped bits are set to zero. */
func bitReaderJumpToByteBoundary(br *bitReader) bool {
var pad_bits_count uint32 = getAvailableBits(br) & 0x7
var pad_bits uint32 = 0
if pad_bits_count != 0 {
takeBits(br, pad_bits_count, &pad_bits)
}
return pad_bits == 0
}
/* Copies remaining input bytes stored in the bit reader to the output. Value
|num| may not be larger than BrotliGetRemainingBytes. The bit reader must be
warmed up again after this. */
func copyBytes(dest []byte, br *bitReader, num uint) {
for getAvailableBits(br) >= 8 && num > 0 {
dest[0] = byte(getBitsUnmasked(br))
dropBits(br, 8)
dest = dest[1:]
num--
}
copy(dest, br.input[br.byte_pos:][:num])
br.byte_pos += num
}
func initBitReader(br *bitReader) {
br.val_ = 0
br.bit_pos_ = 64
}
func warmupBitReader(br *bitReader) bool {
/* Fixing alignment after unaligned BrotliFillWindow would result accumulator
overflow. If unalignment is caused by BrotliSafeReadBits, then there is
enough space in accumulator to fix alignment. */
if getAvailableBits(br) == 0 {
if !pullByte(br) {
return false
}
}
return true
}
+56
View File
@@ -0,0 +1,56 @@
package brotli
/* Copyright 2010 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Write bits into a byte array. */
type bitWriter struct {
dst []byte
// Data waiting to be written is the low nbits of bits.
bits uint64
nbits uint
}
func (w *bitWriter) writeBits(nb uint, b uint64) {
w.bits |= b << w.nbits
w.nbits += nb
if w.nbits >= 32 {
bits := w.bits
w.bits >>= 32
w.nbits -= 32
w.dst = append(w.dst,
byte(bits),
byte(bits>>8),
byte(bits>>16),
byte(bits>>24),
)
}
}
func (w *bitWriter) writeSingleBit(bit bool) {
if bit {
w.writeBits(1, 1)
} else {
w.writeBits(1, 0)
}
}
func (w *bitWriter) jumpToByteBoundary() {
dst := w.dst
for w.nbits != 0 {
dst = append(dst, byte(w.bits))
w.bits >>= 8
if w.nbits > 8 { // Avoid underflow
w.nbits -= 8
} else {
w.nbits = 0
}
}
w.bits = 0
w.dst = dst
}
+144
View File
@@ -0,0 +1,144 @@
package brotli
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Block split point selection utilities. */
type blockSplit struct {
num_types uint
num_blocks uint
types []byte
lengths []uint32
types_alloc_size uint
lengths_alloc_size uint
}
const (
kMaxLiteralHistograms uint = 100
kMaxCommandHistograms uint = 50
kLiteralBlockSwitchCost float64 = 28.1
kCommandBlockSwitchCost float64 = 13.5
kDistanceBlockSwitchCost float64 = 14.6
kLiteralStrideLength uint = 70
kCommandStrideLength uint = 40
kSymbolsPerLiteralHistogram uint = 544
kSymbolsPerCommandHistogram uint = 530
kSymbolsPerDistanceHistogram uint = 544
kMinLengthForBlockSplitting uint = 128
kIterMulForRefining uint = 2
kMinItersForRefining uint = 100
)
func countLiterals(cmds []command) uint {
var total_length uint = 0
/* Count how many we have. */
for i := range cmds {
total_length += uint(cmds[i].insert_len_)
}
return total_length
}
func copyLiteralsToByteArray(cmds []command, data []byte, offset uint, mask uint, literals []byte) {
var pos uint = 0
var from_pos uint = offset & mask
for i := range cmds {
var insert_len uint = uint(cmds[i].insert_len_)
if from_pos+insert_len > mask {
var head_size uint = mask + 1 - from_pos
copy(literals[pos:], data[from_pos:][:head_size])
from_pos = 0
pos += head_size
insert_len -= head_size
}
if insert_len > 0 {
copy(literals[pos:], data[from_pos:][:insert_len])
pos += insert_len
}
from_pos = uint((uint32(from_pos+insert_len) + commandCopyLen(&cmds[i])) & uint32(mask))
}
}
func myRand(seed *uint32) uint32 {
/* Initial seed should be 7. In this case, loop length is (1 << 29). */
*seed *= 16807
return *seed
}
func bitCost(count uint) float64 {
if count == 0 {
return -2.0
} else {
return fastLog2(count)
}
}
const histogramsPerBatch = 64
const clustersPerBatch = 16
func initBlockSplit(self *blockSplit) {
self.num_types = 0
self.num_blocks = 0
self.types = self.types[:0]
self.lengths = self.lengths[:0]
self.types_alloc_size = 0
self.lengths_alloc_size = 0
}
func splitBlock(cmds []command, data []byte, pos uint, mask uint, params *encoderParams, literal_split *blockSplit, insert_and_copy_split *blockSplit, dist_split *blockSplit) {
{
var literals_count uint = countLiterals(cmds)
var literals []byte = make([]byte, literals_count)
/* Create a continuous array of literals. */
copyLiteralsToByteArray(cmds, data, pos, mask, literals)
/* Create the block split on the array of literals.
Literal histograms have alphabet size 256. */
splitByteVectorLiteral(literals, literals_count, kSymbolsPerLiteralHistogram, kMaxLiteralHistograms, kLiteralStrideLength, kLiteralBlockSwitchCost, params, literal_split)
literals = nil
}
{
var insert_and_copy_codes []uint16 = make([]uint16, len(cmds))
/* Compute prefix codes for commands. */
for i := range cmds {
insert_and_copy_codes[i] = cmds[i].cmd_prefix_
}
/* Create the block split on the array of command prefixes. */
splitByteVectorCommand(insert_and_copy_codes, kSymbolsPerCommandHistogram, kMaxCommandHistograms, kCommandStrideLength, kCommandBlockSwitchCost, params, insert_and_copy_split)
/* TODO: reuse for distances? */
insert_and_copy_codes = nil
}
{
var distance_prefixes []uint16 = make([]uint16, len(cmds))
var j uint = 0
/* Create a continuous array of distance prefixes. */
for i := range cmds {
var cmd *command = &cmds[i]
if commandCopyLen(cmd) != 0 && cmd.cmd_prefix_ >= 128 {
distance_prefixes[j] = cmd.dist_prefix_ & 0x3FF
j++
}
}
/* Create the block split on the array of distance prefixes. */
splitByteVectorDistance(distance_prefixes, j, kSymbolsPerDistanceHistogram, kMaxCommandHistograms, kCommandStrideLength, kDistanceBlockSwitchCost, params, dist_split)
distance_prefixes = nil
}
}
+434
View File
@@ -0,0 +1,434 @@
package brotli
import "math"
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
func initialEntropyCodesCommand(data []uint16, length uint, stride uint, num_histograms uint, histograms []histogramCommand) {
var seed uint32 = 7
var block_length uint = length / num_histograms
var i uint
clearHistogramsCommand(histograms, num_histograms)
for i = 0; i < num_histograms; i++ {
var pos uint = length * i / num_histograms
if i != 0 {
pos += uint(myRand(&seed) % uint32(block_length))
}
if pos+stride >= length {
pos = length - stride - 1
}
histogramAddVectorCommand(&histograms[i], data[pos:], stride)
}
}
func randomSampleCommand(seed *uint32, data []uint16, length uint, stride uint, sample *histogramCommand) {
var pos uint = 0
if stride >= length {
stride = length
} else {
pos = uint(myRand(seed) % uint32(length-stride+1))
}
histogramAddVectorCommand(sample, data[pos:], stride)
}
func refineEntropyCodesCommand(data []uint16, length uint, stride uint, num_histograms uint, histograms []histogramCommand) {
var iters uint = kIterMulForRefining*length/stride + kMinItersForRefining
var seed uint32 = 7
var iter uint
iters = ((iters + num_histograms - 1) / num_histograms) * num_histograms
for iter = 0; iter < iters; iter++ {
var sample histogramCommand
histogramClearCommand(&sample)
randomSampleCommand(&seed, data, length, stride, &sample)
histogramAddHistogramCommand(&histograms[iter%num_histograms], &sample)
}
}
/* Assigns a block id from the range [0, num_histograms) to each data element
in data[0..length) and fills in block_id[0..length) with the assigned values.
Returns the number of blocks, i.e. one plus the number of block switches. */
func findBlocksCommand(data []uint16, length uint, block_switch_bitcost float64, num_histograms uint, histograms []histogramCommand, insert_cost []float64, cost []float64, switch_signal []byte, block_id []byte) uint {
var data_size uint = histogramDataSizeCommand()
var bitmaplen uint = (num_histograms + 7) >> 3
var num_blocks uint = 1
var i uint
var j uint
assert(num_histograms <= 256)
if num_histograms <= 1 {
for i = 0; i < length; i++ {
block_id[i] = 0
}
return 1
}
for i := 0; i < int(data_size*num_histograms); i++ {
insert_cost[i] = 0
}
for i = 0; i < num_histograms; i++ {
insert_cost[i] = fastLog2(uint(uint32(histograms[i].total_count_)))
}
for i = data_size; i != 0; {
i--
for j = 0; j < num_histograms; j++ {
insert_cost[i*num_histograms+j] = insert_cost[j] - bitCost(uint(histograms[j].data_[i]))
}
}
for i := 0; i < int(num_histograms); i++ {
cost[i] = 0
}
for i := 0; i < int(length*bitmaplen); i++ {
switch_signal[i] = 0
}
/* After each iteration of this loop, cost[k] will contain the difference
between the minimum cost of arriving at the current byte position using
entropy code k, and the minimum cost of arriving at the current byte
position. This difference is capped at the block switch cost, and if it
reaches block switch cost, it means that when we trace back from the last
position, we need to switch here. */
for i = 0; i < length; i++ {
var byte_ix uint = i
var ix uint = byte_ix * bitmaplen
var insert_cost_ix uint = uint(data[byte_ix]) * num_histograms
var min_cost float64 = 1e99
var block_switch_cost float64 = block_switch_bitcost
var k uint
for k = 0; k < num_histograms; k++ {
/* We are coding the symbol in data[byte_ix] with entropy code k. */
cost[k] += insert_cost[insert_cost_ix+k]
if cost[k] < min_cost {
min_cost = cost[k]
block_id[byte_ix] = byte(k)
}
}
/* More blocks for the beginning. */
if byte_ix < 2000 {
block_switch_cost *= 0.77 + 0.07*float64(byte_ix)/2000
}
for k = 0; k < num_histograms; k++ {
cost[k] -= min_cost
if cost[k] >= block_switch_cost {
var mask byte = byte(1 << (k & 7))
cost[k] = block_switch_cost
assert(k>>3 < bitmaplen)
switch_signal[ix+(k>>3)] |= mask
/* Trace back from the last position and switch at the marked places. */
}
}
}
{
var byte_ix uint = length - 1
var ix uint = byte_ix * bitmaplen
var cur_id byte = block_id[byte_ix]
for byte_ix > 0 {
var mask byte = byte(1 << (cur_id & 7))
assert(uint(cur_id)>>3 < bitmaplen)
byte_ix--
ix -= bitmaplen
if switch_signal[ix+uint(cur_id>>3)]&mask != 0 {
if cur_id != block_id[byte_ix] {
cur_id = block_id[byte_ix]
num_blocks++
}
}
block_id[byte_ix] = cur_id
}
}
return num_blocks
}
var remapBlockIdsCommand_kInvalidId uint16 = 256
func remapBlockIdsCommand(block_ids []byte, length uint, new_id []uint16, num_histograms uint) uint {
var next_id uint16 = 0
var i uint
for i = 0; i < num_histograms; i++ {
new_id[i] = remapBlockIdsCommand_kInvalidId
}
for i = 0; i < length; i++ {
assert(uint(block_ids[i]) < num_histograms)
if new_id[block_ids[i]] == remapBlockIdsCommand_kInvalidId {
new_id[block_ids[i]] = next_id
next_id++
}
}
for i = 0; i < length; i++ {
block_ids[i] = byte(new_id[block_ids[i]])
assert(uint(block_ids[i]) < num_histograms)
}
assert(uint(next_id) <= num_histograms)
return uint(next_id)
}
func buildBlockHistogramsCommand(data []uint16, length uint, block_ids []byte, num_histograms uint, histograms []histogramCommand) {
var i uint
clearHistogramsCommand(histograms, num_histograms)
for i = 0; i < length; i++ {
histogramAddCommand(&histograms[block_ids[i]], uint(data[i]))
}
}
var clusterBlocksCommand_kInvalidIndex uint32 = math.MaxUint32
func clusterBlocksCommand(data []uint16, length uint, num_blocks uint, block_ids []byte, split *blockSplit) {
var histogram_symbols []uint32 = make([]uint32, num_blocks)
var block_lengths []uint32 = make([]uint32, num_blocks)
var expected_num_clusters uint = clustersPerBatch * (num_blocks + histogramsPerBatch - 1) / histogramsPerBatch
var all_histograms_size uint = 0
var all_histograms_capacity uint = expected_num_clusters
var all_histograms []histogramCommand = make([]histogramCommand, all_histograms_capacity)
var cluster_size_size uint = 0
var cluster_size_capacity uint = expected_num_clusters
var cluster_size []uint32 = make([]uint32, cluster_size_capacity)
var num_clusters uint = 0
var histograms []histogramCommand = make([]histogramCommand, brotli_min_size_t(num_blocks, histogramsPerBatch))
var max_num_pairs uint = histogramsPerBatch * histogramsPerBatch / 2
var pairs_capacity uint = max_num_pairs + 1
var pairs []histogramPair = make([]histogramPair, pairs_capacity)
var pos uint = 0
var clusters []uint32
var num_final_clusters uint
var new_index []uint32
var i uint
var sizes = [histogramsPerBatch]uint32{0}
var new_clusters = [histogramsPerBatch]uint32{0}
var symbols = [histogramsPerBatch]uint32{0}
var remap = [histogramsPerBatch]uint32{0}
for i := 0; i < int(num_blocks); i++ {
block_lengths[i] = 0
}
{
var block_idx uint = 0
for i = 0; i < length; i++ {
assert(block_idx < num_blocks)
block_lengths[block_idx]++
if i+1 == length || block_ids[i] != block_ids[i+1] {
block_idx++
}
}
assert(block_idx == num_blocks)
}
for i = 0; i < num_blocks; i += histogramsPerBatch {
var num_to_combine uint = brotli_min_size_t(num_blocks-i, histogramsPerBatch)
var num_new_clusters uint
var j uint
for j = 0; j < num_to_combine; j++ {
var k uint
histogramClearCommand(&histograms[j])
for k = 0; uint32(k) < block_lengths[i+j]; k++ {
histogramAddCommand(&histograms[j], uint(data[pos]))
pos++
}
histograms[j].bit_cost_ = populationCostCommand(&histograms[j])
new_clusters[j] = uint32(j)
symbols[j] = uint32(j)
sizes[j] = 1
}
num_new_clusters = histogramCombineCommand(histograms, sizes[:], symbols[:], new_clusters[:], []histogramPair(pairs), num_to_combine, num_to_combine, histogramsPerBatch, max_num_pairs)
if all_histograms_capacity < (all_histograms_size + num_new_clusters) {
var _new_size uint
if all_histograms_capacity == 0 {
_new_size = all_histograms_size + num_new_clusters
} else {
_new_size = all_histograms_capacity
}
var new_array []histogramCommand
for _new_size < (all_histograms_size + num_new_clusters) {
_new_size *= 2
}
new_array = make([]histogramCommand, _new_size)
if all_histograms_capacity != 0 {
copy(new_array, all_histograms[:all_histograms_capacity])
}
all_histograms = new_array
all_histograms_capacity = _new_size
}
brotli_ensure_capacity_uint32_t(&cluster_size, &cluster_size_capacity, cluster_size_size+num_new_clusters)
for j = 0; j < num_new_clusters; j++ {
all_histograms[all_histograms_size] = histograms[new_clusters[j]]
all_histograms_size++
cluster_size[cluster_size_size] = sizes[new_clusters[j]]
cluster_size_size++
remap[new_clusters[j]] = uint32(j)
}
for j = 0; j < num_to_combine; j++ {
histogram_symbols[i+j] = uint32(num_clusters) + remap[symbols[j]]
}
num_clusters += num_new_clusters
assert(num_clusters == cluster_size_size)
assert(num_clusters == all_histograms_size)
}
histograms = nil
max_num_pairs = brotli_min_size_t(64*num_clusters, (num_clusters/2)*num_clusters)
if pairs_capacity < max_num_pairs+1 {
pairs = nil
pairs = make([]histogramPair, (max_num_pairs + 1))
}
clusters = make([]uint32, num_clusters)
for i = 0; i < num_clusters; i++ {
clusters[i] = uint32(i)
}
num_final_clusters = histogramCombineCommand(all_histograms, cluster_size, histogram_symbols, clusters, pairs, num_clusters, num_blocks, maxNumberOfBlockTypes, max_num_pairs)
pairs = nil
cluster_size = nil
new_index = make([]uint32, num_clusters)
for i = 0; i < num_clusters; i++ {
new_index[i] = clusterBlocksCommand_kInvalidIndex
}
pos = 0
{
var next_index uint32 = 0
for i = 0; i < num_blocks; i++ {
var histo histogramCommand
var j uint
var best_out uint32
var best_bits float64
histogramClearCommand(&histo)
for j = 0; uint32(j) < block_lengths[i]; j++ {
histogramAddCommand(&histo, uint(data[pos]))
pos++
}
if i == 0 {
best_out = histogram_symbols[0]
} else {
best_out = histogram_symbols[i-1]
}
best_bits = histogramBitCostDistanceCommand(&histo, &all_histograms[best_out])
for j = 0; j < num_final_clusters; j++ {
var cur_bits float64 = histogramBitCostDistanceCommand(&histo, &all_histograms[clusters[j]])
if cur_bits < best_bits {
best_bits = cur_bits
best_out = clusters[j]
}
}
histogram_symbols[i] = best_out
if new_index[best_out] == clusterBlocksCommand_kInvalidIndex {
new_index[best_out] = next_index
next_index++
}
}
}
clusters = nil
all_histograms = nil
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, num_blocks)
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, num_blocks)
{
var cur_length uint32 = 0
var block_idx uint = 0
var max_type byte = 0
for i = 0; i < num_blocks; i++ {
cur_length += block_lengths[i]
if i+1 == num_blocks || histogram_symbols[i] != histogram_symbols[i+1] {
var id byte = byte(new_index[histogram_symbols[i]])
split.types[block_idx] = id
split.lengths[block_idx] = cur_length
max_type = brotli_max_uint8_t(max_type, id)
cur_length = 0
block_idx++
}
}
split.num_blocks = block_idx
split.num_types = uint(max_type) + 1
}
new_index = nil
block_lengths = nil
histogram_symbols = nil
}
func splitByteVectorCommand(data []uint16, literals_per_histogram uint, max_histograms uint, sampling_stride_length uint, block_switch_cost float64, params *encoderParams, split *blockSplit) {
length := uint(len(data))
var data_size uint = histogramDataSizeCommand()
var num_histograms uint = length/literals_per_histogram + 1
var histograms []histogramCommand
if num_histograms > max_histograms {
num_histograms = max_histograms
}
if length == 0 {
split.num_types = 1
return
} else if length < kMinLengthForBlockSplitting {
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, split.num_blocks+1)
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, split.num_blocks+1)
split.num_types = 1
split.types[split.num_blocks] = 0
split.lengths[split.num_blocks] = uint32(length)
split.num_blocks++
return
}
histograms = make([]histogramCommand, num_histograms)
/* Find good entropy codes. */
initialEntropyCodesCommand(data, length, sampling_stride_length, num_histograms, histograms)
refineEntropyCodesCommand(data, length, sampling_stride_length, num_histograms, histograms)
{
var block_ids []byte = make([]byte, length)
var num_blocks uint = 0
var bitmaplen uint = (num_histograms + 7) >> 3
var insert_cost []float64 = make([]float64, (data_size * num_histograms))
var cost []float64 = make([]float64, num_histograms)
var switch_signal []byte = make([]byte, (length * bitmaplen))
var new_id []uint16 = make([]uint16, num_histograms)
var iters uint
if params.quality < hqZopflificationQuality {
iters = 3
} else {
iters = 10
}
/* Find a good path through literals with the good entropy codes. */
var i uint
for i = 0; i < iters; i++ {
num_blocks = findBlocksCommand(data, length, block_switch_cost, num_histograms, histograms, insert_cost, cost, switch_signal, block_ids)
num_histograms = remapBlockIdsCommand(block_ids, length, new_id, num_histograms)
buildBlockHistogramsCommand(data, length, block_ids, num_histograms, histograms)
}
insert_cost = nil
cost = nil
switch_signal = nil
new_id = nil
histograms = nil
clusterBlocksCommand(data, length, num_blocks, block_ids, split)
block_ids = nil
}
}
+433
View File
@@ -0,0 +1,433 @@
package brotli
import "math"
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
func initialEntropyCodesDistance(data []uint16, length uint, stride uint, num_histograms uint, histograms []histogramDistance) {
var seed uint32 = 7
var block_length uint = length / num_histograms
var i uint
clearHistogramsDistance(histograms, num_histograms)
for i = 0; i < num_histograms; i++ {
var pos uint = length * i / num_histograms
if i != 0 {
pos += uint(myRand(&seed) % uint32(block_length))
}
if pos+stride >= length {
pos = length - stride - 1
}
histogramAddVectorDistance(&histograms[i], data[pos:], stride)
}
}
func randomSampleDistance(seed *uint32, data []uint16, length uint, stride uint, sample *histogramDistance) {
var pos uint = 0
if stride >= length {
stride = length
} else {
pos = uint(myRand(seed) % uint32(length-stride+1))
}
histogramAddVectorDistance(sample, data[pos:], stride)
}
func refineEntropyCodesDistance(data []uint16, length uint, stride uint, num_histograms uint, histograms []histogramDistance) {
var iters uint = kIterMulForRefining*length/stride + kMinItersForRefining
var seed uint32 = 7
var iter uint
iters = ((iters + num_histograms - 1) / num_histograms) * num_histograms
for iter = 0; iter < iters; iter++ {
var sample histogramDistance
histogramClearDistance(&sample)
randomSampleDistance(&seed, data, length, stride, &sample)
histogramAddHistogramDistance(&histograms[iter%num_histograms], &sample)
}
}
/* Assigns a block id from the range [0, num_histograms) to each data element
in data[0..length) and fills in block_id[0..length) with the assigned values.
Returns the number of blocks, i.e. one plus the number of block switches. */
func findBlocksDistance(data []uint16, length uint, block_switch_bitcost float64, num_histograms uint, histograms []histogramDistance, insert_cost []float64, cost []float64, switch_signal []byte, block_id []byte) uint {
var data_size uint = histogramDataSizeDistance()
var bitmaplen uint = (num_histograms + 7) >> 3
var num_blocks uint = 1
var i uint
var j uint
assert(num_histograms <= 256)
if num_histograms <= 1 {
for i = 0; i < length; i++ {
block_id[i] = 0
}
return 1
}
for i := 0; i < int(data_size*num_histograms); i++ {
insert_cost[i] = 0
}
for i = 0; i < num_histograms; i++ {
insert_cost[i] = fastLog2(uint(uint32(histograms[i].total_count_)))
}
for i = data_size; i != 0; {
i--
for j = 0; j < num_histograms; j++ {
insert_cost[i*num_histograms+j] = insert_cost[j] - bitCost(uint(histograms[j].data_[i]))
}
}
for i := 0; i < int(num_histograms); i++ {
cost[i] = 0
}
for i := 0; i < int(length*bitmaplen); i++ {
switch_signal[i] = 0
}
/* After each iteration of this loop, cost[k] will contain the difference
between the minimum cost of arriving at the current byte position using
entropy code k, and the minimum cost of arriving at the current byte
position. This difference is capped at the block switch cost, and if it
reaches block switch cost, it means that when we trace back from the last
position, we need to switch here. */
for i = 0; i < length; i++ {
var byte_ix uint = i
var ix uint = byte_ix * bitmaplen
var insert_cost_ix uint = uint(data[byte_ix]) * num_histograms
var min_cost float64 = 1e99
var block_switch_cost float64 = block_switch_bitcost
var k uint
for k = 0; k < num_histograms; k++ {
/* We are coding the symbol in data[byte_ix] with entropy code k. */
cost[k] += insert_cost[insert_cost_ix+k]
if cost[k] < min_cost {
min_cost = cost[k]
block_id[byte_ix] = byte(k)
}
}
/* More blocks for the beginning. */
if byte_ix < 2000 {
block_switch_cost *= 0.77 + 0.07*float64(byte_ix)/2000
}
for k = 0; k < num_histograms; k++ {
cost[k] -= min_cost
if cost[k] >= block_switch_cost {
var mask byte = byte(1 << (k & 7))
cost[k] = block_switch_cost
assert(k>>3 < bitmaplen)
switch_signal[ix+(k>>3)] |= mask
/* Trace back from the last position and switch at the marked places. */
}
}
}
{
var byte_ix uint = length - 1
var ix uint = byte_ix * bitmaplen
var cur_id byte = block_id[byte_ix]
for byte_ix > 0 {
var mask byte = byte(1 << (cur_id & 7))
assert(uint(cur_id)>>3 < bitmaplen)
byte_ix--
ix -= bitmaplen
if switch_signal[ix+uint(cur_id>>3)]&mask != 0 {
if cur_id != block_id[byte_ix] {
cur_id = block_id[byte_ix]
num_blocks++
}
}
block_id[byte_ix] = cur_id
}
}
return num_blocks
}
var remapBlockIdsDistance_kInvalidId uint16 = 256
func remapBlockIdsDistance(block_ids []byte, length uint, new_id []uint16, num_histograms uint) uint {
var next_id uint16 = 0
var i uint
for i = 0; i < num_histograms; i++ {
new_id[i] = remapBlockIdsDistance_kInvalidId
}
for i = 0; i < length; i++ {
assert(uint(block_ids[i]) < num_histograms)
if new_id[block_ids[i]] == remapBlockIdsDistance_kInvalidId {
new_id[block_ids[i]] = next_id
next_id++
}
}
for i = 0; i < length; i++ {
block_ids[i] = byte(new_id[block_ids[i]])
assert(uint(block_ids[i]) < num_histograms)
}
assert(uint(next_id) <= num_histograms)
return uint(next_id)
}
func buildBlockHistogramsDistance(data []uint16, length uint, block_ids []byte, num_histograms uint, histograms []histogramDistance) {
var i uint
clearHistogramsDistance(histograms, num_histograms)
for i = 0; i < length; i++ {
histogramAddDistance(&histograms[block_ids[i]], uint(data[i]))
}
}
var clusterBlocksDistance_kInvalidIndex uint32 = math.MaxUint32
func clusterBlocksDistance(data []uint16, length uint, num_blocks uint, block_ids []byte, split *blockSplit) {
var histogram_symbols []uint32 = make([]uint32, num_blocks)
var block_lengths []uint32 = make([]uint32, num_blocks)
var expected_num_clusters uint = clustersPerBatch * (num_blocks + histogramsPerBatch - 1) / histogramsPerBatch
var all_histograms_size uint = 0
var all_histograms_capacity uint = expected_num_clusters
var all_histograms []histogramDistance = make([]histogramDistance, all_histograms_capacity)
var cluster_size_size uint = 0
var cluster_size_capacity uint = expected_num_clusters
var cluster_size []uint32 = make([]uint32, cluster_size_capacity)
var num_clusters uint = 0
var histograms []histogramDistance = make([]histogramDistance, brotli_min_size_t(num_blocks, histogramsPerBatch))
var max_num_pairs uint = histogramsPerBatch * histogramsPerBatch / 2
var pairs_capacity uint = max_num_pairs + 1
var pairs []histogramPair = make([]histogramPair, pairs_capacity)
var pos uint = 0
var clusters []uint32
var num_final_clusters uint
var new_index []uint32
var i uint
var sizes = [histogramsPerBatch]uint32{0}
var new_clusters = [histogramsPerBatch]uint32{0}
var symbols = [histogramsPerBatch]uint32{0}
var remap = [histogramsPerBatch]uint32{0}
for i := 0; i < int(num_blocks); i++ {
block_lengths[i] = 0
}
{
var block_idx uint = 0
for i = 0; i < length; i++ {
assert(block_idx < num_blocks)
block_lengths[block_idx]++
if i+1 == length || block_ids[i] != block_ids[i+1] {
block_idx++
}
}
assert(block_idx == num_blocks)
}
for i = 0; i < num_blocks; i += histogramsPerBatch {
var num_to_combine uint = brotli_min_size_t(num_blocks-i, histogramsPerBatch)
var num_new_clusters uint
var j uint
for j = 0; j < num_to_combine; j++ {
var k uint
histogramClearDistance(&histograms[j])
for k = 0; uint32(k) < block_lengths[i+j]; k++ {
histogramAddDistance(&histograms[j], uint(data[pos]))
pos++
}
histograms[j].bit_cost_ = populationCostDistance(&histograms[j])
new_clusters[j] = uint32(j)
symbols[j] = uint32(j)
sizes[j] = 1
}
num_new_clusters = histogramCombineDistance(histograms, sizes[:], symbols[:], new_clusters[:], []histogramPair(pairs), num_to_combine, num_to_combine, histogramsPerBatch, max_num_pairs)
if all_histograms_capacity < (all_histograms_size + num_new_clusters) {
var _new_size uint
if all_histograms_capacity == 0 {
_new_size = all_histograms_size + num_new_clusters
} else {
_new_size = all_histograms_capacity
}
var new_array []histogramDistance
for _new_size < (all_histograms_size + num_new_clusters) {
_new_size *= 2
}
new_array = make([]histogramDistance, _new_size)
if all_histograms_capacity != 0 {
copy(new_array, all_histograms[:all_histograms_capacity])
}
all_histograms = new_array
all_histograms_capacity = _new_size
}
brotli_ensure_capacity_uint32_t(&cluster_size, &cluster_size_capacity, cluster_size_size+num_new_clusters)
for j = 0; j < num_new_clusters; j++ {
all_histograms[all_histograms_size] = histograms[new_clusters[j]]
all_histograms_size++
cluster_size[cluster_size_size] = sizes[new_clusters[j]]
cluster_size_size++
remap[new_clusters[j]] = uint32(j)
}
for j = 0; j < num_to_combine; j++ {
histogram_symbols[i+j] = uint32(num_clusters) + remap[symbols[j]]
}
num_clusters += num_new_clusters
assert(num_clusters == cluster_size_size)
assert(num_clusters == all_histograms_size)
}
histograms = nil
max_num_pairs = brotli_min_size_t(64*num_clusters, (num_clusters/2)*num_clusters)
if pairs_capacity < max_num_pairs+1 {
pairs = nil
pairs = make([]histogramPair, (max_num_pairs + 1))
}
clusters = make([]uint32, num_clusters)
for i = 0; i < num_clusters; i++ {
clusters[i] = uint32(i)
}
num_final_clusters = histogramCombineDistance(all_histograms, cluster_size, histogram_symbols, clusters, pairs, num_clusters, num_blocks, maxNumberOfBlockTypes, max_num_pairs)
pairs = nil
cluster_size = nil
new_index = make([]uint32, num_clusters)
for i = 0; i < num_clusters; i++ {
new_index[i] = clusterBlocksDistance_kInvalidIndex
}
pos = 0
{
var next_index uint32 = 0
for i = 0; i < num_blocks; i++ {
var histo histogramDistance
var j uint
var best_out uint32
var best_bits float64
histogramClearDistance(&histo)
for j = 0; uint32(j) < block_lengths[i]; j++ {
histogramAddDistance(&histo, uint(data[pos]))
pos++
}
if i == 0 {
best_out = histogram_symbols[0]
} else {
best_out = histogram_symbols[i-1]
}
best_bits = histogramBitCostDistanceDistance(&histo, &all_histograms[best_out])
for j = 0; j < num_final_clusters; j++ {
var cur_bits float64 = histogramBitCostDistanceDistance(&histo, &all_histograms[clusters[j]])
if cur_bits < best_bits {
best_bits = cur_bits
best_out = clusters[j]
}
}
histogram_symbols[i] = best_out
if new_index[best_out] == clusterBlocksDistance_kInvalidIndex {
new_index[best_out] = next_index
next_index++
}
}
}
clusters = nil
all_histograms = nil
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, num_blocks)
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, num_blocks)
{
var cur_length uint32 = 0
var block_idx uint = 0
var max_type byte = 0
for i = 0; i < num_blocks; i++ {
cur_length += block_lengths[i]
if i+1 == num_blocks || histogram_symbols[i] != histogram_symbols[i+1] {
var id byte = byte(new_index[histogram_symbols[i]])
split.types[block_idx] = id
split.lengths[block_idx] = cur_length
max_type = brotli_max_uint8_t(max_type, id)
cur_length = 0
block_idx++
}
}
split.num_blocks = block_idx
split.num_types = uint(max_type) + 1
}
new_index = nil
block_lengths = nil
histogram_symbols = nil
}
func splitByteVectorDistance(data []uint16, length uint, literals_per_histogram uint, max_histograms uint, sampling_stride_length uint, block_switch_cost float64, params *encoderParams, split *blockSplit) {
var data_size uint = histogramDataSizeDistance()
var num_histograms uint = length/literals_per_histogram + 1
var histograms []histogramDistance
if num_histograms > max_histograms {
num_histograms = max_histograms
}
if length == 0 {
split.num_types = 1
return
} else if length < kMinLengthForBlockSplitting {
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, split.num_blocks+1)
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, split.num_blocks+1)
split.num_types = 1
split.types[split.num_blocks] = 0
split.lengths[split.num_blocks] = uint32(length)
split.num_blocks++
return
}
histograms = make([]histogramDistance, num_histograms)
/* Find good entropy codes. */
initialEntropyCodesDistance(data, length, sampling_stride_length, num_histograms, histograms)
refineEntropyCodesDistance(data, length, sampling_stride_length, num_histograms, histograms)
{
var block_ids []byte = make([]byte, length)
var num_blocks uint = 0
var bitmaplen uint = (num_histograms + 7) >> 3
var insert_cost []float64 = make([]float64, (data_size * num_histograms))
var cost []float64 = make([]float64, num_histograms)
var switch_signal []byte = make([]byte, (length * bitmaplen))
var new_id []uint16 = make([]uint16, num_histograms)
var iters uint
if params.quality < hqZopflificationQuality {
iters = 3
} else {
iters = 10
}
/* Find a good path through literals with the good entropy codes. */
var i uint
for i = 0; i < iters; i++ {
num_blocks = findBlocksDistance(data, length, block_switch_cost, num_histograms, histograms, insert_cost, cost, switch_signal, block_ids)
num_histograms = remapBlockIdsDistance(block_ids, length, new_id, num_histograms)
buildBlockHistogramsDistance(data, length, block_ids, num_histograms, histograms)
}
insert_cost = nil
cost = nil
switch_signal = nil
new_id = nil
histograms = nil
clusterBlocksDistance(data, length, num_blocks, block_ids, split)
block_ids = nil
}
}
+433
View File
@@ -0,0 +1,433 @@
package brotli
import "math"
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
func initialEntropyCodesLiteral(data []byte, length uint, stride uint, num_histograms uint, histograms []histogramLiteral) {
var seed uint32 = 7
var block_length uint = length / num_histograms
var i uint
clearHistogramsLiteral(histograms, num_histograms)
for i = 0; i < num_histograms; i++ {
var pos uint = length * i / num_histograms
if i != 0 {
pos += uint(myRand(&seed) % uint32(block_length))
}
if pos+stride >= length {
pos = length - stride - 1
}
histogramAddVectorLiteral(&histograms[i], data[pos:], stride)
}
}
func randomSampleLiteral(seed *uint32, data []byte, length uint, stride uint, sample *histogramLiteral) {
var pos uint = 0
if stride >= length {
stride = length
} else {
pos = uint(myRand(seed) % uint32(length-stride+1))
}
histogramAddVectorLiteral(sample, data[pos:], stride)
}
func refineEntropyCodesLiteral(data []byte, length uint, stride uint, num_histograms uint, histograms []histogramLiteral) {
var iters uint = kIterMulForRefining*length/stride + kMinItersForRefining
var seed uint32 = 7
var iter uint
iters = ((iters + num_histograms - 1) / num_histograms) * num_histograms
for iter = 0; iter < iters; iter++ {
var sample histogramLiteral
histogramClearLiteral(&sample)
randomSampleLiteral(&seed, data, length, stride, &sample)
histogramAddHistogramLiteral(&histograms[iter%num_histograms], &sample)
}
}
/* Assigns a block id from the range [0, num_histograms) to each data element
in data[0..length) and fills in block_id[0..length) with the assigned values.
Returns the number of blocks, i.e. one plus the number of block switches. */
func findBlocksLiteral(data []byte, length uint, block_switch_bitcost float64, num_histograms uint, histograms []histogramLiteral, insert_cost []float64, cost []float64, switch_signal []byte, block_id []byte) uint {
var data_size uint = histogramDataSizeLiteral()
var bitmaplen uint = (num_histograms + 7) >> 3
var num_blocks uint = 1
var i uint
var j uint
assert(num_histograms <= 256)
if num_histograms <= 1 {
for i = 0; i < length; i++ {
block_id[i] = 0
}
return 1
}
for i := 0; i < int(data_size*num_histograms); i++ {
insert_cost[i] = 0
}
for i = 0; i < num_histograms; i++ {
insert_cost[i] = fastLog2(uint(uint32(histograms[i].total_count_)))
}
for i = data_size; i != 0; {
i--
for j = 0; j < num_histograms; j++ {
insert_cost[i*num_histograms+j] = insert_cost[j] - bitCost(uint(histograms[j].data_[i]))
}
}
for i := 0; i < int(num_histograms); i++ {
cost[i] = 0
}
for i := 0; i < int(length*bitmaplen); i++ {
switch_signal[i] = 0
}
/* After each iteration of this loop, cost[k] will contain the difference
between the minimum cost of arriving at the current byte position using
entropy code k, and the minimum cost of arriving at the current byte
position. This difference is capped at the block switch cost, and if it
reaches block switch cost, it means that when we trace back from the last
position, we need to switch here. */
for i = 0; i < length; i++ {
var byte_ix uint = i
var ix uint = byte_ix * bitmaplen
var insert_cost_ix uint = uint(data[byte_ix]) * num_histograms
var min_cost float64 = 1e99
var block_switch_cost float64 = block_switch_bitcost
var k uint
for k = 0; k < num_histograms; k++ {
/* We are coding the symbol in data[byte_ix] with entropy code k. */
cost[k] += insert_cost[insert_cost_ix+k]
if cost[k] < min_cost {
min_cost = cost[k]
block_id[byte_ix] = byte(k)
}
}
/* More blocks for the beginning. */
if byte_ix < 2000 {
block_switch_cost *= 0.77 + 0.07*float64(byte_ix)/2000
}
for k = 0; k < num_histograms; k++ {
cost[k] -= min_cost
if cost[k] >= block_switch_cost {
var mask byte = byte(1 << (k & 7))
cost[k] = block_switch_cost
assert(k>>3 < bitmaplen)
switch_signal[ix+(k>>3)] |= mask
/* Trace back from the last position and switch at the marked places. */
}
}
}
{
var byte_ix uint = length - 1
var ix uint = byte_ix * bitmaplen
var cur_id byte = block_id[byte_ix]
for byte_ix > 0 {
var mask byte = byte(1 << (cur_id & 7))
assert(uint(cur_id)>>3 < bitmaplen)
byte_ix--
ix -= bitmaplen
if switch_signal[ix+uint(cur_id>>3)]&mask != 0 {
if cur_id != block_id[byte_ix] {
cur_id = block_id[byte_ix]
num_blocks++
}
}
block_id[byte_ix] = cur_id
}
}
return num_blocks
}
var remapBlockIdsLiteral_kInvalidId uint16 = 256
func remapBlockIdsLiteral(block_ids []byte, length uint, new_id []uint16, num_histograms uint) uint {
var next_id uint16 = 0
var i uint
for i = 0; i < num_histograms; i++ {
new_id[i] = remapBlockIdsLiteral_kInvalidId
}
for i = 0; i < length; i++ {
assert(uint(block_ids[i]) < num_histograms)
if new_id[block_ids[i]] == remapBlockIdsLiteral_kInvalidId {
new_id[block_ids[i]] = next_id
next_id++
}
}
for i = 0; i < length; i++ {
block_ids[i] = byte(new_id[block_ids[i]])
assert(uint(block_ids[i]) < num_histograms)
}
assert(uint(next_id) <= num_histograms)
return uint(next_id)
}
func buildBlockHistogramsLiteral(data []byte, length uint, block_ids []byte, num_histograms uint, histograms []histogramLiteral) {
var i uint
clearHistogramsLiteral(histograms, num_histograms)
for i = 0; i < length; i++ {
histogramAddLiteral(&histograms[block_ids[i]], uint(data[i]))
}
}
var clusterBlocksLiteral_kInvalidIndex uint32 = math.MaxUint32
func clusterBlocksLiteral(data []byte, length uint, num_blocks uint, block_ids []byte, split *blockSplit) {
var histogram_symbols []uint32 = make([]uint32, num_blocks)
var block_lengths []uint32 = make([]uint32, num_blocks)
var expected_num_clusters uint = clustersPerBatch * (num_blocks + histogramsPerBatch - 1) / histogramsPerBatch
var all_histograms_size uint = 0
var all_histograms_capacity uint = expected_num_clusters
var all_histograms []histogramLiteral = make([]histogramLiteral, all_histograms_capacity)
var cluster_size_size uint = 0
var cluster_size_capacity uint = expected_num_clusters
var cluster_size []uint32 = make([]uint32, cluster_size_capacity)
var num_clusters uint = 0
var histograms []histogramLiteral = make([]histogramLiteral, brotli_min_size_t(num_blocks, histogramsPerBatch))
var max_num_pairs uint = histogramsPerBatch * histogramsPerBatch / 2
var pairs_capacity uint = max_num_pairs + 1
var pairs []histogramPair = make([]histogramPair, pairs_capacity)
var pos uint = 0
var clusters []uint32
var num_final_clusters uint
var new_index []uint32
var i uint
var sizes = [histogramsPerBatch]uint32{0}
var new_clusters = [histogramsPerBatch]uint32{0}
var symbols = [histogramsPerBatch]uint32{0}
var remap = [histogramsPerBatch]uint32{0}
for i := 0; i < int(num_blocks); i++ {
block_lengths[i] = 0
}
{
var block_idx uint = 0
for i = 0; i < length; i++ {
assert(block_idx < num_blocks)
block_lengths[block_idx]++
if i+1 == length || block_ids[i] != block_ids[i+1] {
block_idx++
}
}
assert(block_idx == num_blocks)
}
for i = 0; i < num_blocks; i += histogramsPerBatch {
var num_to_combine uint = brotli_min_size_t(num_blocks-i, histogramsPerBatch)
var num_new_clusters uint
var j uint
for j = 0; j < num_to_combine; j++ {
var k uint
histogramClearLiteral(&histograms[j])
for k = 0; uint32(k) < block_lengths[i+j]; k++ {
histogramAddLiteral(&histograms[j], uint(data[pos]))
pos++
}
histograms[j].bit_cost_ = populationCostLiteral(&histograms[j])
new_clusters[j] = uint32(j)
symbols[j] = uint32(j)
sizes[j] = 1
}
num_new_clusters = histogramCombineLiteral(histograms, sizes[:], symbols[:], new_clusters[:], []histogramPair(pairs), num_to_combine, num_to_combine, histogramsPerBatch, max_num_pairs)
if all_histograms_capacity < (all_histograms_size + num_new_clusters) {
var _new_size uint
if all_histograms_capacity == 0 {
_new_size = all_histograms_size + num_new_clusters
} else {
_new_size = all_histograms_capacity
}
var new_array []histogramLiteral
for _new_size < (all_histograms_size + num_new_clusters) {
_new_size *= 2
}
new_array = make([]histogramLiteral, _new_size)
if all_histograms_capacity != 0 {
copy(new_array, all_histograms[:all_histograms_capacity])
}
all_histograms = new_array
all_histograms_capacity = _new_size
}
brotli_ensure_capacity_uint32_t(&cluster_size, &cluster_size_capacity, cluster_size_size+num_new_clusters)
for j = 0; j < num_new_clusters; j++ {
all_histograms[all_histograms_size] = histograms[new_clusters[j]]
all_histograms_size++
cluster_size[cluster_size_size] = sizes[new_clusters[j]]
cluster_size_size++
remap[new_clusters[j]] = uint32(j)
}
for j = 0; j < num_to_combine; j++ {
histogram_symbols[i+j] = uint32(num_clusters) + remap[symbols[j]]
}
num_clusters += num_new_clusters
assert(num_clusters == cluster_size_size)
assert(num_clusters == all_histograms_size)
}
histograms = nil
max_num_pairs = brotli_min_size_t(64*num_clusters, (num_clusters/2)*num_clusters)
if pairs_capacity < max_num_pairs+1 {
pairs = nil
pairs = make([]histogramPair, (max_num_pairs + 1))
}
clusters = make([]uint32, num_clusters)
for i = 0; i < num_clusters; i++ {
clusters[i] = uint32(i)
}
num_final_clusters = histogramCombineLiteral(all_histograms, cluster_size, histogram_symbols, clusters, pairs, num_clusters, num_blocks, maxNumberOfBlockTypes, max_num_pairs)
pairs = nil
cluster_size = nil
new_index = make([]uint32, num_clusters)
for i = 0; i < num_clusters; i++ {
new_index[i] = clusterBlocksLiteral_kInvalidIndex
}
pos = 0
{
var next_index uint32 = 0
for i = 0; i < num_blocks; i++ {
var histo histogramLiteral
var j uint
var best_out uint32
var best_bits float64
histogramClearLiteral(&histo)
for j = 0; uint32(j) < block_lengths[i]; j++ {
histogramAddLiteral(&histo, uint(data[pos]))
pos++
}
if i == 0 {
best_out = histogram_symbols[0]
} else {
best_out = histogram_symbols[i-1]
}
best_bits = histogramBitCostDistanceLiteral(&histo, &all_histograms[best_out])
for j = 0; j < num_final_clusters; j++ {
var cur_bits float64 = histogramBitCostDistanceLiteral(&histo, &all_histograms[clusters[j]])
if cur_bits < best_bits {
best_bits = cur_bits
best_out = clusters[j]
}
}
histogram_symbols[i] = best_out
if new_index[best_out] == clusterBlocksLiteral_kInvalidIndex {
new_index[best_out] = next_index
next_index++
}
}
}
clusters = nil
all_histograms = nil
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, num_blocks)
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, num_blocks)
{
var cur_length uint32 = 0
var block_idx uint = 0
var max_type byte = 0
for i = 0; i < num_blocks; i++ {
cur_length += block_lengths[i]
if i+1 == num_blocks || histogram_symbols[i] != histogram_symbols[i+1] {
var id byte = byte(new_index[histogram_symbols[i]])
split.types[block_idx] = id
split.lengths[block_idx] = cur_length
max_type = brotli_max_uint8_t(max_type, id)
cur_length = 0
block_idx++
}
}
split.num_blocks = block_idx
split.num_types = uint(max_type) + 1
}
new_index = nil
block_lengths = nil
histogram_symbols = nil
}
func splitByteVectorLiteral(data []byte, length uint, literals_per_histogram uint, max_histograms uint, sampling_stride_length uint, block_switch_cost float64, params *encoderParams, split *blockSplit) {
var data_size uint = histogramDataSizeLiteral()
var num_histograms uint = length/literals_per_histogram + 1
var histograms []histogramLiteral
if num_histograms > max_histograms {
num_histograms = max_histograms
}
if length == 0 {
split.num_types = 1
return
} else if length < kMinLengthForBlockSplitting {
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, split.num_blocks+1)
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, split.num_blocks+1)
split.num_types = 1
split.types[split.num_blocks] = 0
split.lengths[split.num_blocks] = uint32(length)
split.num_blocks++
return
}
histograms = make([]histogramLiteral, num_histograms)
/* Find good entropy codes. */
initialEntropyCodesLiteral(data, length, sampling_stride_length, num_histograms, histograms)
refineEntropyCodesLiteral(data, length, sampling_stride_length, num_histograms, histograms)
{
var block_ids []byte = make([]byte, length)
var num_blocks uint = 0
var bitmaplen uint = (num_histograms + 7) >> 3
var insert_cost []float64 = make([]float64, (data_size * num_histograms))
var cost []float64 = make([]float64, num_histograms)
var switch_signal []byte = make([]byte, (length * bitmaplen))
var new_id []uint16 = make([]uint16, num_histograms)
var iters uint
if params.quality < hqZopflificationQuality {
iters = 3
} else {
iters = 10
}
/* Find a good path through literals with the good entropy codes. */
var i uint
for i = 0; i < iters; i++ {
num_blocks = findBlocksLiteral(data, length, block_switch_cost, num_histograms, histograms, insert_cost, cost, switch_signal, block_ids)
num_histograms = remapBlockIdsLiteral(block_ids, length, new_id, num_histograms)
buildBlockHistogramsLiteral(data, length, block_ids, num_histograms, histograms)
}
insert_cost = nil
cost = nil
switch_signal = nil
new_id = nil
histograms = nil
clusterBlocksLiteral(data, length, num_blocks, block_ids, split)
block_ids = nil
}
}
File diff suppressed because it is too large. Load diff
+30
View File
@@ -0,0 +1,30 @@
package brotli
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Functions for clustering similar histograms together. */
type histogramPair struct {
idx1 uint32
idx2 uint32
cost_combo float64
cost_diff float64
}
func histogramPairIsLess(p1 *histogramPair, p2 *histogramPair) bool {
if p1.cost_diff != p2.cost_diff {
return p1.cost_diff > p2.cost_diff
}
return (p1.idx2 - p1.idx1) > (p2.idx2 - p2.idx1)
}
/* Returns entropy reduction of the context map when we combine two clusters. */
func clusterCostDiff(size_a uint, size_b uint) float64 {
var size_c uint = size_a + size_b
return float64(size_a)*fastLog2(size_a) + float64(size_b)*fastLog2(size_b) - float64(size_c)*fastLog2(size_c)
}
+164
View File
@@ -0,0 +1,164 @@
package brotli
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Computes the bit cost reduction by combining out[idx1] and out[idx2] and if
it is below a threshold, stores the pair (idx1, idx2) in the *pairs queue. */
func compareAndPushToQueueCommand(out []histogramCommand, cluster_size []uint32, idx1 uint32, idx2 uint32, max_num_pairs uint, pairs []histogramPair, num_pairs *uint) {
var is_good_pair bool = false
var p histogramPair
p.idx2 = 0
p.idx1 = p.idx2
p.cost_combo = 0
p.cost_diff = p.cost_combo
if idx1 == idx2 {
return
}
if idx2 < idx1 {
var t uint32 = idx2
idx2 = idx1
idx1 = t
}
p.idx1 = idx1
p.idx2 = idx2
p.cost_diff = 0.5 * clusterCostDiff(uint(cluster_size[idx1]), uint(cluster_size[idx2]))
p.cost_diff -= out[idx1].bit_cost_
p.cost_diff -= out[idx2].bit_cost_
if out[idx1].total_count_ == 0 {
p.cost_combo = out[idx2].bit_cost_
is_good_pair = true
} else if out[idx2].total_count_ == 0 {
p.cost_combo = out[idx1].bit_cost_
is_good_pair = true
} else {
var threshold float64
if *num_pairs == 0 {
threshold = 1e99
} else {
threshold = brotli_max_double(0.0, pairs[0].cost_diff)
}
var combo histogramCommand = out[idx1]
var cost_combo float64
histogramAddHistogramCommand(&combo, &out[idx2])
cost_combo = populationCostCommand(&combo)
if cost_combo < threshold-p.cost_diff {
p.cost_combo = cost_combo
is_good_pair = true
}
}
if is_good_pair {
p.cost_diff += p.cost_combo
if *num_pairs > 0 && histogramPairIsLess(&pairs[0], &p) {
/* Replace the top of the queue if needed. */
if *num_pairs < max_num_pairs {
pairs[*num_pairs] = pairs[0]
(*num_pairs)++
}
pairs[0] = p
} else if *num_pairs < max_num_pairs {
pairs[*num_pairs] = p
(*num_pairs)++
}
}
}
func histogramCombineCommand(out []histogramCommand, cluster_size []uint32, symbols []uint32, clusters []uint32, pairs []histogramPair, num_clusters uint, symbols_size uint, max_clusters uint, max_num_pairs uint) uint {
var cost_diff_threshold float64 = 0.0
var min_cluster_size uint = 1
var num_pairs uint = 0
{
/* We maintain a vector of histogram pairs, with the property that the pair
with the maximum bit cost reduction is the first. */
var idx1 uint
for idx1 = 0; idx1 < num_clusters; idx1++ {
var idx2 uint
for idx2 = idx1 + 1; idx2 < num_clusters; idx2++ {
compareAndPushToQueueCommand(out, cluster_size, clusters[idx1], clusters[idx2], max_num_pairs, pairs[0:], &num_pairs)
}
}
}
for num_clusters > min_cluster_size {
var best_idx1 uint32
var best_idx2 uint32
var i uint
if pairs[0].cost_diff >= cost_diff_threshold {
cost_diff_threshold = 1e99
min_cluster_size = max_clusters
continue
}
/* Take the best pair from the top of heap. */
best_idx1 = pairs[0].idx1
best_idx2 = pairs[0].idx2
histogramAddHistogramCommand(&out[best_idx1], &out[best_idx2])
out[best_idx1].bit_cost_ = pairs[0].cost_combo
cluster_size[best_idx1] += cluster_size[best_idx2]
for i = 0; i < symbols_size; i++ {
if symbols[i] == best_idx2 {
symbols[i] = best_idx1
}
}
for i = 0; i < num_clusters; i++ {
if clusters[i] == best_idx2 {
copy(clusters[i:], clusters[i+1:][:num_clusters-i-1])
break
}
}
num_clusters--
{
/* Remove pairs intersecting the just combined best pair. */
var copy_to_idx uint = 0
for i = 0; i < num_pairs; i++ {
var p *histogramPair = &pairs[i]
if p.idx1 == best_idx1 || p.idx2 == best_idx1 || p.idx1 == best_idx2 || p.idx2 == best_idx2 {
/* Remove invalid pair from the queue. */
continue
}
if histogramPairIsLess(&pairs[0], p) {
/* Replace the top of the queue if needed. */
var front histogramPair = pairs[0]
pairs[0] = *p
pairs[copy_to_idx] = front
} else {
pairs[copy_to_idx] = *p
}
copy_to_idx++
}
num_pairs = copy_to_idx
}
/* Push new pairs formed with the combined histogram to the heap. */
for i = 0; i < num_clusters; i++ {
compareAndPushToQueueCommand(out, cluster_size, best_idx1, clusters[i], max_num_pairs, pairs[0:], &num_pairs)
}
}
return num_clusters
}
/* What is the bit cost of moving histogram from cur_symbol to candidate. */
func histogramBitCostDistanceCommand(histogram *histogramCommand, candidate *histogramCommand) float64 {
if histogram.total_count_ == 0 {
return 0.0
} else {
var tmp histogramCommand = *histogram
histogramAddHistogramCommand(&tmp, candidate)
return populationCostCommand(&tmp) - candidate.bit_cost_
}
}
+326
View File
@@ -0,0 +1,326 @@
package brotli
import "math"
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Computes the bit cost reduction by combining out[idx1] and out[idx2] and if
it is below a threshold, stores the pair (idx1, idx2) in the *pairs queue. */
func compareAndPushToQueueDistance(out []histogramDistance, cluster_size []uint32, idx1 uint32, idx2 uint32, max_num_pairs uint, pairs []histogramPair, num_pairs *uint) {
var is_good_pair bool = false
var p histogramPair
p.idx2 = 0
p.idx1 = p.idx2
p.cost_combo = 0
p.cost_diff = p.cost_combo
if idx1 == idx2 {
return
}
if idx2 < idx1 {
var t uint32 = idx2
idx2 = idx1
idx1 = t
}
p.idx1 = idx1
p.idx2 = idx2
p.cost_diff = 0.5 * clusterCostDiff(uint(cluster_size[idx1]), uint(cluster_size[idx2]))
p.cost_diff -= out[idx1].bit_cost_
p.cost_diff -= out[idx2].bit_cost_
if out[idx1].total_count_ == 0 {
p.cost_combo = out[idx2].bit_cost_
is_good_pair = true
} else if out[idx2].total_count_ == 0 {
p.cost_combo = out[idx1].bit_cost_
is_good_pair = true
} else {
var threshold float64
if *num_pairs == 0 {
threshold = 1e99
} else {
threshold = brotli_max_double(0.0, pairs[0].cost_diff)
}
var combo histogramDistance = out[idx1]
var cost_combo float64
histogramAddHistogramDistance(&combo, &out[idx2])
cost_combo = populationCostDistance(&combo)
if cost_combo < threshold-p.cost_diff {
p.cost_combo = cost_combo
is_good_pair = true
}
}
if is_good_pair {
p.cost_diff += p.cost_combo
if *num_pairs > 0 && histogramPairIsLess(&pairs[0], &p) {
/* Replace the top of the queue if needed. */
if *num_pairs < max_num_pairs {
pairs[*num_pairs] = pairs[0]
(*num_pairs)++
}
pairs[0] = p
} else if *num_pairs < max_num_pairs {
pairs[*num_pairs] = p
(*num_pairs)++
}
}
}
func histogramCombineDistance(out []histogramDistance, cluster_size []uint32, symbols []uint32, clusters []uint32, pairs []histogramPair, num_clusters uint, symbols_size uint, max_clusters uint, max_num_pairs uint) uint {
var cost_diff_threshold float64 = 0.0
var min_cluster_size uint = 1
var num_pairs uint = 0
{
/* We maintain a vector of histogram pairs, with the property that the pair
with the maximum bit cost reduction is the first. */
var idx1 uint
for idx1 = 0; idx1 < num_clusters; idx1++ {
var idx2 uint
for idx2 = idx1 + 1; idx2 < num_clusters; idx2++ {
compareAndPushToQueueDistance(out, cluster_size, clusters[idx1], clusters[idx2], max_num_pairs, pairs[0:], &num_pairs)
}
}
}
for num_clusters > min_cluster_size {
var best_idx1 uint32
var best_idx2 uint32
var i uint
if pairs[0].cost_diff >= cost_diff_threshold {
cost_diff_threshold = 1e99
min_cluster_size = max_clusters
continue
}
/* Take the best pair from the top of heap. */
best_idx1 = pairs[0].idx1
best_idx2 = pairs[0].idx2
histogramAddHistogramDistance(&out[best_idx1], &out[best_idx2])
out[best_idx1].bit_cost_ = pairs[0].cost_combo
cluster_size[best_idx1] += cluster_size[best_idx2]
for i = 0; i < symbols_size; i++ {
if symbols[i] == best_idx2 {
symbols[i] = best_idx1
}
}
for i = 0; i < num_clusters; i++ {
if clusters[i] == best_idx2 {
copy(clusters[i:], clusters[i+1:][:num_clusters-i-1])
break
}
}
num_clusters--
{
/* Remove pairs intersecting the just combined best pair. */
var copy_to_idx uint = 0
for i = 0; i < num_pairs; i++ {
var p *histogramPair = &pairs[i]
if p.idx1 == best_idx1 || p.idx2 == best_idx1 || p.idx1 == best_idx2 || p.idx2 == best_idx2 {
/* Remove invalid pair from the queue. */
continue
}
if histogramPairIsLess(&pairs[0], p) {
/* Replace the top of the queue if needed. */
var front histogramPair = pairs[0]
pairs[0] = *p
pairs[copy_to_idx] = front
} else {
pairs[copy_to_idx] = *p
}
copy_to_idx++
}
num_pairs = copy_to_idx
}
/* Push new pairs formed with the combined histogram to the heap. */
for i = 0; i < num_clusters; i++ {
compareAndPushToQueueDistance(out, cluster_size, best_idx1, clusters[i], max_num_pairs, pairs[0:], &num_pairs)
}
}
return num_clusters
}
/* What is the bit cost of moving histogram from cur_symbol to candidate. */
func histogramBitCostDistanceDistance(histogram *histogramDistance, candidate *histogramDistance) float64 {
if histogram.total_count_ == 0 {
return 0.0
} else {
var tmp histogramDistance = *histogram
histogramAddHistogramDistance(&tmp, candidate)
return populationCostDistance(&tmp) - candidate.bit_cost_
}
}
/* Find the best 'out' histogram for each of the 'in' histograms.
When called, clusters[0..num_clusters) contains the unique values from
symbols[0..in_size), but this property is not preserved in this function.
Note: we assume that out[]->bit_cost_ is already up-to-date. */
func histogramRemapDistance(in []histogramDistance, in_size uint, clusters []uint32, num_clusters uint, out []histogramDistance, symbols []uint32) {
var i uint
for i = 0; i < in_size; i++ {
var best_out uint32
if i == 0 {
best_out = symbols[0]
} else {
best_out = symbols[i-1]
}
var best_bits float64 = histogramBitCostDistanceDistance(&in[i], &out[best_out])
var j uint
for j = 0; j < num_clusters; j++ {
var cur_bits float64 = histogramBitCostDistanceDistance(&in[i], &out[clusters[j]])
if cur_bits < best_bits {
best_bits = cur_bits
best_out = clusters[j]
}
}
symbols[i] = best_out
}
/* Recompute each out based on raw and symbols. */
for i = 0; i < num_clusters; i++ {
histogramClearDistance(&out[clusters[i]])
}
for i = 0; i < in_size; i++ {
histogramAddHistogramDistance(&out[symbols[i]], &in[i])
}
}
/* Reorders elements of the out[0..length) array and changes values in
symbols[0..length) array in the following way:
* when called, symbols[] contains indexes into out[], and has N unique
values (possibly N < length)
* on return, symbols'[i] = f(symbols[i]) and
out'[symbols'[i]] = out[symbols[i]], for each 0 <= i < length,
where f is a bijection between the range of symbols[] and [0..N), and
the first occurrences of values in symbols'[i] come in consecutive
increasing order.
Returns N, the number of unique values in symbols[]. */
var histogramReindexDistance_kInvalidIndex uint32 = math.MaxUint32
func histogramReindexDistance(out []histogramDistance, symbols []uint32, length uint) uint {
var new_index []uint32 = make([]uint32, length)
var next_index uint32
var tmp []histogramDistance
var i uint
for i = 0; i < length; i++ {
new_index[i] = histogramReindexDistance_kInvalidIndex
}
next_index = 0
for i = 0; i < length; i++ {
if new_index[symbols[i]] == histogramReindexDistance_kInvalidIndex {
new_index[symbols[i]] = next_index
next_index++
}
}
/* TODO: by using idea of "cycle-sort" we can avoid allocation of
tmp and reduce the number of copying by the factor of 2. */
tmp = make([]histogramDistance, next_index)
next_index = 0
for i = 0; i < length; i++ {
if new_index[symbols[i]] == next_index {
tmp[next_index] = out[symbols[i]]
next_index++
}
symbols[i] = new_index[symbols[i]]
}
new_index = nil
for i = 0; uint32(i) < next_index; i++ {
out[i] = tmp[i]
}
tmp = nil
return uint(next_index)
}
func clusterHistogramsDistance(in []histogramDistance, in_size uint, max_histograms uint, out []histogramDistance, out_size *uint, histogram_symbols []uint32) {
var cluster_size []uint32 = make([]uint32, in_size)
var clusters []uint32 = make([]uint32, in_size)
var num_clusters uint = 0
var max_input_histograms uint = 64
var pairs_capacity uint = max_input_histograms * max_input_histograms / 2
var pairs []histogramPair = make([]histogramPair, (pairs_capacity + 1))
var i uint
/* For the first pass of clustering, we allow all pairs. */
for i = 0; i < in_size; i++ {
cluster_size[i] = 1
}
for i = 0; i < in_size; i++ {
out[i] = in[i]
out[i].bit_cost_ = populationCostDistance(&in[i])
histogram_symbols[i] = uint32(i)
}
for i = 0; i < in_size; i += max_input_histograms {
var num_to_combine uint = brotli_min_size_t(in_size-i, max_input_histograms)
var num_new_clusters uint
var j uint
for j = 0; j < num_to_combine; j++ {
clusters[num_clusters+j] = uint32(i + j)
}
num_new_clusters = histogramCombineDistance(out, cluster_size, histogram_symbols[i:], clusters[num_clusters:], pairs, num_to_combine, num_to_combine, max_histograms, pairs_capacity)
num_clusters += num_new_clusters
}
{
/* For the second pass, we limit the total number of histogram pairs.
After this limit is reached, we only keep searching for the best pair. */
var max_num_pairs uint = brotli_min_size_t(64*num_clusters, (num_clusters/2)*num_clusters)
if pairs_capacity < (max_num_pairs + 1) {
var _new_size uint
if pairs_capacity == 0 {
_new_size = max_num_pairs + 1
} else {
_new_size = pairs_capacity
}
var new_array []histogramPair
for _new_size < (max_num_pairs + 1) {
_new_size *= 2
}
new_array = make([]histogramPair, _new_size)
if pairs_capacity != 0 {
copy(new_array, pairs[:pairs_capacity])
}
pairs = new_array
pairs_capacity = _new_size
}
/* Collapse similar histograms. */
num_clusters = histogramCombineDistance(out, cluster_size, histogram_symbols, clusters, pairs, num_clusters, in_size, max_histograms, max_num_pairs)
}
pairs = nil
cluster_size = nil
/* Find the optimal map from original histograms to the final ones. */
histogramRemapDistance(in, in_size, clusters, num_clusters, out, histogram_symbols)
clusters = nil
/* Convert the context map to a canonical form. */
*out_size = histogramReindexDistance(out, histogram_symbols, in_size)
}
+326
View File
@@ -0,0 +1,326 @@
package brotli
import "math"
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Computes the bit cost reduction by combining out[idx1] and out[idx2] and if
it is below a threshold, stores the pair (idx1, idx2) in the *pairs queue. */
func compareAndPushToQueueLiteral(out []histogramLiteral, cluster_size []uint32, idx1 uint32, idx2 uint32, max_num_pairs uint, pairs []histogramPair, num_pairs *uint) {
var is_good_pair bool = false
var p histogramPair
p.idx2 = 0
p.idx1 = p.idx2
p.cost_combo = 0
p.cost_diff = p.cost_combo
if idx1 == idx2 {
return
}
if idx2 < idx1 {
var t uint32 = idx2
idx2 = idx1
idx1 = t
}
p.idx1 = idx1
p.idx2 = idx2
p.cost_diff = 0.5 * clusterCostDiff(uint(cluster_size[idx1]), uint(cluster_size[idx2]))
p.cost_diff -= out[idx1].bit_cost_
p.cost_diff -= out[idx2].bit_cost_
if out[idx1].total_count_ == 0 {
p.cost_combo = out[idx2].bit_cost_
is_good_pair = true
} else if out[idx2].total_count_ == 0 {
p.cost_combo = out[idx1].bit_cost_
is_good_pair = true
} else {
var threshold float64
if *num_pairs == 0 {
threshold = 1e99
} else {
threshold = brotli_max_double(0.0, pairs[0].cost_diff)
}
var combo histogramLiteral = out[idx1]
var cost_combo float64
histogramAddHistogramLiteral(&combo, &out[idx2])
cost_combo = populationCostLiteral(&combo)
if cost_combo < threshold-p.cost_diff {
p.cost_combo = cost_combo
is_good_pair = true
}
}
if is_good_pair {
p.cost_diff += p.cost_combo
if *num_pairs > 0 && histogramPairIsLess(&pairs[0], &p) {
/* Replace the top of the queue if needed. */
if *num_pairs < max_num_pairs {
pairs[*num_pairs] = pairs[0]
(*num_pairs)++
}
pairs[0] = p
} else if *num_pairs < max_num_pairs {
pairs[*num_pairs] = p
(*num_pairs)++
}
}
}
func histogramCombineLiteral(out []histogramLiteral, cluster_size []uint32, symbols []uint32, clusters []uint32, pairs []histogramPair, num_clusters uint, symbols_size uint, max_clusters uint, max_num_pairs uint) uint {
var cost_diff_threshold float64 = 0.0
var min_cluster_size uint = 1
var num_pairs uint = 0
{
/* We maintain a vector of histogram pairs, with the property that the pair
with the maximum bit cost reduction is the first. */
var idx1 uint
for idx1 = 0; idx1 < num_clusters; idx1++ {
var idx2 uint
for idx2 = idx1 + 1; idx2 < num_clusters; idx2++ {
compareAndPushToQueueLiteral(out, cluster_size, clusters[idx1], clusters[idx2], max_num_pairs, pairs[0:], &num_pairs)
}
}
}
for num_clusters > min_cluster_size {
var best_idx1 uint32
var best_idx2 uint32
var i uint
if pairs[0].cost_diff >= cost_diff_threshold {
cost_diff_threshold = 1e99
min_cluster_size = max_clusters
continue
}
/* Take the best pair from the top of heap. */
best_idx1 = pairs[0].idx1
best_idx2 = pairs[0].idx2
histogramAddHistogramLiteral(&out[best_idx1], &out[best_idx2])
out[best_idx1].bit_cost_ = pairs[0].cost_combo
cluster_size[best_idx1] += cluster_size[best_idx2]
for i = 0; i < symbols_size; i++ {
if symbols[i] == best_idx2 {
symbols[i] = best_idx1
}
}
for i = 0; i < num_clusters; i++ {
if clusters[i] == best_idx2 {
copy(clusters[i:], clusters[i+1:][:num_clusters-i-1])
break
}
}
num_clusters--
{
/* Remove pairs intersecting the just combined best pair. */
var copy_to_idx uint = 0
for i = 0; i < num_pairs; i++ {
var p *histogramPair = &pairs[i]
if p.idx1 == best_idx1 || p.idx2 == best_idx1 || p.idx1 == best_idx2 || p.idx2 == best_idx2 {
/* Remove invalid pair from the queue. */
continue
}
if histogramPairIsLess(&pairs[0], p) {
/* Replace the top of the queue if needed. */
var front histogramPair = pairs[0]
pairs[0] = *p
pairs[copy_to_idx] = front
} else {
pairs[copy_to_idx] = *p
}
copy_to_idx++
}
num_pairs = copy_to_idx
}
/* Push new pairs formed with the combined histogram to the heap. */
for i = 0; i < num_clusters; i++ {
compareAndPushToQueueLiteral(out, cluster_size, best_idx1, clusters[i], max_num_pairs, pairs[0:], &num_pairs)
}
}
return num_clusters
}
/* What is the bit cost of moving histogram from cur_symbol to candidate. */
func histogramBitCostDistanceLiteral(histogram *histogramLiteral, candidate *histogramLiteral) float64 {
if histogram.total_count_ == 0 {
return 0.0
} else {
var tmp histogramLiteral = *histogram
histogramAddHistogramLiteral(&tmp, candidate)
return populationCostLiteral(&tmp) - candidate.bit_cost_
}
}
/* Find the best 'out' histogram for each of the 'in' histograms.
When called, clusters[0..num_clusters) contains the unique values from
symbols[0..in_size), but this property is not preserved in this function.
Note: we assume that out[]->bit_cost_ is already up-to-date. */
func histogramRemapLiteral(in []histogramLiteral, in_size uint, clusters []uint32, num_clusters uint, out []histogramLiteral, symbols []uint32) {
var i uint
for i = 0; i < in_size; i++ {
var best_out uint32
if i == 0 {
best_out = symbols[0]
} else {
best_out = symbols[i-1]
}
var best_bits float64 = histogramBitCostDistanceLiteral(&in[i], &out[best_out])
var j uint
for j = 0; j < num_clusters; j++ {
var cur_bits float64 = histogramBitCostDistanceLiteral(&in[i], &out[clusters[j]])
if cur_bits < best_bits {
best_bits = cur_bits
best_out = clusters[j]
}
}
symbols[i] = best_out
}
/* Recompute each out based on raw and symbols. */
for i = 0; i < num_clusters; i++ {
histogramClearLiteral(&out[clusters[i]])
}
for i = 0; i < in_size; i++ {
histogramAddHistogramLiteral(&out[symbols[i]], &in[i])
}
}
/* Reorders elements of the out[0..length) array and changes values in
symbols[0..length) array in the following way:
* when called, symbols[] contains indexes into out[], and has N unique
values (possibly N < length)
* on return, symbols'[i] = f(symbols[i]) and
out'[symbols'[i]] = out[symbols[i]], for each 0 <= i < length,
where f is a bijection between the range of symbols[] and [0..N), and
the first occurrences of values in symbols'[i] come in consecutive
increasing order.
Returns N, the number of unique values in symbols[]. */
var histogramReindexLiteral_kInvalidIndex uint32 = math.MaxUint32
func histogramReindexLiteral(out []histogramLiteral, symbols []uint32, length uint) uint {
var new_index []uint32 = make([]uint32, length)
var next_index uint32
var tmp []histogramLiteral
var i uint
for i = 0; i < length; i++ {
new_index[i] = histogramReindexLiteral_kInvalidIndex
}
next_index = 0
for i = 0; i < length; i++ {
if new_index[symbols[i]] == histogramReindexLiteral_kInvalidIndex {
new_index[symbols[i]] = next_index
next_index++
}
}
/* TODO: by using idea of "cycle-sort" we can avoid allocation of
tmp and reduce the number of copying by the factor of 2. */
tmp = make([]histogramLiteral, next_index)
next_index = 0
for i = 0; i < length; i++ {
if new_index[symbols[i]] == next_index {
tmp[next_index] = out[symbols[i]]
next_index++
}
symbols[i] = new_index[symbols[i]]
}
new_index = nil
for i = 0; uint32(i) < next_index; i++ {
out[i] = tmp[i]
}
tmp = nil
return uint(next_index)
}
func clusterHistogramsLiteral(in []histogramLiteral, in_size uint, max_histograms uint, out []histogramLiteral, out_size *uint, histogram_symbols []uint32) {
var cluster_size []uint32 = make([]uint32, in_size)
var clusters []uint32 = make([]uint32, in_size)
var num_clusters uint = 0
var max_input_histograms uint = 64
var pairs_capacity uint = max_input_histograms * max_input_histograms / 2
var pairs []histogramPair = make([]histogramPair, (pairs_capacity + 1))
var i uint
/* For the first pass of clustering, we allow all pairs. */
for i = 0; i < in_size; i++ {
cluster_size[i] = 1
}
for i = 0; i < in_size; i++ {
out[i] = in[i]
out[i].bit_cost_ = populationCostLiteral(&in[i])
histogram_symbols[i] = uint32(i)
}
for i = 0; i < in_size; i += max_input_histograms {
var num_to_combine uint = brotli_min_size_t(in_size-i, max_input_histograms)
var num_new_clusters uint
var j uint
for j = 0; j < num_to_combine; j++ {
clusters[num_clusters+j] = uint32(i + j)
}
num_new_clusters = histogramCombineLiteral(out, cluster_size, histogram_symbols[i:], clusters[num_clusters:], pairs, num_to_combine, num_to_combine, max_histograms, pairs_capacity)
num_clusters += num_new_clusters
}
{
/* For the second pass, we limit the total number of histogram pairs.
After this limit is reached, we only keep searching for the best pair. */
var max_num_pairs uint = brotli_min_size_t(64*num_clusters, (num_clusters/2)*num_clusters)
if pairs_capacity < (max_num_pairs + 1) {
var _new_size uint
if pairs_capacity == 0 {
_new_size = max_num_pairs + 1
} else {
_new_size = pairs_capacity
}
var new_array []histogramPair
for _new_size < (max_num_pairs + 1) {
_new_size *= 2
}
new_array = make([]histogramPair, _new_size)
if pairs_capacity != 0 {
copy(new_array, pairs[:pairs_capacity])
}
pairs = new_array
pairs_capacity = _new_size
}
/* Collapse similar histograms. */
num_clusters = histogramCombineLiteral(out, cluster_size, histogram_symbols, clusters, pairs, num_clusters, in_size, max_histograms, max_num_pairs)
}
pairs = nil
cluster_size = nil
/* Find the optimal map from original histograms to the final ones. */
histogramRemapLiteral(in, in_size, clusters, num_clusters, out, histogram_symbols)
clusters = nil
/* Convert the context map to a canonical form. */
*out_size = histogramReindexLiteral(out, histogram_symbols, in_size)
}
+254
View File
@@ -0,0 +1,254 @@
package brotli
var kInsBase = []uint32{
0,
1,
2,
3,
4,
5,
6,
8,
10,
14,
18,
26,
34,
50,
66,
98,
130,
194,
322,
578,
1090,
2114,
6210,
22594,
}
var kInsExtra = []uint32{
0,
0,
0,
0,
0,
0,
1,
1,
2,
2,
3,
3,
4,
4,
5,
5,
6,
7,
8,
9,
10,
12,
14,
24,
}
var kCopyBase = []uint32{
2,
3,
4,
5,
6,
7,
8,
9,
10,
12,
14,
18,
22,
30,
38,
54,
70,
102,
134,
198,
326,
582,
1094,
2118,
}
var kCopyExtra = []uint32{
0,
0,
0,
0,
0,
0,
0,
0,
1,
1,
2,
2,
3,
3,
4,
4,
5,
5,
6,
7,
8,
9,
10,
24,
}
func getInsertLengthCode(insertlen uint) uint16 {
if insertlen < 6 {
return uint16(insertlen)
} else if insertlen < 130 {
var nbits uint32 = log2FloorNonZero(insertlen-2) - 1
return uint16((nbits << 1) + uint32((insertlen-2)>>nbits) + 2)
} else if insertlen < 2114 {
return uint16(log2FloorNonZero(insertlen-66) + 10)
} else if insertlen < 6210 {
return 21
} else if insertlen < 22594 {
return 22
} else {
return 23
}
}
func getCopyLengthCode(copylen uint) uint16 {
if copylen < 10 {
return uint16(copylen - 2)
} else if copylen < 134 {
var nbits uint32 = log2FloorNonZero(copylen-6) - 1
return uint16((nbits << 1) + uint32((copylen-6)>>nbits) + 4)
} else if copylen < 2118 {
return uint16(log2FloorNonZero(copylen-70) + 12)
} else {
return 23
}
}
func combineLengthCodes(inscode uint16, copycode uint16, use_last_distance bool) uint16 {
var bits64 uint16 = uint16(copycode&0x7 | (inscode&0x7)<<3)
if use_last_distance && inscode < 8 && copycode < 16 {
if copycode < 8 {
return bits64
} else {
return bits64 | 64
}
} else {
/* Specification: 5 Encoding of ... (last table) */
/* offset = 2 * index, where index is in range [0..8] */
var offset uint32 = 2 * ((uint32(copycode) >> 3) + 3*(uint32(inscode)>>3))
/* All values in specification are K * 64,
where K = [2, 3, 6, 4, 5, 8, 7, 9, 10],
i + 1 = [1, 2, 3, 4, 5, 6, 7, 8, 9],
K - i - 1 = [1, 1, 3, 0, 0, 2, 0, 1, 2] = D.
All values in D require only 2 bits to encode.
Magic constant is shifted 6 bits left, to avoid final multiplication. */
offset = (offset << 5) + 0x40 + ((0x520D40 >> offset) & 0xC0)
return uint16(offset | uint32(bits64))
}
}
func getLengthCode(insertlen uint, copylen uint, use_last_distance bool, code *uint16) {
var inscode uint16 = getInsertLengthCode(insertlen)
var copycode uint16 = getCopyLengthCode(copylen)
*code = combineLengthCodes(inscode, copycode, use_last_distance)
}
func getInsertBase(inscode uint16) uint32 {
return kInsBase[inscode]
}
func getInsertExtra(inscode uint16) uint32 {
return kInsExtra[inscode]
}
func getCopyBase(copycode uint16) uint32 {
return kCopyBase[copycode]
}
func getCopyExtra(copycode uint16) uint32 {
return kCopyExtra[copycode]
}
type command struct {
insert_len_ uint32
copy_len_ uint32
dist_extra_ uint32
cmd_prefix_ uint16
dist_prefix_ uint16
}
/* distance_code is e.g. 0 for same-as-last short code, or 16 for offset 1. */
func makeCommand(dist *distanceParams, insertlen uint, copylen uint, copylen_code_delta int, distance_code uint) (cmd command) {
/* Don't rely on signed int representation, use honest casts. */
var delta uint32 = uint32(byte(int8(copylen_code_delta)))
cmd.insert_len_ = uint32(insertlen)
cmd.copy_len_ = uint32(uint32(copylen) | delta<<25)
/* The distance prefix and extra bits are stored in this Command as if
npostfix and ndirect were 0, they are only recomputed later after the
clustering if needed. */
prefixEncodeCopyDistance(distance_code, uint(dist.num_direct_distance_codes), uint(dist.distance_postfix_bits), &cmd.dist_prefix_, &cmd.dist_extra_)
getLengthCode(insertlen, uint(int(copylen)+copylen_code_delta), (cmd.dist_prefix_&0x3FF == 0), &cmd.cmd_prefix_)
return cmd
}
func makeInsertCommand(insertlen uint) (cmd command) {
cmd.insert_len_ = uint32(insertlen)
cmd.copy_len_ = 4 << 25
cmd.dist_extra_ = 0
cmd.dist_prefix_ = numDistanceShortCodes
getLengthCode(insertlen, 4, false, &cmd.cmd_prefix_)
return cmd
}
func commandRestoreDistanceCode(self *command, dist *distanceParams) uint32 {
if uint32(self.dist_prefix_&0x3FF) < numDistanceShortCodes+dist.num_direct_distance_codes {
return uint32(self.dist_prefix_) & 0x3FF
} else {
var dcode uint32 = uint32(self.dist_prefix_) & 0x3FF
var nbits uint32 = uint32(self.dist_prefix_) >> 10
var extra uint32 = self.dist_extra_
var postfix_mask uint32 = (1 << dist.distance_postfix_bits) - 1
var hcode uint32 = (dcode - dist.num_direct_distance_codes - numDistanceShortCodes) >> dist.distance_postfix_bits
var lcode uint32 = (dcode - dist.num_direct_distance_codes - numDistanceShortCodes) & postfix_mask
var offset uint32 = ((2 + (hcode & 1)) << nbits) - 4
return ((offset + extra) << dist.distance_postfix_bits) + lcode + dist.num_direct_distance_codes + numDistanceShortCodes
}
}
func commandDistanceContext(self *command) uint32 {
var r uint32 = uint32(self.cmd_prefix_) >> 6
var c uint32 = uint32(self.cmd_prefix_) & 7
if (r == 0 || r == 2 || r == 4 || r == 7) && (c <= 2) {
return c
}
return 3
}
func commandCopyLen(self *command) uint32 {
return self.copy_len_ & 0x1FFFFFF
}
func commandCopyLenCode(self *command) uint32 {
var modifier uint32 = self.copy_len_ >> 25
var delta int32 = int32(int8(byte(modifier | (modifier&0x40)<<1)))
return uint32(int32(self.copy_len_&0x1FFFFFF) + delta)
}
+834
View File
@@ -0,0 +1,834 @@
package brotli
import "encoding/binary"
/* Copyright 2015 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Function for fast encoding of an input fragment, independently from the input
history. This function uses one-pass processing: when we find a backward
match, we immediately emit the corresponding command and literal codes to
the bit stream.
Adapted from the CompressFragment() function in
https://github.com/google/snappy/blob/master/snappy.cc */
const maxDistance_compress_fragment = 262128
func hash5(p []byte, shift uint) uint32 {
var h uint64 = (binary.LittleEndian.Uint64(p) << 24) * uint64(kHashMul32)
return uint32(h >> shift)
}
func hashBytesAtOffset5(v uint64, offset int, shift uint) uint32 {
assert(offset >= 0)
assert(offset <= 3)
{
var h uint64 = ((v >> uint(8*offset)) << 24) * uint64(kHashMul32)
return uint32(h >> shift)
}
}
func isMatch5(p1 []byte, p2 []byte) bool {
return binary.LittleEndian.Uint32(p1) == binary.LittleEndian.Uint32(p2) &&
p1[4] == p2[4]
}
/* Builds a literal prefix code into "depths" and "bits" based on the statistics
of the "input" string and stores it into the bit stream.
Note that the prefix code here is built from the pre-LZ77 input, therefore
we can only approximate the statistics of the actual literal stream.
Moreover, for long inputs we build a histogram from a sample of the input
and thus have to assign a non-zero depth for each literal.
Returns estimated compression ratio millibytes/char for encoding given input
with generated code. */
func buildAndStoreLiteralPrefixCode(input []byte, input_size uint, depths []byte, bits []uint16, storage_ix *uint, storage []byte) uint {
var histogram = [256]uint32{0}
var histogram_total uint
var i uint
if input_size < 1<<15 {
for i = 0; i < input_size; i++ {
histogram[input[i]]++
}
histogram_total = input_size
for i = 0; i < 256; i++ {
/* We weigh the first 11 samples with weight 3 to account for the
balancing effect of the LZ77 phase on the histogram. */
var adjust uint32 = 2 * brotli_min_uint32_t(histogram[i], 11)
histogram[i] += adjust
histogram_total += uint(adjust)
}
} else {
const kSampleRate uint = 29
for i = 0; i < input_size; i += kSampleRate {
histogram[input[i]]++
}
histogram_total = (input_size + kSampleRate - 1) / kSampleRate
for i = 0; i < 256; i++ {
/* We add 1 to each population count to avoid 0 bit depths (since this is
only a sample and we don't know if the symbol appears or not), and we
weigh the first 11 samples with weight 3 to account for the balancing
effect of the LZ77 phase on the histogram (more frequent symbols are
more likely to be in backward references instead as literals). */
var adjust uint32 = 1 + 2*brotli_min_uint32_t(histogram[i], 11)
histogram[i] += adjust
histogram_total += uint(adjust)
}
}
buildAndStoreHuffmanTreeFast(histogram[:], histogram_total, /* max_bits = */
8, depths, bits, storage_ix, storage)
{
var literal_ratio uint = 0
for i = 0; i < 256; i++ {
if histogram[i] != 0 {
literal_ratio += uint(histogram[i] * uint32(depths[i]))
}
}
/* Estimated encoding ratio, millibytes per symbol. */
return (literal_ratio * 125) / histogram_total
}
}
/* Builds a command and distance prefix code (each 64 symbols) into "depth" and
"bits" based on "histogram" and stores it into the bit stream. */
func buildAndStoreCommandPrefixCode1(histogram []uint32, depth []byte, bits []uint16, storage_ix *uint, storage []byte) {
var tree [129]huffmanTree
var cmd_depth = [numCommandSymbols]byte{0}
/* Tree size for building a tree over 64 symbols is 2 * 64 + 1. */
var cmd_bits [64]uint16
createHuffmanTree(histogram, 64, 15, tree[:], depth)
createHuffmanTree(histogram[64:], 64, 14, tree[:], depth[64:])
/* We have to jump through a few hoops here in order to compute
the command bits because the symbols are in a different order than in
the full alphabet. This looks complicated, but having the symbols
in this order in the command bits saves a few branches in the Emit*
functions. */
copy(cmd_depth[:], depth[:24])
copy(cmd_depth[24:][:], depth[40:][:8])
copy(cmd_depth[32:][:], depth[24:][:8])
copy(cmd_depth[40:][:], depth[48:][:8])
copy(cmd_depth[48:][:], depth[32:][:8])
copy(cmd_depth[56:][:], depth[56:][:8])
convertBitDepthsToSymbols(cmd_depth[:], 64, cmd_bits[:])
copy(bits, cmd_bits[:24])
copy(bits[24:], cmd_bits[32:][:8])
copy(bits[32:], cmd_bits[48:][:8])
copy(bits[40:], cmd_bits[24:][:8])
copy(bits[48:], cmd_bits[40:][:8])
copy(bits[56:], cmd_bits[56:][:8])
convertBitDepthsToSymbols(depth[64:], 64, bits[64:])
{
/* Create the bit length array for the full command alphabet. */
var i uint
for i := 0; i < int(64); i++ {
cmd_depth[i] = 0
} /* only 64 first values were used */
copy(cmd_depth[:], depth[:8])
copy(cmd_depth[64:][:], depth[8:][:8])
copy(cmd_depth[128:][:], depth[16:][:8])
copy(cmd_depth[192:][:], depth[24:][:8])
copy(cmd_depth[384:][:], depth[32:][:8])
for i = 0; i < 8; i++ {
cmd_depth[128+8*i] = depth[40+i]
cmd_depth[256+8*i] = depth[48+i]
cmd_depth[448+8*i] = depth[56+i]
}
storeHuffmanTree(cmd_depth[:], numCommandSymbols, tree[:], storage_ix, storage)
}
storeHuffmanTree(depth[64:], 64, tree[:], storage_ix, storage)
}
/* REQUIRES: insertlen < 6210 */
func emitInsertLen1(insertlen uint, depth []byte, bits []uint16, histo []uint32, storage_ix *uint, storage []byte) {
if insertlen < 6 {
var code uint = insertlen + 40
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
histo[code]++
} else if insertlen < 130 {
var tail uint = insertlen - 2
var nbits uint32 = log2FloorNonZero(tail) - 1
var prefix uint = tail >> nbits
var inscode uint = uint((nbits << 1) + uint32(prefix) + 42)
writeBits(uint(depth[inscode]), uint64(bits[inscode]), storage_ix, storage)
writeBits(uint(nbits), uint64(tail)-(uint64(prefix)<<nbits), storage_ix, storage)
histo[inscode]++
} else if insertlen < 2114 {
var tail uint = insertlen - 66
var nbits uint32 = log2FloorNonZero(tail)
var code uint = uint(nbits + 50)
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
writeBits(uint(nbits), uint64(tail)-(uint64(uint(1))<<nbits), storage_ix, storage)
histo[code]++
} else {
writeBits(uint(depth[61]), uint64(bits[61]), storage_ix, storage)
writeBits(12, uint64(insertlen)-2114, storage_ix, storage)
histo[61]++
}
}
func emitLongInsertLen(insertlen uint, depth []byte, bits []uint16, histo []uint32, storage_ix *uint, storage []byte) {
if insertlen < 22594 {
writeBits(uint(depth[62]), uint64(bits[62]), storage_ix, storage)
writeBits(14, uint64(insertlen)-6210, storage_ix, storage)
histo[62]++
} else {
writeBits(uint(depth[63]), uint64(bits[63]), storage_ix, storage)
writeBits(24, uint64(insertlen)-22594, storage_ix, storage)
histo[63]++
}
}
func emitCopyLen1(copylen uint, depth []byte, bits []uint16, histo []uint32, storage_ix *uint, storage []byte) {
if copylen < 10 {
writeBits(uint(depth[copylen+14]), uint64(bits[copylen+14]), storage_ix, storage)
histo[copylen+14]++
} else if copylen < 134 {
var tail uint = copylen - 6
var nbits uint32 = log2FloorNonZero(tail) - 1
var prefix uint = tail >> nbits
var code uint = uint((nbits << 1) + uint32(prefix) + 20)
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
writeBits(uint(nbits), uint64(tail)-(uint64(prefix)<<nbits), storage_ix, storage)
histo[code]++
} else if copylen < 2118 {
var tail uint = copylen - 70
var nbits uint32 = log2FloorNonZero(tail)
var code uint = uint(nbits + 28)
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
writeBits(uint(nbits), uint64(tail)-(uint64(uint(1))<<nbits), storage_ix, storage)
histo[code]++
} else {
writeBits(uint(depth[39]), uint64(bits[39]), storage_ix, storage)
writeBits(24, uint64(copylen)-2118, storage_ix, storage)
histo[39]++
}
}
func emitCopyLenLastDistance1(copylen uint, depth []byte, bits []uint16, histo []uint32, storage_ix *uint, storage []byte) {
if copylen < 12 {
writeBits(uint(depth[copylen-4]), uint64(bits[copylen-4]), storage_ix, storage)
histo[copylen-4]++
} else if copylen < 72 {
var tail uint = copylen - 8
var nbits uint32 = log2FloorNonZero(tail) - 1
var prefix uint = tail >> nbits
var code uint = uint((nbits << 1) + uint32(prefix) + 4)
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
writeBits(uint(nbits), uint64(tail)-(uint64(prefix)<<nbits), storage_ix, storage)
histo[code]++
} else if copylen < 136 {
var tail uint = copylen - 8
var code uint = (tail >> 5) + 30
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
writeBits(5, uint64(tail)&31, storage_ix, storage)
writeBits(uint(depth[64]), uint64(bits[64]), storage_ix, storage)
histo[code]++
histo[64]++
} else if copylen < 2120 {
var tail uint = copylen - 72
var nbits uint32 = log2FloorNonZero(tail)
var code uint = uint(nbits + 28)
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
writeBits(uint(nbits), uint64(tail)-(uint64(uint(1))<<nbits), storage_ix, storage)
writeBits(uint(depth[64]), uint64(bits[64]), storage_ix, storage)
histo[code]++
histo[64]++
} else {
writeBits(uint(depth[39]), uint64(bits[39]), storage_ix, storage)
writeBits(24, uint64(copylen)-2120, storage_ix, storage)
writeBits(uint(depth[64]), uint64(bits[64]), storage_ix, storage)
histo[39]++
histo[64]++
}
}
func emitDistance1(distance uint, depth []byte, bits []uint16, histo []uint32, storage_ix *uint, storage []byte) {
var d uint = distance + 3
var nbits uint32 = log2FloorNonZero(d) - 1
var prefix uint = (d >> nbits) & 1
var offset uint = (2 + prefix) << nbits
var distcode uint = uint(2*(nbits-1) + uint32(prefix) + 80)
writeBits(uint(depth[distcode]), uint64(bits[distcode]), storage_ix, storage)
writeBits(uint(nbits), uint64(d)-uint64(offset), storage_ix, storage)
histo[distcode]++
}
func emitLiterals(input []byte, len uint, depth []byte, bits []uint16, storage_ix *uint, storage []byte) {
var j uint
for j = 0; j < len; j++ {
var lit byte = input[j]
writeBits(uint(depth[lit]), uint64(bits[lit]), storage_ix, storage)
}
}
/* REQUIRES: len <= 1 << 24. */
func storeMetaBlockHeader1(len uint, is_uncompressed bool, storage_ix *uint, storage []byte) {
var nibbles uint = 6
/* ISLAST */
writeBits(1, 0, storage_ix, storage)
if len <= 1<<16 {
nibbles = 4
} else if len <= 1<<20 {
nibbles = 5
}
writeBits(2, uint64(nibbles)-4, storage_ix, storage)
writeBits(nibbles*4, uint64(len)-1, storage_ix, storage)
/* ISUNCOMPRESSED */
writeSingleBit(is_uncompressed, storage_ix, storage)
}
func updateBits(n_bits uint, bits uint32, pos uint, array []byte) {
for n_bits > 0 {
var byte_pos uint = pos >> 3
var n_unchanged_bits uint = pos & 7
var n_changed_bits uint = brotli_min_size_t(n_bits, 8-n_unchanged_bits)
var total_bits uint = n_unchanged_bits + n_changed_bits
var mask uint32 = (^((1 << total_bits) - 1)) | ((1 << n_unchanged_bits) - 1)
var unchanged_bits uint32 = uint32(array[byte_pos]) & mask
var changed_bits uint32 = bits & ((1 << n_changed_bits) - 1)
array[byte_pos] = byte(changed_bits<<n_unchanged_bits | unchanged_bits)
n_bits -= n_changed_bits
bits >>= n_changed_bits
pos += n_changed_bits
}
}
func rewindBitPosition1(new_storage_ix uint, storage_ix *uint, storage []byte) {
var bitpos uint = new_storage_ix & 7
var mask uint = (1 << bitpos) - 1
storage[new_storage_ix>>3] &= byte(mask)
*storage_ix = new_storage_ix
}
var shouldMergeBlock_kSampleRate uint = 43
func shouldMergeBlock(data []byte, len uint, depths []byte) bool {
var histo = [256]uint{0}
var i uint
for i = 0; i < len; i += shouldMergeBlock_kSampleRate {
histo[data[i]]++
}
{
var total uint = (len + shouldMergeBlock_kSampleRate - 1) / shouldMergeBlock_kSampleRate
var r float64 = (fastLog2(total)+0.5)*float64(total) + 200
for i = 0; i < 256; i++ {
r -= float64(histo[i]) * (float64(depths[i]) + fastLog2(histo[i]))
}
return r >= 0.0
}
}
func shouldUseUncompressedMode(metablock_start []byte, next_emit []byte, insertlen uint, literal_ratio uint) bool {
var compressed uint = uint(-cap(next_emit) + cap(metablock_start))
if compressed*50 > insertlen {
return false
} else {
return literal_ratio > 980
}
}
func emitUncompressedMetaBlock1(begin []byte, end []byte, storage_ix_start uint, storage_ix *uint, storage []byte) {
var len uint = uint(-cap(end) + cap(begin))
rewindBitPosition1(storage_ix_start, storage_ix, storage)
storeMetaBlockHeader1(uint(len), true, storage_ix, storage)
*storage_ix = (*storage_ix + 7) &^ 7
copy(storage[*storage_ix>>3:], begin[:len])
*storage_ix += uint(len << 3)
storage[*storage_ix>>3] = 0
}
var kCmdHistoSeed = [128]uint32{
0,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
0,
0,
0,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
0,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
0,
0,
0,
0,
}
var compressFragmentFastImpl_kFirstBlockSize uint = 3 << 15
var compressFragmentFastImpl_kMergeBlockSize uint = 1 << 16
func compressFragmentFastImpl(in []byte, input_size uint, is_last bool, table []int, table_bits uint, cmd_depth []byte, cmd_bits []uint16, cmd_code_numbits *uint, cmd_code []byte, storage_ix *uint, storage []byte) {
var cmd_histo [128]uint32
var ip_end int
var next_emit int = 0
var base_ip int = 0
var input int = 0
const kInputMarginBytes uint = windowGap
const kMinMatchLen uint = 5
var metablock_start int = input
var block_size uint = brotli_min_size_t(input_size, compressFragmentFastImpl_kFirstBlockSize)
var total_block_size uint = block_size
var mlen_storage_ix uint = *storage_ix + 3
var lit_depth [256]byte
var lit_bits [256]uint16
var literal_ratio uint
var ip int
var last_distance int
var shift uint = 64 - table_bits
/* "next_emit" is a pointer to the first byte that is not covered by a
previous copy. Bytes between "next_emit" and the start of the next copy or
the end of the input will be emitted as literal bytes. */
/* Save the start of the first block for position and distance computations.
*/
/* Save the bit position of the MLEN field of the meta-block header, so that
we can update it later if we decide to extend this meta-block. */
storeMetaBlockHeader1(block_size, false, storage_ix, storage)
/* No block splits, no contexts. */
writeBits(13, 0, storage_ix, storage)
literal_ratio = buildAndStoreLiteralPrefixCode(in[input:], block_size, lit_depth[:], lit_bits[:], storage_ix, storage)
{
/* Store the pre-compressed command and distance prefix codes. */
var i uint
for i = 0; i+7 < *cmd_code_numbits; i += 8 {
writeBits(8, uint64(cmd_code[i>>3]), storage_ix, storage)
}
}
writeBits(*cmd_code_numbits&7, uint64(cmd_code[*cmd_code_numbits>>3]), storage_ix, storage)
/* Initialize the command and distance histograms. We will gather
statistics of command and distance codes during the processing
of this block and use it to update the command and distance
prefix codes for the next block. */
emit_commands:
copy(cmd_histo[:], kCmdHistoSeed[:])
/* "ip" is the input pointer. */
ip = input
last_distance = -1
ip_end = int(uint(input) + block_size)
if block_size >= kInputMarginBytes {
var len_limit uint = brotli_min_size_t(block_size-kMinMatchLen, input_size-kInputMarginBytes)
var ip_limit int = int(uint(input) + len_limit)
/* For the last block, we need to keep a 16 bytes margin so that we can be
sure that all distances are at most window size - 16.
For all other blocks, we only need to keep a margin of 5 bytes so that
we don't go over the block size with a copy. */
var next_hash uint32
ip++
for next_hash = hash5(in[ip:], shift); ; {
var skip uint32 = 32
var next_ip int = ip
/* Step 1: Scan forward in the input looking for a 5-byte-long match.
If we get close to exhausting the input then goto emit_remainder.
Heuristic match skipping: If 32 bytes are scanned with no matches
found, start looking only at every other byte. If 32 more bytes are
scanned, look at every third byte, etc.. When a match is found,
immediately go back to looking at every byte. This is a small loss
(~5% performance, ~0.1% density) for compressible data due to more
bookkeeping, but for non-compressible data (such as JPEG) it's a huge
win since the compressor quickly "realizes" the data is incompressible
and doesn't bother looking for matches everywhere.
The "skip" variable keeps track of how many bytes there are since the
last match; dividing it by 32 (i.e. right-shifting by five) gives the
number of bytes to move ahead for each iteration. */
var candidate int
assert(next_emit < ip)
trawl:
for {
var hash uint32 = next_hash
var bytes_between_hash_lookups uint32 = skip >> 5
skip++
assert(hash == hash5(in[next_ip:], shift))
ip = next_ip
next_ip = int(uint32(ip) + bytes_between_hash_lookups)
if next_ip > ip_limit {
goto emit_remainder
}
next_hash = hash5(in[next_ip:], shift)
candidate = ip - last_distance
if isMatch5(in[ip:], in[candidate:]) {
if candidate < ip {
table[hash] = int(ip - base_ip)
break
}
}
candidate = base_ip + table[hash]
assert(candidate >= base_ip)
assert(candidate < ip)
table[hash] = int(ip - base_ip)
if isMatch5(in[ip:], in[candidate:]) {
break
}
}
/* Check copy distance. If candidate is not feasible, continue search.
Checking is done outside of hot loop to reduce overhead. */
if ip-candidate > maxDistance_compress_fragment {
goto trawl
}
/* Step 2: Emit the found match together with the literal bytes from
"next_emit" to the bit stream, and then see if we can find a next match
immediately afterwards. Repeat until we find no match for the input
without emitting some literal bytes. */
{
var base int = ip
/* > 0 */
var matched uint = 5 + findMatchLengthWithLimit(in[candidate+5:], in[ip+5:], uint(ip_end-ip)-5)
var distance int = int(base - candidate)
/* We have a 5-byte match at ip, and we need to emit bytes in
[next_emit, ip). */
var insert uint = uint(base - next_emit)
ip += int(matched)
if insert < 6210 {
emitInsertLen1(insert, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
} else if shouldUseUncompressedMode(in[metablock_start:], in[next_emit:], insert, literal_ratio) {
emitUncompressedMetaBlock1(in[metablock_start:], in[base:], mlen_storage_ix-3, storage_ix, storage)
input_size -= uint(base - input)
input = base
next_emit = input
goto next_block
} else {
emitLongInsertLen(insert, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
}
emitLiterals(in[next_emit:], insert, lit_depth[:], lit_bits[:], storage_ix, storage)
if distance == last_distance {
writeBits(uint(cmd_depth[64]), uint64(cmd_bits[64]), storage_ix, storage)
cmd_histo[64]++
} else {
emitDistance1(uint(distance), cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
last_distance = distance
}
emitCopyLenLastDistance1(matched, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
next_emit = ip
if ip >= ip_limit {
goto emit_remainder
}
/* We could immediately start working at ip now, but to improve
compression we first update "table" with the hashes of some positions
within the last copy. */
{
var input_bytes uint64 = binary.LittleEndian.Uint64(in[ip-3:])
var prev_hash uint32 = hashBytesAtOffset5(input_bytes, 0, shift)
var cur_hash uint32 = hashBytesAtOffset5(input_bytes, 3, shift)
table[prev_hash] = int(ip - base_ip - 3)
prev_hash = hashBytesAtOffset5(input_bytes, 1, shift)
table[prev_hash] = int(ip - base_ip - 2)
prev_hash = hashBytesAtOffset5(input_bytes, 2, shift)
table[prev_hash] = int(ip - base_ip - 1)
candidate = base_ip + table[cur_hash]
table[cur_hash] = int(ip - base_ip)
}
}
for isMatch5(in[ip:], in[candidate:]) {
var base int = ip
/* We have a 5-byte match at ip, and no need to emit any literal bytes
prior to ip. */
var matched uint = 5 + findMatchLengthWithLimit(in[candidate+5:], in[ip+5:], uint(ip_end-ip)-5)
if ip-candidate > maxDistance_compress_fragment {
break
}
ip += int(matched)
last_distance = int(base - candidate) /* > 0 */
emitCopyLen1(matched, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
emitDistance1(uint(last_distance), cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
next_emit = ip
if ip >= ip_limit {
goto emit_remainder
}
/* We could immediately start working at ip now, but to improve
compression we first update "table" with the hashes of some positions
within the last copy. */
{
var input_bytes uint64 = binary.LittleEndian.Uint64(in[ip-3:])
var prev_hash uint32 = hashBytesAtOffset5(input_bytes, 0, shift)
var cur_hash uint32 = hashBytesAtOffset5(input_bytes, 3, shift)
table[prev_hash] = int(ip - base_ip - 3)
prev_hash = hashBytesAtOffset5(input_bytes, 1, shift)
table[prev_hash] = int(ip - base_ip - 2)
prev_hash = hashBytesAtOffset5(input_bytes, 2, shift)
table[prev_hash] = int(ip - base_ip - 1)
candidate = base_ip + table[cur_hash]
table[cur_hash] = int(ip - base_ip)
}
}
ip++
next_hash = hash5(in[ip:], shift)
}
}
emit_remainder:
assert(next_emit <= ip_end)
input += int(block_size)
input_size -= block_size
block_size = brotli_min_size_t(input_size, compressFragmentFastImpl_kMergeBlockSize)
/* Decide if we want to continue this meta-block instead of emitting the
last insert-only command. */
if input_size > 0 && total_block_size+block_size <= 1<<20 && shouldMergeBlock(in[input:], block_size, lit_depth[:]) {
assert(total_block_size > 1<<16)
/* Update the size of the current meta-block and continue emitting commands.
We can do this because the current size and the new size both have 5
nibbles. */
total_block_size += block_size
updateBits(20, uint32(total_block_size-1), mlen_storage_ix, storage)
goto emit_commands
}
/* Emit the remaining bytes as literals. */
if next_emit < ip_end {
var insert uint = uint(ip_end - next_emit)
if insert < 6210 {
emitInsertLen1(insert, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
emitLiterals(in[next_emit:], insert, lit_depth[:], lit_bits[:], storage_ix, storage)
} else if shouldUseUncompressedMode(in[metablock_start:], in[next_emit:], insert, literal_ratio) {
emitUncompressedMetaBlock1(in[metablock_start:], in[ip_end:], mlen_storage_ix-3, storage_ix, storage)
} else {
emitLongInsertLen(insert, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
emitLiterals(in[next_emit:], insert, lit_depth[:], lit_bits[:], storage_ix, storage)
}
}
next_emit = ip_end
/* If we have more data, write a new meta-block header and prefix codes and
then continue emitting commands. */
next_block:
if input_size > 0 {
metablock_start = input
block_size = brotli_min_size_t(input_size, compressFragmentFastImpl_kFirstBlockSize)
total_block_size = block_size
/* Save the bit position of the MLEN field of the meta-block header, so that
we can update it later if we decide to extend this meta-block. */
mlen_storage_ix = *storage_ix + 3
storeMetaBlockHeader1(block_size, false, storage_ix, storage)
/* No block splits, no contexts. */
writeBits(13, 0, storage_ix, storage)
literal_ratio = buildAndStoreLiteralPrefixCode(in[input:], block_size, lit_depth[:], lit_bits[:], storage_ix, storage)
buildAndStoreCommandPrefixCode1(cmd_histo[:], cmd_depth, cmd_bits, storage_ix, storage)
goto emit_commands
}
if !is_last {
/* If this is not the last block, update the command and distance prefix
codes for the next block and store the compressed forms. */
cmd_code[0] = 0
*cmd_code_numbits = 0
buildAndStoreCommandPrefixCode1(cmd_histo[:], cmd_depth, cmd_bits, cmd_code_numbits, cmd_code)
}
}
/* Compresses "input" string to the "*storage" buffer as one or more complete
meta-blocks, and updates the "*storage_ix" bit position.
If "is_last" is 1, emits an additional empty last meta-block.
"cmd_depth" and "cmd_bits" contain the command and distance prefix codes
(see comment in encode.h) used for the encoding of this input fragment.
If "is_last" is 0, they are updated to reflect the statistics
of this input fragment, to be used for the encoding of the next fragment.
"*cmd_code_numbits" is the number of bits of the compressed representation
of the command and distance prefix codes, and "cmd_code" is an array of
at least "(*cmd_code_numbits + 7) >> 3" size that contains the compressed
command and distance prefix codes. If "is_last" is 0, these are also
updated to represent the updated "cmd_depth" and "cmd_bits".
REQUIRES: "input_size" is greater than zero, or "is_last" is 1.
REQUIRES: "input_size" is less or equal to maximal metablock size (1 << 24).
REQUIRES: All elements in "table[0..table_size-1]" are initialized to zero.
REQUIRES: "table_size" is an odd (9, 11, 13, 15) power of two
OUTPUT: maximal copy distance <= |input_size|
OUTPUT: maximal copy distance <= BROTLI_MAX_BACKWARD_LIMIT(18) */
func compressFragmentFast(input []byte, input_size uint, is_last bool, table []int, table_size uint, cmd_depth []byte, cmd_bits []uint16, cmd_code_numbits *uint, cmd_code []byte, storage_ix *uint, storage []byte) {
var initial_storage_ix uint = *storage_ix
var table_bits uint = uint(log2FloorNonZero(table_size))
if input_size == 0 {
assert(is_last)
writeBits(1, 1, storage_ix, storage) /* islast */
writeBits(1, 1, storage_ix, storage) /* isempty */
*storage_ix = (*storage_ix + 7) &^ 7
return
}
compressFragmentFastImpl(input, input_size, is_last, table, table_bits, cmd_depth, cmd_bits, cmd_code_numbits, cmd_code, storage_ix, storage)
/* If output is larger than single uncompressed block, rewrite it. */
if *storage_ix-initial_storage_ix > 31+(input_size<<3) {
emitUncompressedMetaBlock1(input, input[input_size:], initial_storage_ix, storage_ix, storage)
}
if is_last {
writeBits(1, 1, storage_ix, storage) /* islast */
writeBits(1, 1, storage_ix, storage) /* isempty */
*storage_ix = (*storage_ix + 7) &^ 7
}
}
+773
View File
@@ -0,0 +1,773 @@
package brotli
import "encoding/binary"
/* Copyright 2015 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Function for fast encoding of an input fragment, independently from the input
history. This function uses two-pass processing: in the first pass we save
the found backward matches and literal bytes into a buffer, and in the
second pass we emit them into the bit stream using prefix codes built based
on the actual command and literal byte histograms. */
const kCompressFragmentTwoPassBlockSize uint = 1 << 17
func hash1(p []byte, shift uint, length uint) uint32 {
var h uint64 = (binary.LittleEndian.Uint64(p) << ((8 - length) * 8)) * uint64(kHashMul32)
return uint32(h >> shift)
}
func hashBytesAtOffset(v uint64, offset uint, shift uint, length uint) uint32 {
assert(offset <= 8-length)
{
var h uint64 = ((v >> (8 * offset)) << ((8 - length) * 8)) * uint64(kHashMul32)
return uint32(h >> shift)
}
}
func isMatch1(p1 []byte, p2 []byte, length uint) bool {
if binary.LittleEndian.Uint32(p1) != binary.LittleEndian.Uint32(p2) {
return false
}
if length == 4 {
return true
}
return p1[4] == p2[4] && p1[5] == p2[5]
}
/*
Builds a command and distance prefix code (each 64 symbols) into "depth" and
"bits" based on "histogram" and stores it into the bit stream.
*/
func buildAndStoreCommandPrefixCode(histogram []uint32, depth []byte, bits []uint16, storage_ix *uint, storage []byte) {
var tree [129]huffmanTree
var cmd_depth = [numCommandSymbols]byte{0}
/* Tree size for building a tree over 64 symbols is 2 * 64 + 1. */
var cmd_bits [64]uint16
createHuffmanTree(histogram, 64, 15, tree[:], depth)
createHuffmanTree(histogram[64:], 64, 14, tree[:], depth[64:])
/* We have to jump through a few hoops here in order to compute
the command bits because the symbols are in a different order than in
the full alphabet. This looks complicated, but having the symbols
in this order in the command bits saves a few branches in the Emit*
functions. */
copy(cmd_depth[:], depth[24:][:24])
copy(cmd_depth[24:][:], depth[:8])
copy(cmd_depth[32:][:], depth[48:][:8])
copy(cmd_depth[40:][:], depth[8:][:8])
copy(cmd_depth[48:][:], depth[56:][:8])
copy(cmd_depth[56:][:], depth[16:][:8])
convertBitDepthsToSymbols(cmd_depth[:], 64, cmd_bits[:])
copy(bits, cmd_bits[24:][:8])
copy(bits[8:], cmd_bits[40:][:8])
copy(bits[16:], cmd_bits[56:][:8])
copy(bits[24:], cmd_bits[:24])
copy(bits[48:], cmd_bits[32:][:8])
copy(bits[56:], cmd_bits[48:][:8])
convertBitDepthsToSymbols(depth[64:], 64, bits[64:])
{
/* Create the bit length array for the full command alphabet. */
var i uint
for i := 0; i < int(64); i++ {
cmd_depth[i] = 0
} /* only 64 first values were used */
copy(cmd_depth[:], depth[24:][:8])
copy(cmd_depth[64:][:], depth[32:][:8])
copy(cmd_depth[128:][:], depth[40:][:8])
copy(cmd_depth[192:][:], depth[48:][:8])
copy(cmd_depth[384:][:], depth[56:][:8])
for i = 0; i < 8; i++ {
cmd_depth[128+8*i] = depth[i]
cmd_depth[256+8*i] = depth[8+i]
cmd_depth[448+8*i] = depth[16+i]
}
storeHuffmanTree(cmd_depth[:], numCommandSymbols, tree[:], storage_ix, storage)
}
storeHuffmanTree(depth[64:], 64, tree[:], storage_ix, storage)
}
func emitInsertLen(insertlen uint32, commands *[]uint32) {
if insertlen < 6 {
(*commands)[0] = insertlen
} else if insertlen < 130 {
var tail uint32 = insertlen - 2
var nbits uint32 = log2FloorNonZero(uint(tail)) - 1
var prefix uint32 = tail >> nbits
var inscode uint32 = (nbits << 1) + prefix + 2
var extra uint32 = tail - (prefix << nbits)
(*commands)[0] = inscode | extra<<8
} else if insertlen < 2114 {
var tail uint32 = insertlen - 66
var nbits uint32 = log2FloorNonZero(uint(tail))
var code uint32 = nbits + 10
var extra uint32 = tail - (1 << nbits)
(*commands)[0] = code | extra<<8
} else if insertlen < 6210 {
var extra uint32 = insertlen - 2114
(*commands)[0] = 21 | extra<<8
} else if insertlen < 22594 {
var extra uint32 = insertlen - 6210
(*commands)[0] = 22 | extra<<8
} else {
var extra uint32 = insertlen - 22594
(*commands)[0] = 23 | extra<<8
}
*commands = (*commands)[1:]
}
func emitCopyLen(copylen uint, commands *[]uint32) {
if copylen < 10 {
(*commands)[0] = uint32(copylen + 38)
} else if copylen < 134 {
var tail uint = copylen - 6
var nbits uint = uint(log2FloorNonZero(tail) - 1)
var prefix uint = tail >> nbits
var code uint = (nbits << 1) + prefix + 44
var extra uint = tail - (prefix << nbits)
(*commands)[0] = uint32(code | extra<<8)
} else if copylen < 2118 {
var tail uint = copylen - 70
var nbits uint = uint(log2FloorNonZero(tail))
var code uint = nbits + 52
var extra uint = tail - (uint(1) << nbits)
(*commands)[0] = uint32(code | extra<<8)
} else {
var extra uint = copylen - 2118
(*commands)[0] = uint32(63 | extra<<8)
}
*commands = (*commands)[1:]
}
func emitCopyLenLastDistance(copylen uint, commands *[]uint32) {
if copylen < 12 {
(*commands)[0] = uint32(copylen + 20)
*commands = (*commands)[1:]
} else if copylen < 72 {
var tail uint = copylen - 8
var nbits uint = uint(log2FloorNonZero(tail) - 1)
var prefix uint = tail >> nbits
var code uint = (nbits << 1) + prefix + 28
var extra uint = tail - (prefix << nbits)
(*commands)[0] = uint32(code | extra<<8)
*commands = (*commands)[1:]
} else if copylen < 136 {
var tail uint = copylen - 8
var code uint = (tail >> 5) + 54
var extra uint = tail & 31
(*commands)[0] = uint32(code | extra<<8)
*commands = (*commands)[1:]
(*commands)[0] = 64
*commands = (*commands)[1:]
} else if copylen < 2120 {
var tail uint = copylen - 72
var nbits uint = uint(log2FloorNonZero(tail))
var code uint = nbits + 52
var extra uint = tail - (uint(1) << nbits)
(*commands)[0] = uint32(code | extra<<8)
*commands = (*commands)[1:]
(*commands)[0] = 64
*commands = (*commands)[1:]
} else {
var extra uint = copylen - 2120
(*commands)[0] = uint32(63 | extra<<8)
*commands = (*commands)[1:]
(*commands)[0] = 64
*commands = (*commands)[1:]
}
}
func emitDistance(distance uint32, commands *[]uint32) {
var d uint32 = distance + 3
var nbits uint32 = log2FloorNonZero(uint(d)) - 1
var prefix uint32 = (d >> nbits) & 1
var offset uint32 = (2 + prefix) << nbits
var distcode uint32 = 2*(nbits-1) + prefix + 80
var extra uint32 = d - offset
(*commands)[0] = distcode | extra<<8
*commands = (*commands)[1:]
}
/* REQUIRES: len <= 1 << 24. */
func storeMetaBlockHeader(len uint, is_uncompressed bool, storage_ix *uint, storage []byte) {
var nibbles uint = 6
/* ISLAST */
writeBits(1, 0, storage_ix, storage)
if len <= 1<<16 {
nibbles = 4
} else if len <= 1<<20 {
nibbles = 5
}
writeBits(2, uint64(nibbles)-4, storage_ix, storage)
writeBits(nibbles*4, uint64(len)-1, storage_ix, storage)
/* ISUNCOMPRESSED */
writeSingleBit(is_uncompressed, storage_ix, storage)
}
func storeMetaBlockHeaderBW(len uint, is_uncompressed bool, bw *bitWriter) {
var nibbles uint = 6
/* ISLAST */
bw.writeBits(1, 0)
if len <= 1<<16 {
nibbles = 4
} else if len <= 1<<20 {
nibbles = 5
}
bw.writeBits(2, uint64(nibbles)-4)
bw.writeBits(nibbles*4, uint64(len)-1)
/* ISUNCOMPRESSED */
bw.writeSingleBit(is_uncompressed)
}
func createCommands(input []byte, block_size uint, input_size uint, base_ip_ptr []byte, table []int, table_bits uint, min_match uint, literals *[]byte, commands *[]uint32) {
var ip int = 0
var shift uint = 64 - table_bits
var ip_end int = int(block_size)
var base_ip int = -cap(base_ip_ptr) + cap(input)
var next_emit int = 0
var last_distance int = -1
/* "ip" is the input pointer. */
const kInputMarginBytes uint = windowGap
/* "next_emit" is a pointer to the first byte that is not covered by a
previous copy. Bytes between "next_emit" and the start of the next copy or
the end of the input will be emitted as literal bytes. */
if block_size >= kInputMarginBytes {
var len_limit uint = brotli_min_size_t(block_size-min_match, input_size-kInputMarginBytes)
var ip_limit int = int(len_limit)
/* For the last block, we need to keep a 16 bytes margin so that we can be
sure that all distances are at most window size - 16.
For all other blocks, we only need to keep a margin of 5 bytes so that
we don't go over the block size with a copy. */
var next_hash uint32
ip++
for next_hash = hash1(input[ip:], shift, min_match); ; {
var skip uint32 = 32
var next_ip int = ip
/* Step 1: Scan forward in the input looking for a 6-byte-long match.
If we get close to exhausting the input then goto emit_remainder.
Heuristic match skipping: If 32 bytes are scanned with no matches
found, start looking only at every other byte. If 32 more bytes are
scanned, look at every third byte, etc.. When a match is found,
immediately go back to looking at every byte. This is a small loss
(~5% performance, ~0.1% density) for compressible data due to more
bookkeeping, but for non-compressible data (such as JPEG) it's a huge
win since the compressor quickly "realizes" the data is incompressible
and doesn't bother looking for matches everywhere.
The "skip" variable keeps track of how many bytes there are since the
last match; dividing it by 32 (ie. right-shifting by five) gives the
number of bytes to move ahead for each iteration. */
var candidate int
assert(next_emit < ip)
trawl:
for {
var hash uint32 = next_hash
var bytes_between_hash_lookups uint32 = skip >> 5
skip++
ip = next_ip
assert(hash == hash1(input[ip:], shift, min_match))
next_ip = int(uint32(ip) + bytes_between_hash_lookups)
if next_ip > ip_limit {
goto emit_remainder
}
next_hash = hash1(input[next_ip:], shift, min_match)
candidate = ip - last_distance
if isMatch1(input[ip:], base_ip_ptr[candidate-base_ip:], min_match) {
if candidate < ip {
table[hash] = int(ip - base_ip)
break
}
}
candidate = base_ip + table[hash]
assert(candidate >= base_ip)
assert(candidate < ip)
table[hash] = int(ip - base_ip)
if isMatch1(input[ip:], base_ip_ptr[candidate-base_ip:], min_match) {
break
}
}
/* Check copy distance. If candidate is not feasible, continue search.
Checking is done outside of hot loop to reduce overhead. */
if ip-candidate > maxDistance_compress_fragment {
goto trawl
}
/* Step 2: Emit the found match together with the literal bytes from
"next_emit", and then see if we can find a next match immediately
afterwards. Repeat until we find no match for the input
without emitting some literal bytes. */
{
var base int = ip
/* > 0 */
var matched uint = min_match + findMatchLengthWithLimit(base_ip_ptr[uint(candidate-base_ip)+min_match:], input[uint(ip)+min_match:], uint(ip_end-ip)-min_match)
var distance int = int(base - candidate)
/* We have a 6-byte match at ip, and we need to emit bytes in
[next_emit, ip). */
var insert int = int(base - next_emit)
ip += int(matched)
emitInsertLen(uint32(insert), commands)
copy(*literals, input[next_emit:][:uint(insert)])
*literals = (*literals)[insert:]
if distance == last_distance {
(*commands)[0] = 64
*commands = (*commands)[1:]
} else {
emitDistance(uint32(distance), commands)
last_distance = distance
}
emitCopyLenLastDistance(matched, commands)
next_emit = ip
if ip >= ip_limit {
goto emit_remainder
}
{
var input_bytes uint64
var cur_hash uint32
/* We could immediately start working at ip now, but to improve
compression we first update "table" with the hashes of some
positions within the last copy. */
var prev_hash uint32
if min_match == 4 {
input_bytes = binary.LittleEndian.Uint64(input[ip-3:])
cur_hash = hashBytesAtOffset(input_bytes, 3, shift, min_match)
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
table[prev_hash] = int(ip - base_ip - 3)
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
table[prev_hash] = int(ip - base_ip - 2)
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
table[prev_hash] = int(ip - base_ip - 1)
} else {
input_bytes = binary.LittleEndian.Uint64(input[ip-5:])
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
table[prev_hash] = int(ip - base_ip - 5)
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
table[prev_hash] = int(ip - base_ip - 4)
prev_hash = hashBytesAtOffset(input_bytes, 2, shift, min_match)
table[prev_hash] = int(ip - base_ip - 3)
input_bytes = binary.LittleEndian.Uint64(input[ip-2:])
cur_hash = hashBytesAtOffset(input_bytes, 2, shift, min_match)
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
table[prev_hash] = int(ip - base_ip - 2)
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
table[prev_hash] = int(ip - base_ip - 1)
}
candidate = base_ip + table[cur_hash]
table[cur_hash] = int(ip - base_ip)
}
}
for ip-candidate <= maxDistance_compress_fragment && isMatch1(input[ip:], base_ip_ptr[candidate-base_ip:], min_match) {
var base int = ip
/* We have a 6-byte match at ip, and no need to emit any
literal bytes prior to ip. */
var matched uint = min_match + findMatchLengthWithLimit(base_ip_ptr[uint(candidate-base_ip)+min_match:], input[uint(ip)+min_match:], uint(ip_end-ip)-min_match)
ip += int(matched)
last_distance = int(base - candidate) /* > 0 */
emitCopyLen(matched, commands)
emitDistance(uint32(last_distance), commands)
next_emit = ip
if ip >= ip_limit {
goto emit_remainder
}
{
var input_bytes uint64
var cur_hash uint32
/* We could immediately start working at ip now, but to improve
compression we first update "table" with the hashes of some
positions within the last copy. */
var prev_hash uint32
if min_match == 4 {
input_bytes = binary.LittleEndian.Uint64(input[ip-3:])
cur_hash = hashBytesAtOffset(input_bytes, 3, shift, min_match)
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
table[prev_hash] = int(ip - base_ip - 3)
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
table[prev_hash] = int(ip - base_ip - 2)
prev_hash = hashBytesAtOffset(input_bytes, 2, shift, min_match)
table[prev_hash] = int(ip - base_ip - 1)
} else {
input_bytes = binary.LittleEndian.Uint64(input[ip-5:])
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
table[prev_hash] = int(ip - base_ip - 5)
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
table[prev_hash] = int(ip - base_ip - 4)
prev_hash = hashBytesAtOffset(input_bytes, 2, shift, min_match)
table[prev_hash] = int(ip - base_ip - 3)
input_bytes = binary.LittleEndian.Uint64(input[ip-2:])
cur_hash = hashBytesAtOffset(input_bytes, 2, shift, min_match)
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
table[prev_hash] = int(ip - base_ip - 2)
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
table[prev_hash] = int(ip - base_ip - 1)
}
candidate = base_ip + table[cur_hash]
table[cur_hash] = int(ip - base_ip)
}
}
ip++
next_hash = hash1(input[ip:], shift, min_match)
}
}
emit_remainder:
assert(next_emit <= ip_end)
/* Emit the remaining bytes as literals. */
if next_emit < ip_end {
var insert uint32 = uint32(ip_end - next_emit)
emitInsertLen(insert, commands)
copy(*literals, input[next_emit:][:insert])
*literals = (*literals)[insert:]
}
}
var storeCommands_kNumExtraBits = [128]uint32{
0,
0,
0,
0,
0,
0,
1,
1,
2,
2,
3,
3,
4,
4,
5,
5,
6,
7,
8,
9,
10,
12,
14,
24,
0,
0,
0,
0,
0,
0,
0,
0,
1,
1,
2,
2,
3,
3,
4,
4,
0,
0,
0,
0,
0,
0,
0,
0,
1,
1,
2,
2,
3,
3,
4,
4,
5,
5,
6,
7,
8,
9,
10,
24,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
1,
2,
2,
3,
3,
4,
4,
5,
5,
6,
6,
7,
7,
8,
8,
9,
9,
10,
10,
11,
11,
12,
12,
13,
13,
14,
14,
15,
15,
16,
16,
17,
17,
18,
18,
19,
19,
20,
20,
21,
21,
22,
22,
23,
23,
24,
24,
}
var storeCommands_kInsertOffset = [24]uint32{
0,
1,
2,
3,
4,
5,
6,
8,
10,
14,
18,
26,
34,
50,
66,
98,
130,
194,
322,
578,
1090,
2114,
6210,
22594,
}
func storeCommands(literals []byte, num_literals uint, commands []uint32, num_commands uint, storage_ix *uint, storage []byte) {
var lit_depths [256]byte
var lit_bits [256]uint16
var lit_histo = [256]uint32{0}
var cmd_depths = [128]byte{0}
var cmd_bits = [128]uint16{0}
var cmd_histo = [128]uint32{0}
var i uint
for i = 0; i < num_literals; i++ {
lit_histo[literals[i]]++
}
buildAndStoreHuffmanTreeFast(lit_histo[:], num_literals, /* max_bits = */
8, lit_depths[:], lit_bits[:], storage_ix, storage)
for i = 0; i < num_commands; i++ {
var code uint32 = commands[i] & 0xFF
assert(code < 128)
cmd_histo[code]++
}
cmd_histo[1] += 1
cmd_histo[2] += 1
cmd_histo[64] += 1
cmd_histo[84] += 1
buildAndStoreCommandPrefixCode(cmd_histo[:], cmd_depths[:], cmd_bits[:], storage_ix, storage)
for i = 0; i < num_commands; i++ {
var cmd uint32 = commands[i]
var code uint32 = cmd & 0xFF
var extra uint32 = cmd >> 8
assert(code < 128)
writeBits(uint(cmd_depths[code]), uint64(cmd_bits[code]), storage_ix, storage)
writeBits(uint(storeCommands_kNumExtraBits[code]), uint64(extra), storage_ix, storage)
if code < 24 {
var insert uint32 = storeCommands_kInsertOffset[code] + extra
var j uint32
for j = 0; j < insert; j++ {
var lit byte = literals[0]
writeBits(uint(lit_depths[lit]), uint64(lit_bits[lit]), storage_ix, storage)
literals = literals[1:]
}
}
}
}
/* Acceptable loss for uncompressible speedup is 2% */
const minRatio = 0.98
const sampleRate = 43
func shouldCompress(input []byte, input_size uint, num_literals uint) bool {
var corpus_size float64 = float64(input_size)
if float64(num_literals) < minRatio*corpus_size {
return true
} else {
var literal_histo = [256]uint32{0}
var max_total_bit_cost float64 = corpus_size * 8 * minRatio / sampleRate
var i uint
for i = 0; i < input_size; i += sampleRate {
literal_histo[input[i]]++
}
return bitsEntropy(literal_histo[:], 256) < max_total_bit_cost
}
}
func rewindBitPosition(new_storage_ix uint, storage_ix *uint, storage []byte) {
var bitpos uint = new_storage_ix & 7
var mask uint = (1 << bitpos) - 1
storage[new_storage_ix>>3] &= byte(mask)
*storage_ix = new_storage_ix
}
func emitUncompressedMetaBlock(input []byte, input_size uint, storage_ix *uint, storage []byte) {
storeMetaBlockHeader(input_size, true, storage_ix, storage)
*storage_ix = (*storage_ix + 7) &^ 7
copy(storage[*storage_ix>>3:], input[:input_size])
*storage_ix += input_size << 3
storage[*storage_ix>>3] = 0
}
func compressFragmentTwoPassImpl(input []byte, input_size uint, is_last bool, command_buf []uint32, literal_buf []byte, table []int, table_bits uint, min_match uint, storage_ix *uint, storage []byte) {
/* Save the start of the first block for position and distance computations.
*/
var base_ip []byte = input
for input_size > 0 {
var block_size uint = brotli_min_size_t(input_size, kCompressFragmentTwoPassBlockSize)
var commands []uint32 = command_buf
var literals []byte = literal_buf
var num_literals uint
createCommands(input, block_size, input_size, base_ip, table, table_bits, min_match, &literals, &commands)
num_literals = uint(-cap(literals) + cap(literal_buf))
if shouldCompress(input, block_size, num_literals) {
var num_commands uint = uint(-cap(commands) + cap(command_buf))
storeMetaBlockHeader(block_size, false, storage_ix, storage)
/* No block splits, no contexts. */
writeBits(13, 0, storage_ix, storage)
storeCommands(literal_buf, num_literals, command_buf, num_commands, storage_ix, storage)
} else {
/* Since we did not find many backward references and the entropy of
the data is close to 8 bits, we can simply emit an uncompressed block.
This makes compression speed of uncompressible data about 3x faster. */
emitUncompressedMetaBlock(input, block_size, storage_ix, storage)
}
input = input[block_size:]
input_size -= block_size
}
}
/*
Compresses "input" string to the "*storage" buffer as one or more complete
meta-blocks, and updates the "*storage_ix" bit position.
If "is_last" is 1, emits an additional empty last meta-block.
REQUIRES: "input_size" is greater than zero, or "is_last" is 1.
REQUIRES: "input_size" is less or equal to maximal metablock size (1 << 24).
REQUIRES: "command_buf" and "literal_buf" point to at least
kCompressFragmentTwoPassBlockSize long arrays.
REQUIRES: All elements in "table[0..table_size-1]" are initialized to zero.
REQUIRES: "table_size" is a power of two
OUTPUT: maximal copy distance <= |input_size|
OUTPUT: maximal copy distance <= BROTLI_MAX_BACKWARD_LIMIT(18)
*/
func compressFragmentTwoPass(input []byte, input_size uint, is_last bool, command_buf []uint32, literal_buf []byte, table []int, table_size uint, storage_ix *uint, storage []byte) {
var initial_storage_ix uint = *storage_ix
var table_bits uint = uint(log2FloorNonZero(table_size))
var min_match uint
if table_bits <= 15 {
min_match = 4
} else {
min_match = 6
}
compressFragmentTwoPassImpl(input, input_size, is_last, command_buf, literal_buf, table, table_bits, min_match, storage_ix, storage)
/* If output is larger than single uncompressed block, rewrite it. */
if *storage_ix-initial_storage_ix > 31+(input_size<<3) {
rewindBitPosition(initial_storage_ix, storage_ix, storage)
emitUncompressedMetaBlock(input, input_size, storage_ix, storage)
}
if is_last {
writeBits(1, 1, storage_ix, storage) /* islast */
writeBits(1, 1, storage_ix, storage) /* isempty */
*storage_ix = (*storage_ix + 7) &^ 7
}
}
+77
View File
@@ -0,0 +1,77 @@
package brotli
/* Copyright 2016 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Specification: 7.3. Encoding of the context map */
const contextMapMaxRle = 16
/* Specification: 2. Compressed representation overview */
const maxNumberOfBlockTypes = 256
/* Specification: 3.3. Alphabet sizes: insert-and-copy length */
const numLiteralSymbols = 256
const numCommandSymbols = 704
const numBlockLenSymbols = 26
const maxContextMapSymbols = (maxNumberOfBlockTypes + contextMapMaxRle)
const maxBlockTypeSymbols = (maxNumberOfBlockTypes + 2)
/* Specification: 3.5. Complex prefix codes */
const repeatPreviousCodeLength = 16
const repeatZeroCodeLength = 17
const codeLengthCodes = (repeatZeroCodeLength + 1)
/* "code length of 8 is repeated" */
const initialRepeatedCodeLength = 8
/* "Large Window Brotli" */
const largeMaxDistanceBits = 62
const largeMinWbits = 10
const largeMaxWbits = 30
/* Specification: 4. Encoding of distances */
const numDistanceShortCodes = 16
const maxNpostfix = 3
const maxNdirect = 120
const maxDistanceBits = 24
func distanceAlphabetSize(NPOSTFIX uint, NDIRECT uint, MAXNBITS uint) uint {
return numDistanceShortCodes + NDIRECT + uint(MAXNBITS<<(NPOSTFIX+1))
}
/* numDistanceSymbols == 1128 */
const numDistanceSymbols = 1128
const maxDistance = 0x3FFFFFC
const maxAllowedDistance = 0x7FFFFFFC
/* 7.1. Context modes and context ID lookup for literals */
/* "context IDs for literals are in the range of 0..63" */
const literalContextBits = 6
/* 7.2. Context ID for distances */
const distanceContextBits = 2
/* 9.1. Format of the Stream Header */
/* Number of slack bytes for window size. Don't confuse
with BROTLI_NUM_DISTANCE_SHORT_CODES. */
const windowGap = 16
func maxBackwardLimit(W uint) uint {
return (uint(1) << W) - windowGap
}
+2176
View File
File diff suppressed because it is too large. Load diff
+2581
View File
File diff suppressed because it is too large. Load diff
+122890
View File
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+1220
View File
File diff suppressed because it is too large. Load diff
+177
View File
@@ -0,0 +1,177 @@
package brotli
import "github.com/andybalholm/brotli/matchfinder"
// An Encoder implements the matchfinder.Encoder interface, writing in Brotli format.
type Encoder struct {
wroteHeader bool
bw bitWriter
distCache []distanceCode
}
func (e *Encoder) Reset() {
e.wroteHeader = false
e.bw = bitWriter{}
}
func (e *Encoder) Encode(dst []byte, src []byte, matches []matchfinder.Match, lastBlock bool) []byte {
e.bw.dst = dst
if !e.wroteHeader {
e.bw.writeBits(4, 15)
e.wroteHeader = true
}
if len(src) == 0 {
if lastBlock {
e.bw.writeBits(2, 3) // islast + isempty
e.bw.jumpToByteBoundary()
return e.bw.dst
}
return dst
}
var literalHisto [256]uint32
var commandHisto [704]uint32
var distanceHisto [64]uint32
literalCount := 0
commandCount := 0
distanceCount := 0
if len(e.distCache) < len(matches) {
e.distCache = make([]distanceCode, len(matches))
}
// first pass: build the histograms
pos := 0
// d is the ring buffer of the last 4 distances.
d := [4]int{-10, -10, -10, -10}
for i, m := range matches {
if m.Unmatched > 0 {
for _, c := range src[pos : pos+m.Unmatched] {
literalHisto[c]++
}
literalCount += m.Unmatched
}
insertCode := getInsertLengthCode(uint(m.Unmatched))
copyCode := getCopyLengthCode(uint(m.Length))
if m.Length == 0 {
// If the stream ends with unmatched bytes, we need a dummy copy length.
copyCode = 2
}
command := combineLengthCodes(insertCode, copyCode, false)
commandHisto[command]++
commandCount++
if command >= 128 && m.Length != 0 {
var distCode distanceCode
switch m.Distance {
case d[3]:
distCode.code = 0
case d[2]:
distCode.code = 1
case d[1]:
distCode.code = 2
case d[0]:
distCode.code = 3
case d[3] - 1:
distCode.code = 4
case d[3] + 1:
distCode.code = 5
case d[3] - 2:
distCode.code = 6
case d[3] + 2:
distCode.code = 7
case d[3] - 3:
distCode.code = 8
case d[3] + 3:
distCode.code = 9
// In my testing, codes 10–15 actually reduced the compression ratio.
default:
distCode = getDistanceCode(m.Distance)
}
e.distCache[i] = distCode
distanceHisto[distCode.code]++
distanceCount++
if distCode.code != 0 {
d[0], d[1], d[2], d[3] = d[1], d[2], d[3], m.Distance
}
}
pos += m.Unmatched + m.Length
}
storeMetaBlockHeaderBW(uint(len(src)), false, &e.bw)
e.bw.writeBits(13, 0)
var literalDepths [256]byte
var literalBits [256]uint16
buildAndStoreHuffmanTreeFastBW(literalHisto[:], uint(literalCount), 8, literalDepths[:], literalBits[:], &e.bw)
var commandDepths [704]byte
var commandBits [704]uint16
buildAndStoreHuffmanTreeFastBW(commandHisto[:], uint(commandCount), 10, commandDepths[:], commandBits[:], &e.bw)
var distanceDepths [64]byte
var distanceBits [64]uint16
buildAndStoreHuffmanTreeFastBW(distanceHisto[:], uint(distanceCount), 6, distanceDepths[:], distanceBits[:], &e.bw)
pos = 0
for i, m := range matches {
insertCode := getInsertLengthCode(uint(m.Unmatched))
copyCode := getCopyLengthCode(uint(m.Length))
if m.Length == 0 {
// If the stream ends with unmatched bytes, we need a dummy copy length.
copyCode = 2
}
command := combineLengthCodes(insertCode, copyCode, false)
e.bw.writeBits(uint(commandDepths[command]), uint64(commandBits[command]))
if kInsExtra[insertCode] > 0 {
e.bw.writeBits(uint(kInsExtra[insertCode]), uint64(m.Unmatched)-uint64(kInsBase[insertCode]))
}
if kCopyExtra[copyCode] > 0 {
e.bw.writeBits(uint(kCopyExtra[copyCode]), uint64(m.Length)-uint64(kCopyBase[copyCode]))
}
if m.Unmatched > 0 {
for _, c := range src[pos : pos+m.Unmatched] {
e.bw.writeBits(uint(literalDepths[c]), uint64(literalBits[c]))
}
}
if command >= 128 && m.Length != 0 {
distCode := e.distCache[i]
e.bw.writeBits(uint(distanceDepths[distCode.code]), uint64(distanceBits[distCode.code]))
if distCode.nExtra > 0 {
e.bw.writeBits(distCode.nExtra, distCode.extraBits)
}
}
pos += m.Unmatched + m.Length
}
if lastBlock {
e.bw.writeBits(2, 3) // islast + isempty
e.bw.jumpToByteBoundary()
}
return e.bw.dst
}
type distanceCode struct {
code int
nExtra uint
extraBits uint64
}
func getDistanceCode(distance int) distanceCode {
d := distance + 3
nbits := log2FloorNonZero(uint(d)) - 1
prefix := (d >> nbits) & 1
offset := (2 + prefix) << nbits
distcode := int(2*(nbits-1)) + prefix + 16
extra := d - offset
return distanceCode{distcode, uint(nbits), uint64(extra)}
}
+22
View File
@@ -0,0 +1,22 @@
package brotli
/* Dictionary data (words and transforms) for 1 possible context */
type encoderDictionary struct {
words *dictionary
cutoffTransformsCount uint32
cutoffTransforms uint64
hash_table []uint16
buckets []uint16
dict_words []dictWord
}
func initEncoderDictionary(dict *encoderDictionary) {
dict.words = getDictionary()
dict.hash_table = kStaticDictionaryHash[:]
dict.buckets = kStaticDictionaryBuckets[:]
dict.dict_words = kStaticDictionaryWords[:]
dict.cutoffTransformsCount = kCutoffTransformsCount
dict.cutoffTransforms = kCutoffTransforms
}
+592
View File
@@ -0,0 +1,592 @@
package brotli
import "math"
/* Copyright 2010 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Entropy encoding (Huffman) utilities. */
/* A node of a Huffman tree. */
type huffmanTree struct {
total_count_ uint32
index_left_ int16
index_right_or_value_ int16
}
func initHuffmanTree(self *huffmanTree, count uint32, left int16, right int16) {
self.total_count_ = count
self.index_left_ = left
self.index_right_or_value_ = right
}
/* Input size optimized Shell sort. */
type huffmanTreeComparator func(huffmanTree, huffmanTree) bool
var sortHuffmanTreeItems_gaps = []uint{132, 57, 23, 10, 4, 1}
func sortHuffmanTreeItems(items []huffmanTree, n uint, comparator huffmanTreeComparator) {
if n < 13 {
/* Insertion sort. */
var i uint
for i = 1; i < n; i++ {
var tmp huffmanTree = items[i]
var k uint = i
var j uint = i - 1
for comparator(tmp, items[j]) {
items[k] = items[j]
k = j
if j == 0 {
break
}
j--
}
items[k] = tmp
}
return
} else {
var g int
if n < 57 {
g = 2
} else {
g = 0
}
for ; g < 6; g++ {
var gap uint = sortHuffmanTreeItems_gaps[g]
var i uint
for i = gap; i < n; i++ {
var j uint = i
var tmp huffmanTree = items[i]
for ; j >= gap && comparator(tmp, items[j-gap]); j -= gap {
items[j] = items[j-gap]
}
items[j] = tmp
}
}
}
}
/* Returns 1 if assignment of depths succeeded, otherwise 0. */
func setDepth(p0 int, pool []huffmanTree, depth []byte, max_depth int) bool {
var stack [16]int
var level int = 0
var p int = p0
assert(max_depth <= 15)
stack[0] = -1
for {
if pool[p].index_left_ >= 0 {
level++
if level > max_depth {
return false
}
stack[level] = int(pool[p].index_right_or_value_)
p = int(pool[p].index_left_)
continue
} else {
depth[pool[p].index_right_or_value_] = byte(level)
}
for level >= 0 && stack[level] == -1 {
level--
}
if level < 0 {
return true
}
p = stack[level]
stack[level] = -1
}
}
/* Sort the root nodes, least popular first. */
func sortHuffmanTree(v0 huffmanTree, v1 huffmanTree) bool {
if v0.total_count_ != v1.total_count_ {
return v0.total_count_ < v1.total_count_
}
return v0.index_right_or_value_ > v1.index_right_or_value_
}
/* This function will create a Huffman tree.
The catch here is that the tree cannot be arbitrarily deep.
Brotli specifies a maximum depth of 15 bits for "code trees"
and 7 bits for "code length code trees."
count_limit is the value that is to be faked as the minimum value
and this minimum value is raised until the tree matches the
maximum length requirement.
This algorithm is not of excellent performance for very long data blocks,
especially when population counts are longer than 2**tree_limit, but
we are not planning to use this with extremely long blocks.
See http://en.wikipedia.org/wiki/Huffman_coding */
func createHuffmanTree(data []uint32, length uint, tree_limit int, tree []huffmanTree, depth []byte) {
var count_limit uint32
var sentinel huffmanTree
initHuffmanTree(&sentinel, math.MaxUint32, -1, -1)
/* For block sizes below 64 kB, we never need to do a second iteration
of this loop. Probably all of our block sizes will be smaller than
that, so this loop is mostly of academic interest. If we actually
would need this, we would be better off with the Katajainen algorithm. */
for count_limit = 1; ; count_limit *= 2 {
var n uint = 0
var i uint
var j uint
var k uint
for i = length; i != 0; {
i--
if data[i] != 0 {
var count uint32 = brotli_max_uint32_t(data[i], count_limit)
initHuffmanTree(&tree[n], count, -1, int16(i))
n++
}
}
if n == 1 {
depth[tree[0].index_right_or_value_] = 1 /* Only one element. */
break
}
sortHuffmanTreeItems(tree, n, huffmanTreeComparator(sortHuffmanTree))
/* The nodes are:
[0, n): the sorted leaf nodes that we start with.
[n]: we add a sentinel here.
[n + 1, 2n): new parent nodes are added here, starting from
(n+1). These are naturally in ascending order.
[2n]: we add a sentinel at the end as well.
There will be (2n+1) elements at the end. */
tree[n] = sentinel
tree[n+1] = sentinel
i = 0 /* Points to the next leaf node. */
j = n + 1 /* Points to the next non-leaf node. */
for k = n - 1; k != 0; k-- {
var left uint
var right uint
if tree[i].total_count_ <= tree[j].total_count_ {
left = i
i++
} else {
left = j
j++
}
if tree[i].total_count_ <= tree[j].total_count_ {
right = i
i++
} else {
right = j
j++
}
{
/* The sentinel node becomes the parent node. */
var j_end uint = 2*n - k
tree[j_end].total_count_ = tree[left].total_count_ + tree[right].total_count_
tree[j_end].index_left_ = int16(left)
tree[j_end].index_right_or_value_ = int16(right)
/* Add back the last sentinel node. */
tree[j_end+1] = sentinel
}
}
if setDepth(int(2*n-1), tree[0:], depth, tree_limit) {
/* We need to pack the Huffman tree in tree_limit bits. If this was not
successful, add fake entities to the lowest values and retry. */
break
}
}
}
func reverse(v []byte, start uint, end uint) {
end--
for start < end {
var tmp byte = v[start]
v[start] = v[end]
v[end] = tmp
start++
end--
}
}
func writeHuffmanTreeRepetitions(previous_value byte, value byte, repetitions uint, tree_size *uint, tree []byte, extra_bits_data []byte) {
assert(repetitions > 0)
if previous_value != value {
tree[*tree_size] = value
extra_bits_data[*tree_size] = 0
(*tree_size)++
repetitions--
}
if repetitions == 7 {
tree[*tree_size] = value
extra_bits_data[*tree_size] = 0
(*tree_size)++
repetitions--
}
if repetitions < 3 {
var i uint
for i = 0; i < repetitions; i++ {
tree[*tree_size] = value
extra_bits_data[*tree_size] = 0
(*tree_size)++
}
} else {
var start uint = *tree_size
repetitions -= 3
for {
tree[*tree_size] = repeatPreviousCodeLength
extra_bits_data[*tree_size] = byte(repetitions & 0x3)
(*tree_size)++
repetitions >>= 2
if repetitions == 0 {
break
}
repetitions--
}
reverse(tree, start, *tree_size)
reverse(extra_bits_data, start, *tree_size)
}
}
func writeHuffmanTreeRepetitionsZeros(repetitions uint, tree_size *uint, tree []byte, extra_bits_data []byte) {
if repetitions == 11 {
tree[*tree_size] = 0
extra_bits_data[*tree_size] = 0
(*tree_size)++
repetitions--
}
if repetitions < 3 {
var i uint
for i = 0; i < repetitions; i++ {
tree[*tree_size] = 0
extra_bits_data[*tree_size] = 0
(*tree_size)++
}
} else {
var start uint = *tree_size
repetitions -= 3
for {
tree[*tree_size] = repeatZeroCodeLength
extra_bits_data[*tree_size] = byte(repetitions & 0x7)
(*tree_size)++
repetitions >>= 3
if repetitions == 0 {
break
}
repetitions--
}
reverse(tree, start, *tree_size)
reverse(extra_bits_data, start, *tree_size)
}
}
/* Change the population counts in a way that the consequent
Huffman tree compression, especially its RLE-part will be more
likely to compress this data more efficiently.
length contains the size of the histogram.
counts contains the population counts.
good_for_rle is a buffer of at least length size */
func optimizeHuffmanCountsForRLE(length uint, counts []uint32, good_for_rle []byte) {
var nonzero_count uint = 0
var stride uint
var limit uint
var sum uint
var streak_limit uint = 1240
var i uint
/* Let's make the Huffman code more compatible with RLE encoding. */
for i = 0; i < length; i++ {
if counts[i] != 0 {
nonzero_count++
}
}
if nonzero_count < 16 {
return
}
for length != 0 && counts[length-1] == 0 {
length--
}
if length == 0 {
return /* All zeros. */
}
/* Now counts[0..length - 1] does not have trailing zeros. */
{
var nonzeros uint = 0
var smallest_nonzero uint32 = 1 << 30
for i = 0; i < length; i++ {
if counts[i] != 0 {
nonzeros++
if smallest_nonzero > counts[i] {
smallest_nonzero = counts[i]
}
}
}
if nonzeros < 5 {
/* Small histogram will model it well. */
return
}
if smallest_nonzero < 4 {
var zeros uint = length - nonzeros
if zeros < 6 {
for i = 1; i < length-1; i++ {
if counts[i-1] != 0 && counts[i] == 0 && counts[i+1] != 0 {
counts[i] = 1
}
}
}
}
if nonzeros < 28 {
return
}
}
/* 2) Let's mark all population counts that already can be encoded
with an RLE code. */
for i := 0; i < int(length); i++ {
good_for_rle[i] = 0
}
{
var symbol uint32 = counts[0]
/* Let's not spoil any of the existing good RLE codes.
Mark any seq of 0's that is longer as 5 as a good_for_rle.
Mark any seq of non-0's that is longer as 7 as a good_for_rle. */
var step uint = 0
for i = 0; i <= length; i++ {
if i == length || counts[i] != symbol {
if (symbol == 0 && step >= 5) || (symbol != 0 && step >= 7) {
var k uint
for k = 0; k < step; k++ {
good_for_rle[i-k-1] = 1
}
}
step = 1
if i != length {
symbol = counts[i]
}
} else {
step++
}
}
}
/* 3) Let's replace those population counts that lead to more RLE codes.
Math here is in 24.8 fixed point representation. */
stride = 0
limit = uint(256*(counts[0]+counts[1]+counts[2])/3 + 420)
sum = 0
for i = 0; i <= length; i++ {
if i == length || good_for_rle[i] != 0 || (i != 0 && good_for_rle[i-1] != 0) || (256*counts[i]-uint32(limit)+uint32(streak_limit)) >= uint32(2*streak_limit) {
if stride >= 4 || (stride >= 3 && sum == 0) {
var k uint
var count uint = (sum + stride/2) / stride
/* The stride must end, collapse what we have, if we have enough (4). */
if count == 0 {
count = 1
}
if sum == 0 {
/* Don't make an all zeros stride to be upgraded to ones. */
count = 0
}
for k = 0; k < stride; k++ {
/* We don't want to change value at counts[i],
that is already belonging to the next stride. Thus - 1. */
counts[i-k-1] = uint32(count)
}
}
stride = 0
sum = 0
if i < length-2 {
/* All interesting strides have a count of at least 4, */
/* at least when non-zeros. */
limit = uint(256*(counts[i]+counts[i+1]+counts[i+2])/3 + 420)
} else if i < length {
limit = uint(256 * counts[i])
} else {
limit = 0
}
}
stride++
if i != length {
sum += uint(counts[i])
if stride >= 4 {
limit = (256*sum + stride/2) / stride
}
if stride == 4 {
limit += 120
}
}
}
}
func decideOverRLEUse(depth []byte, length uint, use_rle_for_non_zero *bool, use_rle_for_zero *bool) {
var total_reps_zero uint = 0
var total_reps_non_zero uint = 0
var count_reps_zero uint = 1
var count_reps_non_zero uint = 1
var i uint
for i = 0; i < length; {
var value byte = depth[i]
var reps uint = 1
var k uint
for k = i + 1; k < length && depth[k] == value; k++ {
reps++
}
if reps >= 3 && value == 0 {
total_reps_zero += reps
count_reps_zero++
}
if reps >= 4 && value != 0 {
total_reps_non_zero += reps
count_reps_non_zero++
}
i += reps
}
*use_rle_for_non_zero = total_reps_non_zero > count_reps_non_zero*2
*use_rle_for_zero = total_reps_zero > count_reps_zero*2
}
/* Write a Huffman tree from bit depths into the bit-stream representation
of a Huffman tree. The generated Huffman tree is to be compressed once
more using a Huffman tree */
func writeHuffmanTree(depth []byte, length uint, tree_size *uint, tree []byte, extra_bits_data []byte) {
var previous_value byte = initialRepeatedCodeLength
var i uint
var use_rle_for_non_zero bool = false
var use_rle_for_zero bool = false
var new_length uint = length
/* Throw away trailing zeros. */
for i = 0; i < length; i++ {
if depth[length-i-1] == 0 {
new_length--
} else {
break
}
}
/* First gather statistics on if it is a good idea to do RLE. */
if length > 50 {
/* Find RLE coding for longer codes.
Shorter codes seem not to benefit from RLE. */
decideOverRLEUse(depth, new_length, &use_rle_for_non_zero, &use_rle_for_zero)
}
/* Actual RLE coding. */
for i = 0; i < new_length; {
var value byte = depth[i]
var reps uint = 1
if (value != 0 && use_rle_for_non_zero) || (value == 0 && use_rle_for_zero) {
var k uint
for k = i + 1; k < new_length && depth[k] == value; k++ {
reps++
}
}
if value == 0 {
writeHuffmanTreeRepetitionsZeros(reps, tree_size, tree, extra_bits_data)
} else {
writeHuffmanTreeRepetitions(previous_value, value, reps, tree_size, tree, extra_bits_data)
previous_value = value
}
i += reps
}
}
var reverseBits_kLut = [16]uint{
0x00,
0x08,
0x04,
0x0C,
0x02,
0x0A,
0x06,
0x0E,
0x01,
0x09,
0x05,
0x0D,
0x03,
0x0B,
0x07,
0x0F,
}
func reverseBits(num_bits uint, bits uint16) uint16 {
var retval uint = reverseBits_kLut[bits&0x0F]
var i uint
for i = 4; i < num_bits; i += 4 {
retval <<= 4
bits = uint16(bits >> 4)
retval |= reverseBits_kLut[bits&0x0F]
}
retval >>= ((0 - num_bits) & 0x03)
return uint16(retval)
}
/* 0..15 are values for bits */
const maxHuffmanBits = 16
/* Get the actual bit values for a tree of bit depths. */
func convertBitDepthsToSymbols(depth []byte, len uint, bits []uint16) {
var bl_count = [maxHuffmanBits]uint16{0}
var next_code [maxHuffmanBits]uint16
var i uint
/* In Brotli, all bit depths are [1..15]
0 bit depth means that the symbol does not exist. */
var code int = 0
for i = 0; i < len; i++ {
bl_count[depth[i]]++
}
bl_count[0] = 0
next_code[0] = 0
for i = 1; i < maxHuffmanBits; i++ {
code = (code + int(bl_count[i-1])) << 1
next_code[i] = uint16(code)
}
for i = 0; i < len; i++ {
if depth[i] != 0 {
bits[i] = reverseBits(uint(depth[i]), next_code[depth[i]])
next_code[depth[i]]++
}
}
}
File diff suppressed because it is too large. Load diff
+290
View File
@@ -0,0 +1,290 @@
package brotli
import (
"math"
"math/bits"
)
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Utilities for fast computation of logarithms. */
func log2FloorNonZero(n uint) uint32 {
return uint32(bits.Len(n)) - 1
}
/* A lookup table for small values of log2(int) to be used in entropy
computation.
", ".join(["%.16ff" % x for x in [0.0]+[log2(x) for x in range(1, 256)]]) */
var kLog2Table = []float32{
0.0000000000000000,
0.0000000000000000,
1.0000000000000000,
1.5849625007211563,
2.0000000000000000,
2.3219280948873622,
2.5849625007211561,
2.8073549220576042,
3.0000000000000000,
3.1699250014423126,
3.3219280948873626,
3.4594316186372978,
3.5849625007211565,
3.7004397181410922,
3.8073549220576037,
3.9068905956085187,
4.0000000000000000,
4.0874628412503400,
4.1699250014423122,
4.2479275134435852,
4.3219280948873626,
4.3923174227787607,
4.4594316186372973,
4.5235619560570131,
4.5849625007211570,
4.6438561897747244,
4.7004397181410926,
4.7548875021634691,
4.8073549220576037,
4.8579809951275728,
4.9068905956085187,
4.9541963103868758,
5.0000000000000000,
5.0443941193584534,
5.0874628412503400,
5.1292830169449664,
5.1699250014423122,
5.2094533656289501,
5.2479275134435852,
5.2854022188622487,
5.3219280948873626,
5.3575520046180838,
5.3923174227787607,
5.4262647547020979,
5.4594316186372973,
5.4918530963296748,
5.5235619560570131,
5.5545888516776376,
5.5849625007211570,
5.6147098441152083,
5.6438561897747244,
5.6724253419714961,
5.7004397181410926,
5.7279204545631996,
5.7548875021634691,
5.7813597135246599,
5.8073549220576046,
5.8328900141647422,
5.8579809951275719,
5.8826430493618416,
5.9068905956085187,
5.9307373375628867,
5.9541963103868758,
5.9772799234999168,
6.0000000000000000,
6.0223678130284544,
6.0443941193584534,
6.0660891904577721,
6.0874628412503400,
6.1085244567781700,
6.1292830169449672,
6.1497471195046822,
6.1699250014423122,
6.1898245588800176,
6.2094533656289510,
6.2288186904958804,
6.2479275134435861,
6.2667865406949019,
6.2854022188622487,
6.3037807481771031,
6.3219280948873617,
6.3398500028846252,
6.3575520046180847,
6.3750394313469254,
6.3923174227787598,
6.4093909361377026,
6.4262647547020979,
6.4429434958487288,
6.4594316186372982,
6.4757334309663976,
6.4918530963296748,
6.5077946401986964,
6.5235619560570131,
6.5391588111080319,
6.5545888516776376,
6.5698556083309478,
6.5849625007211561,
6.5999128421871278,
6.6147098441152092,
6.6293566200796095,
6.6438561897747253,
6.6582114827517955,
6.6724253419714952,
6.6865005271832185,
6.7004397181410917,
6.7142455176661224,
6.7279204545631988,
6.7414669864011465,
6.7548875021634691,
6.7681843247769260,
6.7813597135246599,
6.7944158663501062,
6.8073549220576037,
6.8201789624151887,
6.8328900141647422,
6.8454900509443757,
6.8579809951275719,
6.8703647195834048,
6.8826430493618416,
6.8948177633079437,
6.9068905956085187,
6.9188632372745955,
6.9307373375628867,
6.9425145053392399,
6.9541963103868758,
6.9657842846620879,
6.9772799234999168,
6.9886846867721664,
7.0000000000000000,
7.0112272554232540,
7.0223678130284544,
7.0334230015374501,
7.0443941193584534,
7.0552824355011898,
7.0660891904577721,
7.0768155970508317,
7.0874628412503400,
7.0980320829605272,
7.1085244567781700,
7.1189410727235076,
7.1292830169449664,
7.1395513523987937,
7.1497471195046822,
7.1598713367783891,
7.1699250014423130,
7.1799090900149345,
7.1898245588800176,
7.1996723448363644,
7.2094533656289492,
7.2191685204621621,
7.2288186904958804,
7.2384047393250794,
7.2479275134435861,
7.2573878426926521,
7.2667865406949019,
7.2761244052742384,
7.2854022188622487,
7.2946207488916270,
7.3037807481771031,
7.3128829552843557,
7.3219280948873617,
7.3309168781146177,
7.3398500028846243,
7.3487281542310781,
7.3575520046180847,
7.3663222142458151,
7.3750394313469254,
7.3837042924740528,
7.3923174227787607,
7.4008794362821844,
7.4093909361377026,
7.4178525148858991,
7.4262647547020979,
7.4346282276367255,
7.4429434958487288,
7.4512111118323299,
7.4594316186372973,
7.4676055500829976,
7.4757334309663976,
7.4838157772642564,
7.4918530963296748,
7.4998458870832057,
7.5077946401986964,
7.5156998382840436,
7.5235619560570131,
7.5313814605163119,
7.5391588111080319,
7.5468944598876373,
7.5545888516776376,
7.5622424242210728,
7.5698556083309478,
7.5774288280357487,
7.5849625007211561,
7.5924570372680806,
7.5999128421871278,
7.6073303137496113,
7.6147098441152075,
7.6220518194563764,
7.6293566200796095,
7.6366246205436488,
7.6438561897747244,
7.6510516911789290,
7.6582114827517955,
7.6653359171851765,
7.6724253419714952,
7.6794800995054464,
7.6865005271832185,
7.6934869574993252,
7.7004397181410926,
7.7073591320808825,
7.7142455176661224,
7.7210991887071856,
7.7279204545631996,
7.7347096202258392,
7.7414669864011465,
7.7481928495894596,
7.7548875021634691,
7.7615512324444795,
7.7681843247769260,
7.7747870596011737,
7.7813597135246608,
7.7879025593914317,
7.7944158663501062,
7.8008998999203047,
7.8073549220576037,
7.8137811912170374,
7.8201789624151887,
7.8265484872909159,
7.8328900141647422,
7.8392037880969445,
7.8454900509443757,
7.8517490414160571,
7.8579809951275719,
7.8641861446542798,
7.8703647195834048,
7.8765169465650002,
7.8826430493618425,
7.8887432488982601,
7.8948177633079446,
7.9008668079807496,
7.9068905956085187,
7.9128893362299619,
7.9188632372745955,
7.9248125036057813,
7.9307373375628867,
7.9366379390025719,
7.9425145053392399,
7.9483672315846778,
7.9541963103868758,
7.9600019320680806,
7.9657842846620870,
7.9715435539507720,
7.9772799234999168,
7.9829935746943104,
7.9886846867721664,
7.9943534368588578,
}
/* Faster logarithm for small integers, with the property of log2(0) == 0. */
func fastLog2(v uint) float64 {
if v < uint(len(kLog2Table)) {
return float64(kLog2Table[v])
}
return math.Log2(float64(v))
}
+45
View File
@@ -0,0 +1,45 @@
package brotli
import (
"encoding/binary"
"math/bits"
"runtime"
)
/* Copyright 2010 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Function to find maximal matching prefixes of strings. */
func findMatchLengthWithLimit(s1 []byte, s2 []byte, limit uint) uint {
var matched uint = 0
_, _ = s1[limit-1], s2[limit-1] // bounds check
switch runtime.GOARCH {
case "amd64":
// Compare 8 bytes at at time.
for matched+8 <= limit {
w1 := binary.LittleEndian.Uint64(s1[matched:])
w2 := binary.LittleEndian.Uint64(s2[matched:])
if w1 != w2 {
return matched + uint(bits.TrailingZeros64(w1^w2)>>3)
}
matched += 8
}
case "386":
// Compare 4 bytes at at time.
for matched+4 <= limit {
w1 := binary.LittleEndian.Uint32(s1[matched:])
w2 := binary.LittleEndian.Uint32(s2[matched:])
if w1 != w2 {
return matched + uint(bits.TrailingZeros32(w1^w2)>>3)
}
matched += 4
}
}
for matched < limit && s1[matched] == s2[matched] {
matched++
}
return matched
}
+287
View File
@@ -0,0 +1,287 @@
package brotli
import "encoding/binary"
/* Copyright 2016 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
func (*h10) HashTypeLength() uint {
return 4
}
func (*h10) StoreLookahead() uint {
return 128
}
func hashBytesH10(data []byte) uint32 {
var h uint32 = binary.LittleEndian.Uint32(data) * kHashMul32
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return h >> (32 - 17)
}
/* A (forgetful) hash table where each hash bucket contains a binary tree of
sequences whose first 4 bytes share the same hash code.
Each sequence is 128 long and is identified by its starting
position in the input data. The binary tree is sorted by the lexicographic
order of the sequences, and it is also a max-heap with respect to the
starting positions. */
type h10 struct {
hasherCommon
window_mask_ uint
buckets_ [1 << 17]uint32
invalid_pos_ uint32
forest []uint32
}
func (h *h10) Initialize(params *encoderParams) {
h.window_mask_ = (1 << params.lgwin) - 1
h.invalid_pos_ = uint32(0 - h.window_mask_)
var num_nodes uint = uint(1) << params.lgwin
h.forest = make([]uint32, 2*num_nodes)
}
func (h *h10) Prepare(one_shot bool, input_size uint, data []byte) {
var invalid_pos uint32 = h.invalid_pos_
var i uint32
for i = 0; i < 1<<17; i++ {
h.buckets_[i] = invalid_pos
}
}
func leftChildIndexH10(self *h10, pos uint) uint {
return 2 * (pos & self.window_mask_)
}
func rightChildIndexH10(self *h10, pos uint) uint {
return 2*(pos&self.window_mask_) + 1
}
/* Stores the hash of the next 4 bytes and in a single tree-traversal, the
hash bucket's binary tree is searched for matches and is re-rooted at the
current position.
If less than 128 data is available, the hash bucket of the
current position is searched for matches, but the state of the hash table
is not changed, since we can not know the final sorting order of the
current (incomplete) sequence.
This function must be called with increasing cur_ix positions. */
func storeAndFindMatchesH10(self *h10, data []byte, cur_ix uint, ring_buffer_mask uint, max_length uint, max_backward uint, best_len *uint, matches []backwardMatch) []backwardMatch {
var cur_ix_masked uint = cur_ix & ring_buffer_mask
var max_comp_len uint = brotli_min_size_t(max_length, 128)
var should_reroot_tree bool = (max_length >= 128)
var key uint32 = hashBytesH10(data[cur_ix_masked:])
var forest []uint32 = self.forest
var prev_ix uint = uint(self.buckets_[key])
var node_left uint = leftChildIndexH10(self, cur_ix)
var node_right uint = rightChildIndexH10(self, cur_ix)
var best_len_left uint = 0
var best_len_right uint = 0
var depth_remaining uint
/* The forest index of the rightmost node of the left subtree of the new
root, updated as we traverse and re-root the tree of the hash bucket. */
/* The forest index of the leftmost node of the right subtree of the new
root, updated as we traverse and re-root the tree of the hash bucket. */
/* The match length of the rightmost node of the left subtree of the new
root, updated as we traverse and re-root the tree of the hash bucket. */
/* The match length of the leftmost node of the right subtree of the new
root, updated as we traverse and re-root the tree of the hash bucket. */
if should_reroot_tree {
self.buckets_[key] = uint32(cur_ix)
}
for depth_remaining = 64; ; depth_remaining-- {
var backward uint = cur_ix - prev_ix
var prev_ix_masked uint = prev_ix & ring_buffer_mask
if backward == 0 || backward > max_backward || depth_remaining == 0 {
if should_reroot_tree {
forest[node_left] = self.invalid_pos_
forest[node_right] = self.invalid_pos_
}
break
}
{
var cur_len uint = brotli_min_size_t(best_len_left, best_len_right)
var len uint
assert(cur_len <= 128)
len = cur_len + findMatchLengthWithLimit(data[cur_ix_masked+cur_len:], data[prev_ix_masked+cur_len:], max_length-cur_len)
if matches != nil && len > *best_len {
*best_len = uint(len)
initBackwardMatch(&matches[0], backward, uint(len))
matches = matches[1:]
}
if len >= max_comp_len {
if should_reroot_tree {
forest[node_left] = forest[leftChildIndexH10(self, prev_ix)]
forest[node_right] = forest[rightChildIndexH10(self, prev_ix)]
}
break
}
if data[cur_ix_masked+len] > data[prev_ix_masked+len] {
best_len_left = uint(len)
if should_reroot_tree {
forest[node_left] = uint32(prev_ix)
}
node_left = rightChildIndexH10(self, prev_ix)
prev_ix = uint(forest[node_left])
} else {
best_len_right = uint(len)
if should_reroot_tree {
forest[node_right] = uint32(prev_ix)
}
node_right = leftChildIndexH10(self, prev_ix)
prev_ix = uint(forest[node_right])
}
}
}
return matches
}
/* Finds all backward matches of &data[cur_ix & ring_buffer_mask] up to the
length of max_length and stores the position cur_ix in the hash table.
Sets *num_matches to the number of matches found, and stores the found
matches in matches[0] to matches[*num_matches - 1]. The matches will be
sorted by strictly increasing length and (non-strictly) increasing
distance. */
func findAllMatchesH10(handle *h10, dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, cur_ix uint, max_length uint, max_backward uint, gap uint, params *encoderParams, matches []backwardMatch) uint {
var orig_matches []backwardMatch = matches
var cur_ix_masked uint = cur_ix & ring_buffer_mask
var best_len uint = 1
var short_match_max_backward uint
if params.quality != hqZopflificationQuality {
short_match_max_backward = 16
} else {
short_match_max_backward = 64
}
var stop uint = cur_ix - short_match_max_backward
var dict_matches [maxStaticDictionaryMatchLen + 1]uint32
var i uint
if cur_ix < short_match_max_backward {
stop = 0
}
for i = cur_ix - 1; i > stop && best_len <= 2; i-- {
var prev_ix uint = i
var backward uint = cur_ix - prev_ix
if backward > max_backward {
break
}
prev_ix &= ring_buffer_mask
if data[cur_ix_masked] != data[prev_ix] || data[cur_ix_masked+1] != data[prev_ix+1] {
continue
}
{
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
if len > best_len {
best_len = uint(len)
initBackwardMatch(&matches[0], backward, uint(len))
matches = matches[1:]
}
}
}
if best_len < max_length {
matches = storeAndFindMatchesH10(handle, data, cur_ix, ring_buffer_mask, max_length, max_backward, &best_len, matches)
}
for i = 0; i <= maxStaticDictionaryMatchLen; i++ {
dict_matches[i] = kInvalidMatch
}
{
var minlen uint = brotli_max_size_t(4, best_len+1)
if findAllStaticDictionaryMatches(dictionary, data[cur_ix_masked:], minlen, max_length, dict_matches[0:]) {
var maxlen uint = brotli_min_size_t(maxStaticDictionaryMatchLen, max_length)
var l uint
for l = minlen; l <= maxlen; l++ {
var dict_id uint32 = dict_matches[l]
if dict_id < kInvalidMatch {
var distance uint = max_backward + gap + uint(dict_id>>5) + 1
if distance <= params.dist.max_distance {
initDictionaryBackwardMatch(&matches[0], distance, l, uint(dict_id&31))
matches = matches[1:]
}
}
}
}
}
return uint(-cap(matches) + cap(orig_matches))
}
/* Stores the hash of the next 4 bytes and re-roots the binary tree at the
current sequence, without returning any matches.
REQUIRES: ix + 128 <= end-of-current-block */
func (h *h10) Store(data []byte, mask uint, ix uint) {
var max_backward uint = h.window_mask_ - windowGap + 1
/* Maximum distance is window size - 16, see section 9.1. of the spec. */
storeAndFindMatchesH10(h, data, ix, mask, 128, max_backward, nil, nil)
}
func (h *h10) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
var i uint = ix_start
var j uint = ix_start
if ix_start+63 <= ix_end {
i = ix_end - 63
}
if ix_start+512 <= i {
for ; j < i; j += 8 {
h.Store(data, mask, j)
}
}
for ; i < ix_end; i++ {
h.Store(data, mask, i)
}
}
func (h *h10) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint) {
if num_bytes >= h.HashTypeLength()-1 && position >= 128 {
var i_start uint = position - 128 + 1
var i_end uint = brotli_min_size_t(position, i_start+num_bytes)
/* Store the last `128 - 1` positions in the hasher.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
var i uint
for i = i_start; i < i_end; i++ {
/* Maximum distance is window size - 16, see section 9.1. of the spec.
Furthermore, we have to make sure that we don't look further back
from the start of the next block than the window size, otherwise we
could access already overwritten areas of the ring-buffer. */
var max_backward uint = h.window_mask_ - brotli_max_size_t(windowGap-1, position-i)
/* We know that i + 128 <= position + num_bytes, i.e. the
end of the current block and that we have at least
128 tail in the ring-buffer. */
storeAndFindMatchesH10(h, ringbuffer, i, ringbuffer_mask, 128, max_backward, nil, nil)
}
}
}
/* MAX_NUM_MATCHES == 64 + MAX_TREE_SEARCH_DEPTH */
const maxNumMatchesH10 = 128
func (*h10) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
panic("unimplemented")
}
func (*h10) PrepareDistanceCache(distance_cache []int) {
panic("unimplemented")
}
+214
View File
@@ -0,0 +1,214 @@
package brotli
import "encoding/binary"
/* Copyright 2010 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* A (forgetful) hash table to the data seen by the compressor, to
help create backward references to previous data.
This is a hash map of fixed size (bucket_size_) to a ring buffer of
fixed size (block_size_). The ring buffer contains the last block_size_
index positions of the given hash key in the compressed data. */
func (*h5) HashTypeLength() uint {
return 4
}
func (*h5) StoreLookahead() uint {
return 4
}
/* HashBytes is the function that chooses the bucket to place the address in. */
func hashBytesH5(data []byte, shift int) uint32 {
var h uint32 = binary.LittleEndian.Uint32(data) * kHashMul32
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return uint32(h >> uint(shift))
}
type h5 struct {
hasherCommon
bucket_size_ uint
block_size_ uint
hash_shift_ int
block_mask_ uint32
num []uint16
buckets []uint32
}
func (h *h5) Initialize(params *encoderParams) {
h.hash_shift_ = 32 - h.params.bucket_bits
h.bucket_size_ = uint(1) << uint(h.params.bucket_bits)
h.block_size_ = uint(1) << uint(h.params.block_bits)
h.block_mask_ = uint32(h.block_size_ - 1)
h.num = make([]uint16, h.bucket_size_)
h.buckets = make([]uint32, h.block_size_*h.bucket_size_)
}
func (h *h5) Prepare(one_shot bool, input_size uint, data []byte) {
var num []uint16 = h.num
var partial_prepare_threshold uint = h.bucket_size_ >> 6
/* Partial preparation is 100 times slower (per socket). */
if one_shot && input_size <= partial_prepare_threshold {
var i uint
for i = 0; i < input_size; i++ {
var key uint32 = hashBytesH5(data[i:], h.hash_shift_)
num[key] = 0
}
} else {
for i := 0; i < int(h.bucket_size_); i++ {
num[i] = 0
}
}
}
/* Look at 4 bytes at &data[ix & mask].
Compute a hash from these, and store the value of ix at that position. */
func (h *h5) Store(data []byte, mask uint, ix uint) {
var num []uint16 = h.num
var key uint32 = hashBytesH5(data[ix&mask:], h.hash_shift_)
var minor_ix uint = uint(num[key]) & uint(h.block_mask_)
var offset uint = minor_ix + uint(key<<uint(h.params.block_bits))
h.buckets[offset] = uint32(ix)
num[key]++
}
func (h *h5) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
var i uint
for i = ix_start; i < ix_end; i++ {
h.Store(data, mask, i)
}
}
func (h *h5) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint) {
if num_bytes >= h.HashTypeLength()-1 && position >= 3 {
/* Prepare the hashes for three last bytes of the last write.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
h.Store(ringbuffer, ringbuffer_mask, position-3)
h.Store(ringbuffer, ringbuffer_mask, position-2)
h.Store(ringbuffer, ringbuffer_mask, position-1)
}
}
func (h *h5) PrepareDistanceCache(distance_cache []int) {
prepareDistanceCache(distance_cache, h.params.num_last_distances_to_check)
}
/* Find a longest backward match of &data[cur_ix] up to the length of
max_length and stores the position cur_ix in the hash table.
REQUIRES: PrepareDistanceCacheH5 must be invoked for current distance cache
values; if this method is invoked repeatedly with the same distance
cache values, it is enough to invoke PrepareDistanceCacheH5 once.
Does not look for matches longer than max_length.
Does not look for matches further away than max_backward.
Writes the best match into |out|.
|out|->score is updated only if a better match is found. */
func (h *h5) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
var num []uint16 = h.num
var buckets []uint32 = h.buckets
var cur_ix_masked uint = cur_ix & ring_buffer_mask
var min_score uint = out.score
var best_score uint = out.score
var best_len uint = out.len
var i uint
var bucket []uint32
/* Don't accept a short copy from far away. */
out.len = 0
out.len_code_delta = 0
/* Try last distance first. */
for i = 0; i < uint(h.params.num_last_distances_to_check); i++ {
var backward uint = uint(distance_cache[i])
var prev_ix uint = uint(cur_ix - backward)
if prev_ix >= cur_ix {
continue
}
if backward > max_backward {
continue
}
prev_ix &= ring_buffer_mask
if cur_ix_masked+best_len > ring_buffer_mask || prev_ix+best_len > ring_buffer_mask || data[cur_ix_masked+best_len] != data[prev_ix+best_len] {
continue
}
{
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
if len >= 3 || (len == 2 && i < 2) {
/* Comparing for >= 2 does not change the semantics, but just saves for
a few unnecessary binary logarithms in backward reference score,
since we are not interested in such short matches. */
var score uint = backwardReferenceScoreUsingLastDistance(uint(len))
if best_score < score {
if i != 0 {
score -= backwardReferencePenaltyUsingLastDistance(i)
}
if best_score < score {
best_score = score
best_len = uint(len)
out.len = best_len
out.distance = backward
out.score = best_score
}
}
}
}
}
{
var key uint32 = hashBytesH5(data[cur_ix_masked:], h.hash_shift_)
bucket = buckets[key<<uint(h.params.block_bits):]
var down uint
if uint(num[key]) > h.block_size_ {
down = uint(num[key]) - h.block_size_
} else {
down = 0
}
for i = uint(num[key]); i > down; {
var prev_ix uint
i--
prev_ix = uint(bucket[uint32(i)&h.block_mask_])
var backward uint = cur_ix - prev_ix
if backward > max_backward {
break
}
prev_ix &= ring_buffer_mask
if cur_ix_masked+best_len > ring_buffer_mask || prev_ix+best_len > ring_buffer_mask || data[cur_ix_masked+best_len] != data[prev_ix+best_len] {
continue
}
{
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
if len >= 4 {
/* Comparing for >= 3 does not change the semantics, but just saves
for a few unnecessary binary logarithms in backward reference
score, since we are not interested in such short matches. */
var score uint = backwardReferenceScore(uint(len), backward)
if best_score < score {
best_score = score
best_len = uint(len)
out.len = best_len
out.distance = backward
out.score = best_score
}
}
}
}
bucket[uint32(num[key])&h.block_mask_] = uint32(cur_ix)
num[key]++
}
if min_score == out.score {
searchInStaticDictionary(dictionary, h, data[cur_ix_masked:], max_length, max_backward+gap, max_distance, out, false)
}
}
+216
View File
@@ -0,0 +1,216 @@
package brotli
import "encoding/binary"
/* Copyright 2010 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* A (forgetful) hash table to the data seen by the compressor, to
help create backward references to previous data.
This is a hash map of fixed size (bucket_size_) to a ring buffer of
fixed size (block_size_). The ring buffer contains the last block_size_
index positions of the given hash key in the compressed data. */
func (*h6) HashTypeLength() uint {
return 8
}
func (*h6) StoreLookahead() uint {
return 8
}
/* HashBytes is the function that chooses the bucket to place the address in. */
func hashBytesH6(data []byte, mask uint64, shift int) uint32 {
var h uint64 = (binary.LittleEndian.Uint64(data) & mask) * kHashMul64Long
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return uint32(h >> uint(shift))
}
type h6 struct {
hasherCommon
bucket_size_ uint
block_size_ uint
hash_shift_ int
hash_mask_ uint64
block_mask_ uint32
num []uint16
buckets []uint32
}
func (h *h6) Initialize(params *encoderParams) {
h.hash_shift_ = 64 - h.params.bucket_bits
h.hash_mask_ = (^(uint64(0))) >> uint(64-8*h.params.hash_len)
h.bucket_size_ = uint(1) << uint(h.params.bucket_bits)
h.block_size_ = uint(1) << uint(h.params.block_bits)
h.block_mask_ = uint32(h.block_size_ - 1)
h.num = make([]uint16, h.bucket_size_)
h.buckets = make([]uint32, h.block_size_*h.bucket_size_)
}
func (h *h6) Prepare(one_shot bool, input_size uint, data []byte) {
var num []uint16 = h.num
var partial_prepare_threshold uint = h.bucket_size_ >> 6
/* Partial preparation is 100 times slower (per socket). */
if one_shot && input_size <= partial_prepare_threshold {
var i uint
for i = 0; i < input_size; i++ {
var key uint32 = hashBytesH6(data[i:], h.hash_mask_, h.hash_shift_)
num[key] = 0
}
} else {
for i := 0; i < int(h.bucket_size_); i++ {
num[i] = 0
}
}
}
/* Look at 4 bytes at &data[ix & mask].
Compute a hash from these, and store the value of ix at that position. */
func (h *h6) Store(data []byte, mask uint, ix uint) {
var num []uint16 = h.num
var key uint32 = hashBytesH6(data[ix&mask:], h.hash_mask_, h.hash_shift_)
var minor_ix uint = uint(num[key]) & uint(h.block_mask_)
var offset uint = minor_ix + uint(key<<uint(h.params.block_bits))
h.buckets[offset] = uint32(ix)
num[key]++
}
func (h *h6) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
var i uint
for i = ix_start; i < ix_end; i++ {
h.Store(data, mask, i)
}
}
func (h *h6) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint) {
if num_bytes >= h.HashTypeLength()-1 && position >= 3 {
/* Prepare the hashes for three last bytes of the last write.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
h.Store(ringbuffer, ringbuffer_mask, position-3)
h.Store(ringbuffer, ringbuffer_mask, position-2)
h.Store(ringbuffer, ringbuffer_mask, position-1)
}
}
func (h *h6) PrepareDistanceCache(distance_cache []int) {
prepareDistanceCache(distance_cache, h.params.num_last_distances_to_check)
}
/* Find a longest backward match of &data[cur_ix] up to the length of
max_length and stores the position cur_ix in the hash table.
REQUIRES: PrepareDistanceCacheH6 must be invoked for current distance cache
values; if this method is invoked repeatedly with the same distance
cache values, it is enough to invoke PrepareDistanceCacheH6 once.
Does not look for matches longer than max_length.
Does not look for matches further away than max_backward.
Writes the best match into |out|.
|out|->score is updated only if a better match is found. */
func (h *h6) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
var num []uint16 = h.num
var buckets []uint32 = h.buckets
var cur_ix_masked uint = cur_ix & ring_buffer_mask
var min_score uint = out.score
var best_score uint = out.score
var best_len uint = out.len
var i uint
var bucket []uint32
/* Don't accept a short copy from far away. */
out.len = 0
out.len_code_delta = 0
/* Try last distance first. */
for i = 0; i < uint(h.params.num_last_distances_to_check); i++ {
var backward uint = uint(distance_cache[i])
var prev_ix uint = uint(cur_ix - backward)
if prev_ix >= cur_ix {
continue
}
if backward > max_backward {
continue
}
prev_ix &= ring_buffer_mask
if cur_ix_masked+best_len > ring_buffer_mask || prev_ix+best_len > ring_buffer_mask || data[cur_ix_masked+best_len] != data[prev_ix+best_len] {
continue
}
{
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
if len >= 3 || (len == 2 && i < 2) {
/* Comparing for >= 2 does not change the semantics, but just saves for
a few unnecessary binary logarithms in backward reference score,
since we are not interested in such short matches. */
var score uint = backwardReferenceScoreUsingLastDistance(uint(len))
if best_score < score {
if i != 0 {
score -= backwardReferencePenaltyUsingLastDistance(i)
}
if best_score < score {
best_score = score
best_len = uint(len)
out.len = best_len
out.distance = backward
out.score = best_score
}
}
}
}
}
{
var key uint32 = hashBytesH6(data[cur_ix_masked:], h.hash_mask_, h.hash_shift_)
bucket = buckets[key<<uint(h.params.block_bits):]
var down uint
if uint(num[key]) > h.block_size_ {
down = uint(num[key]) - h.block_size_
} else {
down = 0
}
for i = uint(num[key]); i > down; {
var prev_ix uint
i--
prev_ix = uint(bucket[uint32(i)&h.block_mask_])
var backward uint = cur_ix - prev_ix
if backward > max_backward {
break
}
prev_ix &= ring_buffer_mask
if cur_ix_masked+best_len > ring_buffer_mask || prev_ix+best_len > ring_buffer_mask || data[cur_ix_masked+best_len] != data[prev_ix+best_len] {
continue
}
{
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
if len >= 4 {
/* Comparing for >= 3 does not change the semantics, but just saves
for a few unnecessary binary logarithms in backward reference
score, since we are not interested in such short matches. */
var score uint = backwardReferenceScore(uint(len), backward)
if best_score < score {
best_score = score
best_len = uint(len)
out.len = best_len
out.distance = backward
out.score = best_score
}
}
}
}
bucket[uint32(num[key])&h.block_mask_] = uint32(cur_ix)
num[key]++
}
if min_score == out.score {
searchInStaticDictionary(dictionary, h, data[cur_ix_masked:], max_length, max_backward+gap, max_distance, out, false)
}
}
+342
View File
@@ -0,0 +1,342 @@
package brotli
import (
"encoding/binary"
"fmt"
)
type hasherCommon struct {
params hasherParams
is_prepared_ bool
dict_num_lookups uint
dict_num_matches uint
}
func (h *hasherCommon) Common() *hasherCommon {
return h
}
type hasherHandle interface {
Common() *hasherCommon
Initialize(params *encoderParams)
Prepare(one_shot bool, input_size uint, data []byte)
StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint)
HashTypeLength() uint
StoreLookahead() uint
PrepareDistanceCache(distance_cache []int)
FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult)
StoreRange(data []byte, mask uint, ix_start uint, ix_end uint)
Store(data []byte, mask uint, ix uint)
}
const kCutoffTransformsCount uint32 = 10
/* 0, 12, 27, 23, 42, 63, 56, 48, 59, 64 */
/* 0+0, 4+8, 8+19, 12+11, 16+26, 20+43, 24+32, 28+20, 32+27, 36+28 */
const kCutoffTransforms uint64 = 0x071B520ADA2D3200
type hasherSearchResult struct {
len uint
distance uint
score uint
len_code_delta int
}
/* kHashMul32 multiplier has these properties:
* The multiplier must be odd. Otherwise we may lose the highest bit.
* No long streaks of ones or zeros.
* There is no effort to ensure that it is a prime, the oddity is enough
for this use.
* The number has been tuned heuristically against compression benchmarks. */
const kHashMul32 uint32 = 0x1E35A7BD
const kHashMul64 uint64 = 0x1E35A7BD1E35A7BD
const kHashMul64Long uint64 = 0x1FE35A7BD3579BD3
func hash14(data []byte) uint32 {
var h uint32 = binary.LittleEndian.Uint32(data) * kHashMul32
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return h >> (32 - 14)
}
func prepareDistanceCache(distance_cache []int, num_distances int) {
if num_distances > 4 {
var last_distance int = distance_cache[0]
distance_cache[4] = last_distance - 1
distance_cache[5] = last_distance + 1
distance_cache[6] = last_distance - 2
distance_cache[7] = last_distance + 2
distance_cache[8] = last_distance - 3
distance_cache[9] = last_distance + 3
if num_distances > 10 {
var next_last_distance int = distance_cache[1]
distance_cache[10] = next_last_distance - 1
distance_cache[11] = next_last_distance + 1
distance_cache[12] = next_last_distance - 2
distance_cache[13] = next_last_distance + 2
distance_cache[14] = next_last_distance - 3
distance_cache[15] = next_last_distance + 3
}
}
}
const literalByteScore = 135
const distanceBitPenalty = 30
/* Score must be positive after applying maximal penalty. */
const scoreBase = (distanceBitPenalty * 8 * 8)
/* Usually, we always choose the longest backward reference. This function
allows for the exception of that rule.
If we choose a backward reference that is further away, it will
usually be coded with more bits. We approximate this by assuming
log2(distance). If the distance can be expressed in terms of the
last four distances, we use some heuristic constants to estimate
the bits cost. For the first up to four literals we use the bit
cost of the literals from the literal cost model, after that we
use the average bit cost of the cost model.
This function is used to sometimes discard a longer backward reference
when it is not much longer and the bit cost for encoding it is more
than the saved literals.
backward_reference_offset MUST be positive. */
func backwardReferenceScore(copy_length uint, backward_reference_offset uint) uint {
return scoreBase + literalByteScore*uint(copy_length) - distanceBitPenalty*uint(log2FloorNonZero(backward_reference_offset))
}
func backwardReferenceScoreUsingLastDistance(copy_length uint) uint {
return literalByteScore*uint(copy_length) + scoreBase + 15
}
func backwardReferencePenaltyUsingLastDistance(distance_short_code uint) uint {
return uint(39) + ((0x1CA10 >> (distance_short_code & 0xE)) & 0xE)
}
func testStaticDictionaryItem(dictionary *encoderDictionary, item uint, data []byte, max_length uint, max_backward uint, max_distance uint, out *hasherSearchResult) bool {
var len uint
var word_idx uint
var offset uint
var matchlen uint
var backward uint
var score uint
len = item & 0x1F
word_idx = item >> 5
offset = uint(dictionary.words.offsets_by_length[len]) + len*word_idx
if len > max_length {
return false
}
matchlen = findMatchLengthWithLimit(data, dictionary.words.data[offset:], uint(len))
if matchlen+uint(dictionary.cutoffTransformsCount) <= len || matchlen == 0 {
return false
}
{
var cut uint = len - matchlen
var transform_id uint = (cut << 2) + uint((dictionary.cutoffTransforms>>(cut*6))&0x3F)
backward = max_backward + 1 + word_idx + (transform_id << dictionary.words.size_bits_by_length[len])
}
if backward > max_distance {
return false
}
score = backwardReferenceScore(matchlen, backward)
if score < out.score {
return false
}
out.len = matchlen
out.len_code_delta = int(len) - int(matchlen)
out.distance = backward
out.score = score
return true
}
func searchInStaticDictionary(dictionary *encoderDictionary, handle hasherHandle, data []byte, max_length uint, max_backward uint, max_distance uint, out *hasherSearchResult, shallow bool) {
var key uint
var i uint
var self *hasherCommon = handle.Common()
if self.dict_num_matches < self.dict_num_lookups>>7 {
return
}
key = uint(hash14(data) << 1)
for i = 0; ; (func() { i++; key++ })() {
var tmp uint
if shallow {
tmp = 1
} else {
tmp = 2
}
if i >= tmp {
break
}
var item uint = uint(dictionary.hash_table[key])
self.dict_num_lookups++
if item != 0 {
var item_matches bool = testStaticDictionaryItem(dictionary, item, data, max_length, max_backward, max_distance, out)
if item_matches {
self.dict_num_matches++
}
}
}
}
type backwardMatch struct {
distance uint32
length_and_code uint32
}
func initBackwardMatch(self *backwardMatch, dist uint, len uint) {
self.distance = uint32(dist)
self.length_and_code = uint32(len << 5)
}
func initDictionaryBackwardMatch(self *backwardMatch, dist uint, len uint, len_code uint) {
self.distance = uint32(dist)
var tmp uint
if len == len_code {
tmp = 0
} else {
tmp = len_code
}
self.length_and_code = uint32(len<<5 | tmp)
}
func backwardMatchLength(self *backwardMatch) uint {
return uint(self.length_and_code >> 5)
}
func backwardMatchLengthCode(self *backwardMatch) uint {
var code uint = uint(self.length_and_code) & 31
if code != 0 {
return code
} else {
return backwardMatchLength(self)
}
}
func hasherReset(handle hasherHandle) {
if handle == nil {
return
}
handle.Common().is_prepared_ = false
}
func newHasher(typ int) hasherHandle {
switch typ {
case 2:
return &hashLongestMatchQuickly{
bucketBits: 16,
bucketSweep: 1,
hashLen: 5,
useDictionary: true,
}
case 3:
return &hashLongestMatchQuickly{
bucketBits: 16,
bucketSweep: 2,
hashLen: 5,
useDictionary: false,
}
case 4:
return &hashLongestMatchQuickly{
bucketBits: 17,
bucketSweep: 4,
hashLen: 5,
useDictionary: true,
}
case 5:
return new(h5)
case 6:
return new(h6)
case 10:
return new(h10)
case 35:
return &hashComposite{
ha: newHasher(3),
hb: &hashRolling{jump: 4},
}
case 40:
return &hashForgetfulChain{
bucketBits: 15,
numBanks: 1,
bankBits: 16,
numLastDistancesToCheck: 4,
}
case 41:
return &hashForgetfulChain{
bucketBits: 15,
numBanks: 1,
bankBits: 16,
numLastDistancesToCheck: 10,
}
case 42:
return &hashForgetfulChain{
bucketBits: 15,
numBanks: 512,
bankBits: 9,
numLastDistancesToCheck: 16,
}
case 54:
return &hashLongestMatchQuickly{
bucketBits: 20,
bucketSweep: 4,
hashLen: 7,
useDictionary: false,
}
case 55:
return &hashComposite{
ha: newHasher(54),
hb: &hashRolling{jump: 4},
}
case 65:
return &hashComposite{
ha: newHasher(6),
hb: &hashRolling{jump: 1},
}
}
panic(fmt.Sprintf("unknown hasher type: %d", typ))
}
func hasherSetup(handle *hasherHandle, params *encoderParams, data []byte, position uint, input_size uint, is_last bool) {
var self hasherHandle = nil
var common *hasherCommon = nil
var one_shot bool = (position == 0 && is_last)
if *handle == nil {
chooseHasher(params, &params.hasher)
self = newHasher(params.hasher.type_)
*handle = self
common = self.Common()
common.params = params.hasher
self.Initialize(params)
}
self = *handle
common = self.Common()
if !common.is_prepared_ {
self.Prepare(one_shot, input_size, data)
if position == 0 {
common.dict_num_lookups = 0
common.dict_num_matches = 0
}
common.is_prepared_ = true
}
}
func initOrStitchToPreviousBlock(handle *hasherHandle, data []byte, mask uint, params *encoderParams, position uint, input_size uint, is_last bool) {
var self hasherHandle
hasherSetup(handle, params, data, position, input_size, is_last)
self = *handle
self.StitchToPreviousBlock(input_size, position, data, mask)
}
+93
View File
@@ -0,0 +1,93 @@
package brotli
/* Copyright 2018 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
func (h *hashComposite) HashTypeLength() uint {
var a uint = h.ha.HashTypeLength()
var b uint = h.hb.HashTypeLength()
if a > b {
return a
} else {
return b
}
}
func (h *hashComposite) StoreLookahead() uint {
var a uint = h.ha.StoreLookahead()
var b uint = h.hb.StoreLookahead()
if a > b {
return a
} else {
return b
}
}
/* Composite hasher: This hasher allows to combine two other hashers, HASHER_A
and HASHER_B. */
type hashComposite struct {
hasherCommon
ha hasherHandle
hb hasherHandle
params *encoderParams
}
func (h *hashComposite) Initialize(params *encoderParams) {
h.params = params
}
/* TODO: Initialize of the hashers is defered to Prepare (and params
remembered here) because we don't get the one_shot and input_size params
here that are needed to know the memory size of them. Instead provide
those params to all hashers InitializehashComposite */
func (h *hashComposite) Prepare(one_shot bool, input_size uint, data []byte) {
if h.ha == nil {
var common_a *hasherCommon
var common_b *hasherCommon
common_a = h.ha.Common()
common_a.params = h.params.hasher
common_a.is_prepared_ = false
common_a.dict_num_lookups = 0
common_a.dict_num_matches = 0
h.ha.Initialize(h.params)
common_b = h.hb.Common()
common_b.params = h.params.hasher
common_b.is_prepared_ = false
common_b.dict_num_lookups = 0
common_b.dict_num_matches = 0
h.hb.Initialize(h.params)
}
h.ha.Prepare(one_shot, input_size, data)
h.hb.Prepare(one_shot, input_size, data)
}
func (h *hashComposite) Store(data []byte, mask uint, ix uint) {
h.ha.Store(data, mask, ix)
h.hb.Store(data, mask, ix)
}
func (h *hashComposite) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
h.ha.StoreRange(data, mask, ix_start, ix_end)
h.hb.StoreRange(data, mask, ix_start, ix_end)
}
func (h *hashComposite) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ring_buffer_mask uint) {
h.ha.StitchToPreviousBlock(num_bytes, position, ringbuffer, ring_buffer_mask)
h.hb.StitchToPreviousBlock(num_bytes, position, ringbuffer, ring_buffer_mask)
}
func (h *hashComposite) PrepareDistanceCache(distance_cache []int) {
h.ha.PrepareDistanceCache(distance_cache)
h.hb.PrepareDistanceCache(distance_cache)
}
func (h *hashComposite) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
h.ha.FindLongestMatch(dictionary, data, ring_buffer_mask, distance_cache, cur_ix, max_length, max_backward, gap, max_distance, out)
h.hb.FindLongestMatch(dictionary, data, ring_buffer_mask, distance_cache, cur_ix, max_length, max_backward, gap, max_distance, out)
}
+252
View File
@@ -0,0 +1,252 @@
package brotli
import "encoding/binary"
/* Copyright 2016 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
func (*hashForgetfulChain) HashTypeLength() uint {
return 4
}
func (*hashForgetfulChain) StoreLookahead() uint {
return 4
}
/* HashBytes is the function that chooses the bucket to place the address in.*/
func (h *hashForgetfulChain) HashBytes(data []byte) uint {
var hash uint32 = binary.LittleEndian.Uint32(data) * kHashMul32
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return uint(hash >> (32 - h.bucketBits))
}
type slot struct {
delta uint16
next uint16
}
/* A (forgetful) hash table to the data seen by the compressor, to
help create backward references to previous data.
Hashes are stored in chains which are bucketed to groups. Group of chains
share a storage "bank". When more than "bank size" chain nodes are added,
oldest nodes are replaced; this way several chains may share a tail. */
type hashForgetfulChain struct {
hasherCommon
bucketBits uint
numBanks uint
bankBits uint
numLastDistancesToCheck int
addr []uint32
head []uint16
tiny_hash [65536]byte
banks [][]slot
free_slot_idx []uint16
max_hops uint
}
func (h *hashForgetfulChain) Initialize(params *encoderParams) {
var q uint
if params.quality > 6 {
q = 7
} else {
q = 8
}
h.max_hops = q << uint(params.quality-4)
bankSize := 1 << h.bankBits
bucketSize := 1 << h.bucketBits
h.addr = make([]uint32, bucketSize)
h.head = make([]uint16, bucketSize)
h.banks = make([][]slot, h.numBanks)
for i := range h.banks {
h.banks[i] = make([]slot, bankSize)
}
h.free_slot_idx = make([]uint16, h.numBanks)
}
func (h *hashForgetfulChain) Prepare(one_shot bool, input_size uint, data []byte) {
var partial_prepare_threshold uint = (1 << h.bucketBits) >> 6
/* Partial preparation is 100 times slower (per socket). */
if one_shot && input_size <= partial_prepare_threshold {
var i uint
for i = 0; i < input_size; i++ {
var bucket uint = h.HashBytes(data[i:])
/* See InitEmpty comment. */
h.addr[bucket] = 0xCCCCCCCC
h.head[bucket] = 0xCCCC
}
} else {
/* Fill |addr| array with 0xCCCCCCCC value. Because of wrapping, position
processed by hasher never reaches 3GB + 64M; this makes all new chains
to be terminated after the first node. */
for i := range h.addr {
h.addr[i] = 0xCCCCCCCC
}
for i := range h.head {
h.head[i] = 0
}
}
h.tiny_hash = [65536]byte{}
for i := range h.free_slot_idx {
h.free_slot_idx[i] = 0
}
}
/* Look at 4 bytes at &data[ix & mask]. Compute a hash from these, and prepend
node to corresponding chain; also update tiny_hash for current position. */
func (h *hashForgetfulChain) Store(data []byte, mask uint, ix uint) {
var key uint = h.HashBytes(data[ix&mask:])
var bank uint = key & (h.numBanks - 1)
idx := uint(h.free_slot_idx[bank]) & ((1 << h.bankBits) - 1)
h.free_slot_idx[bank]++
var delta uint = ix - uint(h.addr[key])
h.tiny_hash[uint16(ix)] = byte(key)
if delta > 0xFFFF {
delta = 0xFFFF
}
h.banks[bank][idx].delta = uint16(delta)
h.banks[bank][idx].next = h.head[key]
h.addr[key] = uint32(ix)
h.head[key] = uint16(idx)
}
func (h *hashForgetfulChain) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
var i uint
for i = ix_start; i < ix_end; i++ {
h.Store(data, mask, i)
}
}
func (h *hashForgetfulChain) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ring_buffer_mask uint) {
if num_bytes >= h.HashTypeLength()-1 && position >= 3 {
/* Prepare the hashes for three last bytes of the last write.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
h.Store(ringbuffer, ring_buffer_mask, position-3)
h.Store(ringbuffer, ring_buffer_mask, position-2)
h.Store(ringbuffer, ring_buffer_mask, position-1)
}
}
func (h *hashForgetfulChain) PrepareDistanceCache(distance_cache []int) {
prepareDistanceCache(distance_cache, h.numLastDistancesToCheck)
}
/* Find a longest backward match of &data[cur_ix] up to the length of
max_length and stores the position cur_ix in the hash table.
REQUIRES: PrepareDistanceCachehashForgetfulChain must be invoked for current distance cache
values; if this method is invoked repeatedly with the same distance
cache values, it is enough to invoke PrepareDistanceCachehashForgetfulChain once.
Does not look for matches longer than max_length.
Does not look for matches further away than max_backward.
Writes the best match into |out|.
|out|->score is updated only if a better match is found. */
func (h *hashForgetfulChain) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
var cur_ix_masked uint = cur_ix & ring_buffer_mask
var min_score uint = out.score
var best_score uint = out.score
var best_len uint = out.len
var key uint = h.HashBytes(data[cur_ix_masked:])
var tiny_hash byte = byte(key)
/* Don't accept a short copy from far away. */
out.len = 0
out.len_code_delta = 0
/* Try last distance first. */
for i := 0; i < h.numLastDistancesToCheck; i++ {
var backward uint = uint(distance_cache[i])
var prev_ix uint = (cur_ix - backward)
/* For distance code 0 we want to consider 2-byte matches. */
if i > 0 && h.tiny_hash[uint16(prev_ix)] != tiny_hash {
continue
}
if prev_ix >= cur_ix || backward > max_backward {
continue
}
prev_ix &= ring_buffer_mask
{
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
if len >= 2 {
var score uint = backwardReferenceScoreUsingLastDistance(uint(len))
if best_score < score {
if i != 0 {
score -= backwardReferencePenaltyUsingLastDistance(uint(i))
}
if best_score < score {
best_score = score
best_len = uint(len)
out.len = best_len
out.distance = backward
out.score = best_score
}
}
}
}
}
{
var bank uint = key & (h.numBanks - 1)
var backward uint = 0
var hops uint = h.max_hops
var delta uint = cur_ix - uint(h.addr[key])
var slot uint = uint(h.head[key])
for {
tmp6 := hops
hops--
if tmp6 == 0 {
break
}
var prev_ix uint
var last uint = slot
backward += delta
if backward > max_backward {
break
}
prev_ix = (cur_ix - backward) & ring_buffer_mask
slot = uint(h.banks[bank][last].next)
delta = uint(h.banks[bank][last].delta)
if cur_ix_masked+best_len > ring_buffer_mask || prev_ix+best_len > ring_buffer_mask || data[cur_ix_masked+best_len] != data[prev_ix+best_len] {
continue
}
{
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
if len >= 4 {
/* Comparing for >= 3 does not change the semantics, but just saves
for a few unnecessary binary logarithms in backward reference
score, since we are not interested in such short matches. */
var score uint = backwardReferenceScore(uint(len), backward)
if best_score < score {
best_score = score
best_len = uint(len)
out.len = best_len
out.distance = backward
out.score = best_score
}
}
}
}
h.Store(data, ring_buffer_mask, cur_ix)
}
if out.score == min_score {
searchInStaticDictionary(dictionary, h, data[cur_ix_masked:], max_length, max_backward+gap, max_distance, out, false)
}
}
+214
View File
@@ -0,0 +1,214 @@
package brotli
import "encoding/binary"
/* Copyright 2010 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* For BUCKET_SWEEP == 1, enabling the dictionary lookup makes compression
a little faster (0.5% - 1%) and it compresses 0.15% better on small text
and HTML inputs. */
func (*hashLongestMatchQuickly) HashTypeLength() uint {
return 8
}
func (*hashLongestMatchQuickly) StoreLookahead() uint {
return 8
}
/* HashBytes is the function that chooses the bucket to place
the address in. The HashLongestMatch and hashLongestMatchQuickly
classes have separate, different implementations of hashing. */
func (h *hashLongestMatchQuickly) HashBytes(data []byte) uint32 {
var hash uint64 = ((binary.LittleEndian.Uint64(data) << (64 - 8*h.hashLen)) * kHashMul64)
/* The higher bits contain more mixture from the multiplication,
so we take our results from there. */
return uint32(hash >> (64 - h.bucketBits))
}
/* A (forgetful) hash table to the data seen by the compressor, to
help create backward references to previous data.
This is a hash map of fixed size (1 << 16). Starting from the
given index, 1 buckets are used to store values of a key. */
type hashLongestMatchQuickly struct {
hasherCommon
bucketBits uint
bucketSweep int
hashLen uint
useDictionary bool
buckets []uint32
}
func (h *hashLongestMatchQuickly) Initialize(params *encoderParams) {
h.buckets = make([]uint32, 1<<h.bucketBits+h.bucketSweep)
}
func (h *hashLongestMatchQuickly) Prepare(one_shot bool, input_size uint, data []byte) {
var partial_prepare_threshold uint = (4 << h.bucketBits) >> 7
/* Partial preparation is 100 times slower (per socket). */
if one_shot && input_size <= partial_prepare_threshold {
var i uint
for i = 0; i < input_size; i++ {
var key uint32 = h.HashBytes(data[i:])
for j := 0; j < h.bucketSweep; j++ {
h.buckets[key+uint32(j)] = 0
}
}
} else {
/* It is not strictly necessary to fill this buffer here, but
not filling will make the results of the compression stochastic
(but correct). This is because random data would cause the
system to find accidentally good backward references here and there. */
for i := range h.buckets {
h.buckets[i] = 0
}
}
}
/* Look at 5 bytes at &data[ix & mask].
Compute a hash from these, and store the value somewhere within
[ix .. ix+3]. */
func (h *hashLongestMatchQuickly) Store(data []byte, mask uint, ix uint) {
var key uint32 = h.HashBytes(data[ix&mask:])
var off uint32 = uint32(ix>>3) % uint32(h.bucketSweep)
/* Wiggle the value with the bucket sweep range. */
h.buckets[key+off] = uint32(ix)
}
func (h *hashLongestMatchQuickly) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
var i uint
for i = ix_start; i < ix_end; i++ {
h.Store(data, mask, i)
}
}
func (h *hashLongestMatchQuickly) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint) {
if num_bytes >= h.HashTypeLength()-1 && position >= 3 {
/* Prepare the hashes for three last bytes of the last write.
These could not be calculated before, since they require knowledge
of both the previous and the current block. */
h.Store(ringbuffer, ringbuffer_mask, position-3)
h.Store(ringbuffer, ringbuffer_mask, position-2)
h.Store(ringbuffer, ringbuffer_mask, position-1)
}
}
func (*hashLongestMatchQuickly) PrepareDistanceCache(distance_cache []int) {
}
/* Find a longest backward match of &data[cur_ix & ring_buffer_mask]
up to the length of max_length and stores the position cur_ix in the
hash table.
Does not look for matches longer than max_length.
Does not look for matches further away than max_backward.
Writes the best match into |out|.
|out|->score is updated only if a better match is found. */
func (h *hashLongestMatchQuickly) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
var best_len_in uint = out.len
var cur_ix_masked uint = cur_ix & ring_buffer_mask
var key uint32 = h.HashBytes(data[cur_ix_masked:])
var compare_char int = int(data[cur_ix_masked+best_len_in])
var min_score uint = out.score
var best_score uint = out.score
var best_len uint = best_len_in
var cached_backward uint = uint(distance_cache[0])
var prev_ix uint = cur_ix - cached_backward
var bucket []uint32
out.len_code_delta = 0
if prev_ix < cur_ix {
prev_ix &= uint(uint32(ring_buffer_mask))
if compare_char == int(data[prev_ix+best_len]) {
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
if len >= 4 {
var score uint = backwardReferenceScoreUsingLastDistance(uint(len))
if best_score < score {
best_score = score
best_len = uint(len)
out.len = uint(len)
out.distance = cached_backward
out.score = best_score
compare_char = int(data[cur_ix_masked+best_len])
if h.bucketSweep == 1 {
h.buckets[key] = uint32(cur_ix)
return
}
}
}
}
}
if h.bucketSweep == 1 {
var backward uint
var len uint
/* Only one to look for, don't bother to prepare for a loop. */
prev_ix = uint(h.buckets[key])
h.buckets[key] = uint32(cur_ix)
backward = cur_ix - prev_ix
prev_ix &= uint(uint32(ring_buffer_mask))
if compare_char != int(data[prev_ix+best_len_in]) {
return
}
if backward == 0 || backward > max_backward {
return
}
len = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
if len >= 4 {
var score uint = backwardReferenceScore(uint(len), backward)
if best_score < score {
out.len = uint(len)
out.distance = backward
out.score = score
return
}
}
} else {
bucket = h.buckets[key:]
var i int
prev_ix = uint(bucket[0])
bucket = bucket[1:]
for i = 0; i < h.bucketSweep; (func() { i++; tmp3 := bucket; bucket = bucket[1:]; prev_ix = uint(tmp3[0]) })() {
var backward uint = cur_ix - prev_ix
var len uint
prev_ix &= uint(uint32(ring_buffer_mask))
if compare_char != int(data[prev_ix+best_len]) {
continue
}
if backward == 0 || backward > max_backward {
continue
}
len = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
if len >= 4 {
var score uint = backwardReferenceScore(uint(len), backward)
if best_score < score {
best_score = score
best_len = uint(len)
out.len = best_len
out.distance = backward
out.score = score
compare_char = int(data[cur_ix_masked+best_len])
}
}
}
}
if h.useDictionary && min_score == out.score {
searchInStaticDictionary(dictionary, h, data[cur_ix_masked:], max_length, max_backward+gap, max_distance, out, true)
}
h.buckets[key+uint32((cur_ix>>3)%uint(h.bucketSweep))] = uint32(cur_ix)
}
+168
View File
@@ -0,0 +1,168 @@
package brotli
/* Copyright 2018 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* NOTE: this hasher does not search in the dictionary. It is used as
backup-hasher, the main hasher already searches in it. */
const kRollingHashMul32 uint32 = 69069
const kInvalidPosHashRolling uint32 = 0xffffffff
/* This hasher uses a longer forward length, but returning a higher value here
will hurt compression by the main hasher when combined with a composite
hasher. The hasher tests for forward itself instead. */
func (*hashRolling) HashTypeLength() uint {
return 4
}
func (*hashRolling) StoreLookahead() uint {
return 4
}
/* Computes a code from a single byte. A lookup table of 256 values could be
used, but simply adding 1 works about as good. */
func (*hashRolling) HashByte(b byte) uint32 {
return uint32(b) + 1
}
func (h *hashRolling) HashRollingFunctionInitial(state uint32, add byte, factor uint32) uint32 {
return uint32(factor*state + h.HashByte(add))
}
func (h *hashRolling) HashRollingFunction(state uint32, add byte, rem byte, factor uint32, factor_remove uint32) uint32 {
return uint32(factor*state + h.HashByte(add) - factor_remove*h.HashByte(rem))
}
/* Rolling hash for long distance long string matches. Stores one position
per bucket, bucket key is computed over a long region. */
type hashRolling struct {
hasherCommon
jump int
state uint32
table []uint32
next_ix uint
factor uint32
factor_remove uint32
}
func (h *hashRolling) Initialize(params *encoderParams) {
h.state = 0
h.next_ix = 0
h.factor = kRollingHashMul32
/* Compute the factor of the oldest byte to remove: factor**steps modulo
0xffffffff (the multiplications rely on 32-bit overflow) */
h.factor_remove = 1
for i := 0; i < 32; i += h.jump {
h.factor_remove *= h.factor
}
h.table = make([]uint32, 16777216)
for i := 0; i < 16777216; i++ {
h.table[i] = kInvalidPosHashRolling
}
}
func (h *hashRolling) Prepare(one_shot bool, input_size uint, data []byte) {
/* Too small size, cannot use this hasher. */
if input_size < 32 {
return
}
h.state = 0
for i := 0; i < 32; i += h.jump {
h.state = h.HashRollingFunctionInitial(h.state, data[i], h.factor)
}
}
func (*hashRolling) Store(data []byte, mask uint, ix uint) {
}
func (*hashRolling) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
}
func (h *hashRolling) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ring_buffer_mask uint) {
var position_masked uint
/* In this case we must re-initialize the hasher from scratch from the
current position. */
var available uint = num_bytes
if position&uint(h.jump-1) != 0 {
var diff uint = uint(h.jump) - (position & uint(h.jump-1))
if diff > available {
available = 0
} else {
available = available - diff
}
position += diff
}
position_masked = position & ring_buffer_mask
/* wrapping around ringbuffer not handled. */
if available > ring_buffer_mask-position_masked {
available = ring_buffer_mask - position_masked
}
h.Prepare(false, available, ringbuffer[position&ring_buffer_mask:])
h.next_ix = position
}
func (*hashRolling) PrepareDistanceCache(distance_cache []int) {
}
func (h *hashRolling) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
var cur_ix_masked uint = cur_ix & ring_buffer_mask
var pos uint = h.next_ix
if cur_ix&uint(h.jump-1) != 0 {
return
}
/* Not enough lookahead */
if max_length < 32 {
return
}
for pos = h.next_ix; pos <= cur_ix; pos += uint(h.jump) {
var code uint32 = h.state & ((16777216 * 64) - 1)
var rem byte = data[pos&ring_buffer_mask]
var add byte = data[(pos+32)&ring_buffer_mask]
var found_ix uint = uint(kInvalidPosHashRolling)
h.state = h.HashRollingFunction(h.state, add, rem, h.factor, h.factor_remove)
if code < 16777216 {
found_ix = uint(h.table[code])
h.table[code] = uint32(pos)
if pos == cur_ix && uint32(found_ix) != kInvalidPosHashRolling {
/* The cast to 32-bit makes backward distances up to 4GB work even
if cur_ix is above 4GB, despite using 32-bit values in the table. */
var backward uint = uint(uint32(cur_ix - found_ix))
if backward <= max_backward {
var found_ix_masked uint = found_ix & ring_buffer_mask
var len uint = findMatchLengthWithLimit(data[found_ix_masked:], data[cur_ix_masked:], max_length)
if len >= 4 && len > out.len {
var score uint = backwardReferenceScore(uint(len), backward)
if score > out.score {
out.len = uint(len)
out.distance = backward
out.score = score
out.len_code_delta = 0
}
}
}
}
}
}
h.next_ix = cur_ix + uint(h.jump)
}
+226
View File
@@ -0,0 +1,226 @@
package brotli
import "math"
/* The distance symbols effectively used by "Large Window Brotli" (32-bit). */
const numHistogramDistanceSymbols = 544
type histogramLiteral struct {
data_ [numLiteralSymbols]uint32
total_count_ uint
bit_cost_ float64
}
func histogramClearLiteral(self *histogramLiteral) {
self.data_ = [numLiteralSymbols]uint32{}
self.total_count_ = 0
self.bit_cost_ = math.MaxFloat64
}
func clearHistogramsLiteral(array []histogramLiteral, length uint) {
var i uint
for i = 0; i < length; i++ {
histogramClearLiteral(&array[i:][0])
}
}
func histogramAddLiteral(self *histogramLiteral, val uint) {
self.data_[val]++
self.total_count_++
}
func histogramAddVectorLiteral(self *histogramLiteral, p []byte, n uint) {
self.total_count_ += n
n += 1
for {
n--
if n == 0 {
break
}
self.data_[p[0]]++
p = p[1:]
}
}
func histogramAddHistogramLiteral(self *histogramLiteral, v *histogramLiteral) {
var i uint
self.total_count_ += v.total_count_
for i = 0; i < numLiteralSymbols; i++ {
self.data_[i] += v.data_[i]
}
}
func histogramDataSizeLiteral() uint {
return numLiteralSymbols
}
type histogramCommand struct {
data_ [numCommandSymbols]uint32
total_count_ uint
bit_cost_ float64
}
func histogramClearCommand(self *histogramCommand) {
self.data_ = [numCommandSymbols]uint32{}
self.total_count_ = 0
self.bit_cost_ = math.MaxFloat64
}
func clearHistogramsCommand(array []histogramCommand, length uint) {
var i uint
for i = 0; i < length; i++ {
histogramClearCommand(&array[i:][0])
}
}
func histogramAddCommand(self *histogramCommand, val uint) {
self.data_[val]++
self.total_count_++
}
func histogramAddVectorCommand(self *histogramCommand, p []uint16, n uint) {
self.total_count_ += n
n += 1
for {
n--
if n == 0 {
break
}
self.data_[p[0]]++
p = p[1:]
}
}
func histogramAddHistogramCommand(self *histogramCommand, v *histogramCommand) {
var i uint
self.total_count_ += v.total_count_
for i = 0; i < numCommandSymbols; i++ {
self.data_[i] += v.data_[i]
}
}
func histogramDataSizeCommand() uint {
return numCommandSymbols
}
type histogramDistance struct {
data_ [numDistanceSymbols]uint32
total_count_ uint
bit_cost_ float64
}
func histogramClearDistance(self *histogramDistance) {
self.data_ = [numDistanceSymbols]uint32{}
self.total_count_ = 0
self.bit_cost_ = math.MaxFloat64
}
func clearHistogramsDistance(array []histogramDistance, length uint) {
var i uint
for i = 0; i < length; i++ {
histogramClearDistance(&array[i:][0])
}
}
func histogramAddDistance(self *histogramDistance, val uint) {
self.data_[val]++
self.total_count_++
}
func histogramAddVectorDistance(self *histogramDistance, p []uint16, n uint) {
self.total_count_ += n
n += 1
for {
n--
if n == 0 {
break
}
self.data_[p[0]]++
p = p[1:]
}
}
func histogramAddHistogramDistance(self *histogramDistance, v *histogramDistance) {
var i uint
self.total_count_ += v.total_count_
for i = 0; i < numDistanceSymbols; i++ {
self.data_[i] += v.data_[i]
}
}
func histogramDataSizeDistance() uint {
return numDistanceSymbols
}
type blockSplitIterator struct {
split_ *blockSplit
idx_ uint
type_ uint
length_ uint
}
func initBlockSplitIterator(self *blockSplitIterator, split *blockSplit) {
self.split_ = split
self.idx_ = 0
self.type_ = 0
if len(split.lengths) > 0 {
self.length_ = uint(split.lengths[0])
} else {
self.length_ = 0
}
}
func blockSplitIteratorNext(self *blockSplitIterator) {
if self.length_ == 0 {
self.idx_++
self.type_ = uint(self.split_.types[self.idx_])
self.length_ = uint(self.split_.lengths[self.idx_])
}
self.length_--
}
func buildHistogramsWithContext(cmds []command, literal_split *blockSplit, insert_and_copy_split *blockSplit, dist_split *blockSplit, ringbuffer []byte, start_pos uint, mask uint, prev_byte byte, prev_byte2 byte, context_modes []int, literal_histograms []histogramLiteral, insert_and_copy_histograms []histogramCommand, copy_dist_histograms []histogramDistance) {
var pos uint = start_pos
var literal_it blockSplitIterator
var insert_and_copy_it blockSplitIterator
var dist_it blockSplitIterator
initBlockSplitIterator(&literal_it, literal_split)
initBlockSplitIterator(&insert_and_copy_it, insert_and_copy_split)
initBlockSplitIterator(&dist_it, dist_split)
for i := range cmds {
var cmd *command = &cmds[i]
var j uint
blockSplitIteratorNext(&insert_and_copy_it)
histogramAddCommand(&insert_and_copy_histograms[insert_and_copy_it.type_], uint(cmd.cmd_prefix_))
/* TODO: unwrap iterator blocks. */
for j = uint(cmd.insert_len_); j != 0; j-- {
var context uint
blockSplitIteratorNext(&literal_it)
context = literal_it.type_
if context_modes != nil {
var lut contextLUT = getContextLUT(context_modes[context])
context = (context << literalContextBits) + uint(getContext(prev_byte, prev_byte2, lut))
}
histogramAddLiteral(&literal_histograms[context], uint(ringbuffer[pos&mask]))
prev_byte2 = prev_byte
prev_byte = ringbuffer[pos&mask]
pos++
}
pos += uint(commandCopyLen(cmd))
if commandCopyLen(cmd) != 0 {
prev_byte2 = ringbuffer[(pos-2)&mask]
prev_byte = ringbuffer[(pos-1)&mask]
if cmd.cmd_prefix_ >= 128 {
var context uint
blockSplitIteratorNext(&dist_it)
context = uint(uint32(dist_it.type_<<distanceContextBits) + commandDistanceContext(cmd))
histogramAddDistance(&copy_dist_histograms[context], uint(cmd.dist_prefix_)&0x3FF)
}
}
}
}
+184
View File
@@ -0,0 +1,184 @@
package brotli
import (
"compress/gzip"
"io"
"net/http"
"strings"
)
// HTTPCompressor chooses a compression method (brotli, gzip, or none) based on
// the Accept-Encoding header, sets the Content-Encoding header, and returns a
// WriteCloser that implements that compression. The Close method must be called
// before the current HTTP handler returns.
func HTTPCompressor(w http.ResponseWriter, r *http.Request) io.WriteCloser {
if w.Header().Get("Vary") == "" {
w.Header().Set("Vary", "Accept-Encoding")
}
encoding := negotiateContentEncoding(r, []string{"br", "gzip"})
switch encoding {
case "br":
w.Header().Set("Content-Encoding", "br")
return NewWriterV2(w, DefaultCompression)
case "gzip":
w.Header().Set("Content-Encoding", "gzip")
return gzip.NewWriter(w)
}
return nopCloser{w}
}
// negotiateContentEncoding returns the best offered content encoding for the
// request's Accept-Encoding header. If two offers match with equal weight and
// then the offer earlier in the list is preferred. If no offers are
// acceptable, then "" is returned.
func negotiateContentEncoding(r *http.Request, offers []string) string {
bestOffer := "identity"
bestQ := -1.0
specs := parseAccept(r.Header, "Accept-Encoding")
for _, offer := range offers {
for _, spec := range specs {
if spec.Q > bestQ &&
(spec.Value == "*" || spec.Value == offer) {
bestQ = spec.Q
bestOffer = offer
}
}
}
if bestQ == 0 {
bestOffer = ""
}
return bestOffer
}
// acceptSpec describes an Accept* header.
type acceptSpec struct {
Value string
Q float64
}
// parseAccept parses Accept* headers.
func parseAccept(header http.Header, key string) (specs []acceptSpec) {
loop:
for _, s := range header[key] {
for {
var spec acceptSpec
spec.Value, s = expectTokenSlash(s)
if spec.Value == "" {
continue loop
}
spec.Q = 1.0
s = skipSpace(s)
if strings.HasPrefix(s, ";") {
s = skipSpace(s[1:])
if !strings.HasPrefix(s, "q=") {
continue loop
}
spec.Q, s = expectQuality(s[2:])
if spec.Q < 0.0 {
continue loop
}
}
specs = append(specs, spec)
s = skipSpace(s)
if !strings.HasPrefix(s, ",") {
continue loop
}
s = skipSpace(s[1:])
}
}
return
}
func skipSpace(s string) (rest string) {
i := 0
for ; i < len(s); i++ {
if octetTypes[s[i]]&isSpace == 0 {
break
}
}
return s[i:]
}
func expectTokenSlash(s string) (token, rest string) {
i := 0
for ; i < len(s); i++ {
b := s[i]
if (octetTypes[b]&isToken == 0) && b != '/' {
break
}
}
return s[:i], s[i:]
}
func expectQuality(s string) (q float64, rest string) {
switch {
case len(s) == 0:
return -1, ""
case s[0] == '0':
q = 0
case s[0] == '1':
q = 1
default:
return -1, ""
}
s = s[1:]
if !strings.HasPrefix(s, ".") {
return q, s
}
s = s[1:]
i := 0
n := 0
d := 1
for ; i < len(s); i++ {
b := s[i]
if b < '0' || b > '9' {
break
}
n = n*10 + int(b) - '0'
d *= 10
}
return q + float64(n)/float64(d), s[i:]
}
// Octet types from RFC 2616.
var octetTypes [256]octetType
type octetType byte
const (
isToken octetType = 1 << iota
isSpace
)
func init() {
// OCTET = <any 8-bit sequence of data>
// CHAR = <any US-ASCII character (octets 0 - 127)>
// CTL = <any US-ASCII control character (octets 0 - 31) and DEL (127)>
// CR = <US-ASCII CR, carriage return (13)>
// LF = <US-ASCII LF, linefeed (10)>
// SP = <US-ASCII SP, space (32)>
// HT = <US-ASCII HT, horizontal-tab (9)>
// <"> = <US-ASCII double-quote mark (34)>
// CRLF = CR LF
// LWS = [CRLF] 1*( SP | HT )
// TEXT = <any OCTET except CTLs, but including LWS>
// separators = "(" | ")" | "<" | ">" | "@" | "," | ";" | ":" | "\" | <">
// | "/" | "[" | "]" | "?" | "=" | "{" | "}" | SP | HT
// token = 1*<any CHAR except CTLs or separators>
// qdtext = <any TEXT except <">>
for c := 0; c < 256; c++ {
var t octetType
isCtl := c <= 31 || c == 127
isChar := 0 <= c && c <= 127
isSeparator := strings.ContainsRune(" \t\"(),/:;<=>?@[]\\{}", rune(c))
if strings.ContainsRune(" \t\r\n", rune(c)) {
t |= isSpace
}
if isChar && !isCtl && !isSeparator {
t |= isToken
}
octetTypes[c] = t
}
}
+653
View File
@@ -0,0 +1,653 @@
package brotli
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Utilities for building Huffman decoding tables. */
const huffmanMaxCodeLength = 15
/* Maximum possible Huffman table size for an alphabet size of (index * 32),
max code length 15 and root table bits 8. */
var kMaxHuffmanTableSize = []uint16{
256,
402,
436,
468,
500,
534,
566,
598,
630,
662,
694,
726,
758,
790,
822,
854,
886,
920,
952,
984,
1016,
1048,
1080,
1112,
1144,
1176,
1208,
1240,
1272,
1304,
1336,
1368,
1400,
1432,
1464,
1496,
1528,
}
/* BROTLI_NUM_BLOCK_LEN_SYMBOLS == 26 */
const huffmanMaxSize26 = 396
/* BROTLI_MAX_BLOCK_TYPE_SYMBOLS == 258 */
const huffmanMaxSize258 = 632
/* BROTLI_MAX_CONTEXT_MAP_SYMBOLS == 272 */
const huffmanMaxSize272 = 646
const huffmanMaxCodeLengthCodeLength = 5
/* Do not create this struct directly - use the ConstructHuffmanCode
* constructor below! */
type huffmanCode struct {
bits byte
value uint16
}
func constructHuffmanCode(bits byte, value uint16) huffmanCode {
var h huffmanCode
h.bits = bits
h.value = value
return h
}
/* Builds Huffman lookup table assuming code lengths are in symbol order. */
/* Builds Huffman lookup table assuming code lengths are in symbol order.
Returns size of resulting table. */
/* Builds a simple Huffman table. The |num_symbols| parameter is to be
interpreted as follows: 0 means 1 symbol, 1 means 2 symbols,
2 means 3 symbols, 3 means 4 symbols with lengths [2, 2, 2, 2],
4 means 4 symbols with lengths [1, 2, 3, 3]. */
/* Contains a collection of Huffman trees with the same alphabet size. */
/* max_symbol is needed due to simple codes since log2(alphabet_size) could be
greater than log2(max_symbol). */
type huffmanTreeGroup struct {
htrees [][]huffmanCode
codes []huffmanCode
alphabet_size uint16
max_symbol uint16
num_htrees uint16
}
const reverseBitsMax = 8
const reverseBitsBase = 0
var kReverseBits = [1 << reverseBitsMax]byte{
0x00,
0x80,
0x40,
0xC0,
0x20,
0xA0,
0x60,
0xE0,
0x10,
0x90,
0x50,
0xD0,
0x30,
0xB0,
0x70,
0xF0,
0x08,
0x88,
0x48,
0xC8,
0x28,
0xA8,
0x68,
0xE8,
0x18,
0x98,
0x58,
0xD8,
0x38,
0xB8,
0x78,
0xF8,
0x04,
0x84,
0x44,
0xC4,
0x24,
0xA4,
0x64,
0xE4,
0x14,
0x94,
0x54,
0xD4,
0x34,
0xB4,
0x74,
0xF4,
0x0C,
0x8C,
0x4C,
0xCC,
0x2C,
0xAC,
0x6C,
0xEC,
0x1C,
0x9C,
0x5C,
0xDC,
0x3C,
0xBC,
0x7C,
0xFC,
0x02,
0x82,
0x42,
0xC2,
0x22,
0xA2,
0x62,
0xE2,
0x12,
0x92,
0x52,
0xD2,
0x32,
0xB2,
0x72,
0xF2,
0x0A,
0x8A,
0x4A,
0xCA,
0x2A,
0xAA,
0x6A,
0xEA,
0x1A,
0x9A,
0x5A,
0xDA,
0x3A,
0xBA,
0x7A,
0xFA,
0x06,
0x86,
0x46,
0xC6,
0x26,
0xA6,
0x66,
0xE6,
0x16,
0x96,
0x56,
0xD6,
0x36,
0xB6,
0x76,
0xF6,
0x0E,
0x8E,
0x4E,
0xCE,
0x2E,
0xAE,
0x6E,
0xEE,
0x1E,
0x9E,
0x5E,
0xDE,
0x3E,
0xBE,
0x7E,
0xFE,
0x01,
0x81,
0x41,
0xC1,
0x21,
0xA1,
0x61,
0xE1,
0x11,
0x91,
0x51,
0xD1,
0x31,
0xB1,
0x71,
0xF1,
0x09,
0x89,
0x49,
0xC9,
0x29,
0xA9,
0x69,
0xE9,
0x19,
0x99,
0x59,
0xD9,
0x39,
0xB9,
0x79,
0xF9,
0x05,
0x85,
0x45,
0xC5,
0x25,
0xA5,
0x65,
0xE5,
0x15,
0x95,
0x55,
0xD5,
0x35,
0xB5,
0x75,
0xF5,
0x0D,
0x8D,
0x4D,
0xCD,
0x2D,
0xAD,
0x6D,
0xED,
0x1D,
0x9D,
0x5D,
0xDD,
0x3D,
0xBD,
0x7D,
0xFD,
0x03,
0x83,
0x43,
0xC3,
0x23,
0xA3,
0x63,
0xE3,
0x13,
0x93,
0x53,
0xD3,
0x33,
0xB3,
0x73,
0xF3,
0x0B,
0x8B,
0x4B,
0xCB,
0x2B,
0xAB,
0x6B,
0xEB,
0x1B,
0x9B,
0x5B,
0xDB,
0x3B,
0xBB,
0x7B,
0xFB,
0x07,
0x87,
0x47,
0xC7,
0x27,
0xA7,
0x67,
0xE7,
0x17,
0x97,
0x57,
0xD7,
0x37,
0xB7,
0x77,
0xF7,
0x0F,
0x8F,
0x4F,
0xCF,
0x2F,
0xAF,
0x6F,
0xEF,
0x1F,
0x9F,
0x5F,
0xDF,
0x3F,
0xBF,
0x7F,
0xFF,
}
const reverseBitsLowest = (uint64(1) << (reverseBitsMax - 1 + reverseBitsBase))
/* Returns reverse(num >> BROTLI_REVERSE_BITS_BASE, BROTLI_REVERSE_BITS_MAX),
where reverse(value, len) is the bit-wise reversal of the len least
significant bits of value. */
func reverseBits8(num uint64) uint64 {
return uint64(kReverseBits[num])
}
/* Stores code in table[0], table[step], table[2*step], ..., table[end] */
/* Assumes that end is an integer multiple of step */
func replicateValue(table []huffmanCode, step int, end int, code huffmanCode) {
for {
end -= step
table[end] = code
if end <= 0 {
break
}
}
}
/* Returns the table width of the next 2nd level table. |count| is the histogram
of bit lengths for the remaining symbols, |len| is the code length of the
next processed symbol. */
func nextTableBitSize(count []uint16, len int, root_bits int) int {
var left int = 1 << uint(len-root_bits)
for len < huffmanMaxCodeLength {
left -= int(count[len])
if left <= 0 {
break
}
len++
left <<= 1
}
return len - root_bits
}
func buildCodeLengthsHuffmanTable(table []huffmanCode, code_lengths []byte, count []uint16) {
var code huffmanCode /* current table entry */ /* symbol index in original or sorted table */ /* prefix code */ /* prefix code addend */ /* step size to replicate values in current table */ /* size of current table */ /* symbols sorted by code length */
var symbol int
var key uint64
var key_step uint64
var step int
var table_size int
var sorted [codeLengthCodes]int
var offset [huffmanMaxCodeLengthCodeLength + 1]int
var bits int
var bits_count int
/* offsets in sorted table for each length */
assert(huffmanMaxCodeLengthCodeLength <= reverseBitsMax)
/* Generate offsets into sorted symbol table by code length. */
symbol = -1
bits = 1
var i int
for i = 0; i < huffmanMaxCodeLengthCodeLength; i++ {
symbol += int(count[bits])
offset[bits] = symbol
bits++
}
/* Symbols with code length 0 are placed after all other symbols. */
offset[0] = codeLengthCodes - 1
/* Sort symbols by length, by symbol order within each length. */
symbol = codeLengthCodes
for {
var i int
for i = 0; i < 6; i++ {
symbol--
sorted[offset[code_lengths[symbol]]] = symbol
offset[code_lengths[symbol]]--
}
if symbol == 0 {
break
}
}
table_size = 1 << huffmanMaxCodeLengthCodeLength
/* Special case: all symbols but one have 0 code length. */
if offset[0] == 0 {
code = constructHuffmanCode(0, uint16(sorted[0]))
for key = 0; key < uint64(table_size); key++ {
table[key] = code
}
return
}
/* Fill in table. */
key = 0
key_step = reverseBitsLowest
symbol = 0
bits = 1
step = 2
for {
for bits_count = int(count[bits]); bits_count != 0; bits_count-- {
code = constructHuffmanCode(byte(bits), uint16(sorted[symbol]))
symbol++
replicateValue(table[reverseBits8(key):], step, table_size, code)
key += key_step
}
step <<= 1
key_step >>= 1
bits++
if bits > huffmanMaxCodeLengthCodeLength {
break
}
}
}
func buildHuffmanTable(root_table []huffmanCode, root_bits int, symbol_lists symbolList, count []uint16) uint32 {
var code huffmanCode /* current table entry */ /* next available space in table */ /* current code length */ /* symbol index in original or sorted table */ /* prefix code */ /* prefix code addend */ /* 2nd level table prefix code */ /* 2nd level table prefix code addend */ /* step size to replicate values in current table */ /* key length of current table */ /* size of current table */ /* sum of root table size and 2nd level table sizes */
var table []huffmanCode
var len int
var symbol int
var key uint64
var key_step uint64
var sub_key uint64
var sub_key_step uint64
var step int
var table_bits int
var table_size int
var total_size int
var max_length int = -1
var bits int
var bits_count int
assert(root_bits <= reverseBitsMax)
assert(huffmanMaxCodeLength-root_bits <= reverseBitsMax)
for symbolListGet(symbol_lists, max_length) == 0xFFFF {
max_length--
}
max_length += huffmanMaxCodeLength + 1
table = root_table
table_bits = root_bits
table_size = 1 << uint(table_bits)
total_size = table_size
/* Fill in the root table. Reduce the table size to if possible,
and create the repetitions by memcpy. */
if table_bits > max_length {
table_bits = max_length
table_size = 1 << uint(table_bits)
}
key = 0
key_step = reverseBitsLowest
bits = 1
step = 2
for {
symbol = bits - (huffmanMaxCodeLength + 1)
for bits_count = int(count[bits]); bits_count != 0; bits_count-- {
symbol = int(symbolListGet(symbol_lists, symbol))
code = constructHuffmanCode(byte(bits), uint16(symbol))
replicateValue(table[reverseBits8(key):], step, table_size, code)
key += key_step
}
step <<= 1
key_step >>= 1
bits++
if bits > table_bits {
break
}
}
/* If root_bits != table_bits then replicate to fill the remaining slots. */
for total_size != table_size {
copy(table[table_size:], table[:uint(table_size)])
table_size <<= 1
}
/* Fill in 2nd level tables and add pointers to root table. */
key_step = reverseBitsLowest >> uint(root_bits-1)
sub_key = reverseBitsLowest << 1
sub_key_step = reverseBitsLowest
len = root_bits + 1
step = 2
for ; len <= max_length; len++ {
symbol = len - (huffmanMaxCodeLength + 1)
for ; count[len] != 0; count[len]-- {
if sub_key == reverseBitsLowest<<1 {
table = table[table_size:]
table_bits = nextTableBitSize(count, int(len), root_bits)
table_size = 1 << uint(table_bits)
total_size += table_size
sub_key = reverseBits8(key)
key += key_step
root_table[sub_key] = constructHuffmanCode(byte(table_bits+root_bits), uint16(uint64(uint(-cap(table)+cap(root_table)))-sub_key))
sub_key = 0
}
symbol = int(symbolListGet(symbol_lists, symbol))
code = constructHuffmanCode(byte(len-root_bits), uint16(symbol))
replicateValue(table[reverseBits8(sub_key):], step, table_size, code)
sub_key += sub_key_step
}
step <<= 1
sub_key_step >>= 1
}
return uint32(total_size)
}
func buildSimpleHuffmanTable(table []huffmanCode, root_bits int, val []uint16, num_symbols uint32) uint32 {
var table_size uint32 = 1
var goal_size uint32 = 1 << uint(root_bits)
switch num_symbols {
case 0:
table[0] = constructHuffmanCode(0, val[0])
case 1:
if val[1] > val[0] {
table[0] = constructHuffmanCode(1, val[0])
table[1] = constructHuffmanCode(1, val[1])
} else {
table[0] = constructHuffmanCode(1, val[1])
table[1] = constructHuffmanCode(1, val[0])
}
table_size = 2
case 2:
table[0] = constructHuffmanCode(1, val[0])
table[2] = constructHuffmanCode(1, val[0])
if val[2] > val[1] {
table[1] = constructHuffmanCode(2, val[1])
table[3] = constructHuffmanCode(2, val[2])
} else {
table[1] = constructHuffmanCode(2, val[2])
table[3] = constructHuffmanCode(2, val[1])
}
table_size = 4
case 3:
var i int
var k int
for i = 0; i < 3; i++ {
for k = i + 1; k < 4; k++ {
if val[k] < val[i] {
var t uint16 = val[k]
val[k] = val[i]
val[i] = t
}
}
}
table[0] = constructHuffmanCode(2, val[0])
table[2] = constructHuffmanCode(2, val[1])
table[1] = constructHuffmanCode(2, val[2])
table[3] = constructHuffmanCode(2, val[3])
table_size = 4
case 4:
if val[3] < val[2] {
var t uint16 = val[3]
val[3] = val[2]
val[2] = t
}
table[0] = constructHuffmanCode(1, val[0])
table[1] = constructHuffmanCode(2, val[1])
table[2] = constructHuffmanCode(1, val[0])
table[3] = constructHuffmanCode(3, val[2])
table[4] = constructHuffmanCode(1, val[0])
table[5] = constructHuffmanCode(2, val[1])
table[6] = constructHuffmanCode(1, val[0])
table[7] = constructHuffmanCode(3, val[3])
table_size = 8
}
for table_size != goal_size {
copy(table[table_size:], table[:uint(table_size)])
table_size <<= 1
}
return goal_size
}
+182
View File
@@ -0,0 +1,182 @@
package brotli
func utf8Position(last uint, c uint, clamp uint) uint {
if c < 128 {
return 0 /* Next one is the 'Byte 1' again. */
} else if c >= 192 { /* Next one is the 'Byte 2' of utf-8 encoding. */
return brotli_min_size_t(1, clamp)
} else {
/* Let's decide over the last byte if this ends the sequence. */
if last < 0xE0 {
return 0 /* Completed two or three byte coding. */ /* Next one is the 'Byte 3' of utf-8 encoding. */
} else {
return brotli_min_size_t(2, clamp)
}
}
}
func decideMultiByteStatsLevel(pos uint, len uint, mask uint, data []byte) uint {
var counts = [3]uint{0} /* should be 2, but 1 compresses better. */
var max_utf8 uint = 1
var last_c uint = 0
var i uint
for i = 0; i < len; i++ {
var c uint = uint(data[(pos+i)&mask])
counts[utf8Position(last_c, c, 2)]++
last_c = c
}
if counts[2] < 500 {
max_utf8 = 1
}
if counts[1]+counts[2] < 25 {
max_utf8 = 0
}
return max_utf8
}
func estimateBitCostsForLiteralsUTF8(pos uint, len uint, mask uint, data []byte, cost []float32) {
var max_utf8 uint = decideMultiByteStatsLevel(pos, uint(len), mask, data)
/* Bootstrap histograms. */
var histogram = [3][256]uint{[256]uint{0}}
var window_half uint = 495
var in_window uint = brotli_min_size_t(window_half, uint(len))
var in_window_utf8 = [3]uint{0}
/* max_utf8 is 0 (normal ASCII single byte modeling),
1 (for 2-byte UTF-8 modeling), or 2 (for 3-byte UTF-8 modeling). */
var i uint
{
var last_c uint = 0
var utf8_pos uint = 0
for i = 0; i < in_window; i++ {
var c uint = uint(data[(pos+i)&mask])
histogram[utf8_pos][c]++
in_window_utf8[utf8_pos]++
utf8_pos = utf8Position(last_c, c, max_utf8)
last_c = c
}
}
/* Compute bit costs with sliding window. */
for i = 0; i < len; i++ {
if i >= window_half {
var c uint
var last_c uint
if i < window_half+1 {
c = 0
} else {
c = uint(data[(pos+i-window_half-1)&mask])
}
if i < window_half+2 {
last_c = 0
} else {
last_c = uint(data[(pos+i-window_half-2)&mask])
}
/* Remove a byte in the past. */
var utf8_pos2 uint = utf8Position(last_c, c, max_utf8)
histogram[utf8_pos2][data[(pos+i-window_half)&mask]]--
in_window_utf8[utf8_pos2]--
}
if i+window_half < len {
var c uint = uint(data[(pos+i+window_half-1)&mask])
var last_c uint = uint(data[(pos+i+window_half-2)&mask])
/* Add a byte in the future. */
var utf8_pos2 uint = utf8Position(last_c, c, max_utf8)
histogram[utf8_pos2][data[(pos+i+window_half)&mask]]++
in_window_utf8[utf8_pos2]++
}
{
var c uint
var last_c uint
if i < 1 {
c = 0
} else {
c = uint(data[(pos+i-1)&mask])
}
if i < 2 {
last_c = 0
} else {
last_c = uint(data[(pos+i-2)&mask])
}
var utf8_pos uint = utf8Position(last_c, c, max_utf8)
var masked_pos uint = (pos + i) & mask
var histo uint = histogram[utf8_pos][data[masked_pos]]
var lit_cost float64
if histo == 0 {
histo = 1
}
lit_cost = fastLog2(in_window_utf8[utf8_pos]) - fastLog2(histo)
lit_cost += 0.02905
if lit_cost < 1.0 {
lit_cost *= 0.5
lit_cost += 0.5
}
/* Make the first bytes more expensive -- seems to help, not sure why.
Perhaps because the entropy source is changing its properties
rapidly in the beginning of the file, perhaps because the beginning
of the data is a statistical "anomaly". */
if i < 2000 {
lit_cost += 0.7 - (float64(2000-i) / 2000.0 * 0.35)
}
cost[i] = float32(lit_cost)
}
}
}
func estimateBitCostsForLiterals(pos uint, len uint, mask uint, data []byte, cost []float32) {
if isMostlyUTF8(data, pos, mask, uint(len), kMinUTF8Ratio) {
estimateBitCostsForLiteralsUTF8(pos, uint(len), mask, data, cost)
return
} else {
var histogram = [256]uint{0}
var window_half uint = 2000
var in_window uint = brotli_min_size_t(window_half, uint(len))
var i uint
/* Bootstrap histogram. */
for i = 0; i < in_window; i++ {
histogram[data[(pos+i)&mask]]++
}
/* Compute bit costs with sliding window. */
for i = 0; i < len; i++ {
var histo uint
if i >= window_half {
/* Remove a byte in the past. */
histogram[data[(pos+i-window_half)&mask]]--
in_window--
}
if i+window_half < len {
/* Add a byte in the future. */
histogram[data[(pos+i+window_half)&mask]]++
in_window++
}
histo = histogram[data[(pos+i)&mask]]
if histo == 0 {
histo = 1
}
{
var lit_cost float64 = fastLog2(in_window) - fastLog2(histo)
lit_cost += 0.029
if lit_cost < 1.0 {
lit_cost *= 0.5
lit_cost += 0.5
}
cost[i] = float32(lit_cost)
}
}
}
}
+34
View File
@@ -0,0 +1,34 @@
package matchfinder
// An absoluteMatch is like a Match, but it stores indexes into the byte
// stream instead of lengths.
type absoluteMatch struct {
// Start is the index of the first byte.
Start int
// End is the index of the byte after the last byte
// (so that End - Start = Length).
End int
// Match is the index of the previous data that matches
// (Start - Match = Distance).
Match int
}
// A matchEmitter manages the output of matches for a MatchFinder.
type matchEmitter struct {
// Dst is the destination slice that Matches are added to.
Dst []Match
// NextEmit is the index of the next byte to emit.
NextEmit int
}
func (e *matchEmitter) emit(m absoluteMatch) {
e.Dst = append(e.Dst, Match{
Unmatched: m.Start - e.NextEmit,
Length: m.End - m.Start,
Distance: m.Start - m.Match,
})
e.NextEmit = m.End
}
+169
View File
@@ -0,0 +1,169 @@
package matchfinder
import (
"encoding/binary"
)
// M0 is an implementation of the MatchFinder interface based
// on the algorithm used by snappy, but modified to be more like the algorithm
// used by compression level 0 of the brotli reference implementation.
//
// It has a maximum block size of 65536 bytes.
type M0 struct {
// Lazy turns on "lazy matching," for higher compression but less speed.
Lazy bool
MaxDistance int
MaxLength int
}
func (M0) Reset() {}
const (
m0HashLen = 5
m0TableBits = 14
m0TableSize = 1 << m0TableBits
m0Shift = 32 - m0TableBits
// m0TableMask is redundant, but helps the compiler eliminate bounds
// checks.
m0TableMask = m0TableSize - 1
)
func (m M0) hash(data uint64) uint64 {
hash := (data << (64 - 8*m0HashLen)) * hashMul64
return hash >> (64 - m0TableBits)
}
// FindMatches looks for matches in src, appends them to dst, and returns dst.
// src must not be longer than 65536 bytes.
func (m M0) FindMatches(dst []Match, src []byte) []Match {
const inputMargin = 16 - 1
const minNonLiteralBlockSize = 1 + 1 + inputMargin
if len(src) < minNonLiteralBlockSize {
dst = append(dst, Match{
Unmatched: len(src),
})
return dst
}
if len(src) > 65536 {
panic("block too long")
}
var table [m0TableSize]uint16
// sLimit is when to stop looking for offset/length copies. The inputMargin
// lets us use a fast path for emitLiteral in the main loop, while we are
// looking for copies.
sLimit := len(src) - inputMargin
// nextEmit is where in src the next emitLiteral should start from.
nextEmit := 0
// The encoded form must start with a literal, as there are no previous
// bytes to copy, so we start looking for hash matches at s == 1.
s := 1
nextHash := m.hash(binary.LittleEndian.Uint64(src[s:]))
for {
// Copied from the C++ snappy implementation:
//
// Heuristic match skipping: If 32 bytes are scanned with no matches
// found, start looking only at every other byte. If 32 more bytes are
// scanned (or skipped), look at every third byte, etc.. When a match
// is found, immediately go back to looking at every byte. This is a
// small loss (~5% performance, ~0.1% density) for compressible data
// due to more bookkeeping, but for non-compressible data (such as
// JPEG) it's a huge win since the compressor quickly "realizes" the
// data is incompressible and doesn't bother looking for matches
// everywhere.
//
// The "skip" variable keeps track of how many bytes there are since
// the last match; dividing it by 32 (ie. right-shifting by five) gives
// the number of bytes to move ahead for each iteration.
skip := 32
nextS := s
candidate := 0
for {
s = nextS
bytesBetweenHashLookups := skip >> 5
nextS = s + bytesBetweenHashLookups
skip += bytesBetweenHashLookups
if nextS > sLimit {
goto emitRemainder
}
candidate = int(table[nextHash&m0TableMask])
table[nextHash&m0TableMask] = uint16(s)
nextHash = m.hash(binary.LittleEndian.Uint64(src[nextS:]))
if m.MaxDistance != 0 && s-candidate > m.MaxDistance {
continue
}
if binary.LittleEndian.Uint32(src[s:]) == binary.LittleEndian.Uint32(src[candidate:]) {
break
}
}
// Invariant: we have a 4-byte match at s.
base := s
s = extendMatch(src, candidate+4, s+4)
origBase := base
if m.Lazy && base+1 < sLimit {
newBase := base + 1
h := m.hash(binary.LittleEndian.Uint64(src[newBase:]))
newCandidate := int(table[h&m0TableMask])
table[h&m0TableMask] = uint16(newBase)
okDistance := true
if m.MaxDistance != 0 && newBase-newCandidate > m.MaxDistance {
okDistance = false
}
if okDistance && binary.LittleEndian.Uint32(src[newBase:]) == binary.LittleEndian.Uint32(src[newCandidate:]) {
newS := extendMatch(src, newCandidate+4, newBase+4)
if newS-newBase > s-base+1 {
s = newS
base = newBase
candidate = newCandidate
}
}
}
if m.MaxLength != 0 && s-base > m.MaxLength {
s = base + m.MaxLength
}
dst = append(dst, Match{
Unmatched: base - nextEmit,
Length: s - base,
Distance: base - candidate,
})
nextEmit = s
if s >= sLimit {
goto emitRemainder
}
if m.Lazy {
// If lazy matching is enabled, we update the hash table for
// every byte in the match.
for i := origBase + 2; i < s-1; i++ {
x := binary.LittleEndian.Uint64(src[i:])
table[m.hash(x)&m0TableMask] = uint16(i)
}
}
// We could immediately start working at s now, but to improve
// compression we first update the hash table at s-1 and at s.
x := binary.LittleEndian.Uint64(src[s-1:])
prevHash := m.hash(x >> 0)
table[prevHash&m0TableMask] = uint16(s - 1)
nextHash = m.hash(x >> 8)
}
emitRemainder:
if nextEmit < len(src) {
dst = append(dst, Match{
Unmatched: len(src) - nextEmit,
})
}
return dst
}
+308
View File
@@ -0,0 +1,308 @@
package matchfinder
import (
"encoding/binary"
"math/bits"
"runtime"
)
// M4 is an implementation of the MatchFinder
// interface that uses a hash table to find matches,
// optional match chains,
// and the advanced parsing technique from
// https://fastcompression.blogspot.com/2011/12/advanced-parsing-strategies.html.
type M4 struct {
// MaxDistance is the maximum distance (in bytes) to look back for
// a match. The default is 65535.
MaxDistance int
// MinLength is the length of the shortest match to return.
// The default is 4.
MinLength int
// HashLen is the number of bytes to use to calculate the hashes.
// The maximum is 8 and the default is 6.
HashLen int
// TableBits is the number of bits in the hash table indexes.
// The default is 17 (128K entries).
TableBits int
// ChainLength is how many entries to search on the "match chain" of older
// locations with the same hash as the current location.
ChainLength int
// DistanceBitCost is used when comparing two matches to see
// which is better. The comparison is primarily based on the length
// of the matches, but it can also take the distance into account,
// in terms of the number of bits needed to represent the distance.
// One byte of length is given a score of 256, so 32 (256/8) would
// be a reasonable first guess for the value of one bit.
// (The default is 0, which bases the comparison solely on length.)
DistanceBitCost int
table []uint32
chain []uint16
history []byte
}
func (q *M4) Reset() {
for i := range q.table {
q.table[i] = 0
}
q.history = q.history[:0]
q.chain = q.chain[:0]
}
func (q *M4) score(m absoluteMatch) int {
return (m.End-m.Start)*256 + (bits.LeadingZeros32(uint32(m.Start-m.Match))-32)*q.DistanceBitCost
}
func (q *M4) FindMatches(dst []Match, src []byte) []Match {
if q.MaxDistance == 0 {
q.MaxDistance = 65535
}
if q.MinLength == 0 {
q.MinLength = 4
}
if q.HashLen == 0 {
q.HashLen = 6
}
if q.TableBits == 0 {
q.TableBits = 17
}
if len(q.table) < 1<<q.TableBits {
q.table = make([]uint32, 1<<q.TableBits)
}
e := matchEmitter{Dst: dst}
if len(q.history) > q.MaxDistance*2 {
// Trim down the history buffer.
delta := len(q.history) - q.MaxDistance
copy(q.history, q.history[delta:])
q.history = q.history[:q.MaxDistance]
if q.ChainLength > 0 {
q.chain = q.chain[:q.MaxDistance]
}
for i, v := range q.table {
newV := int(v) - delta
if newV < 0 {
newV = 0
}
q.table[i] = uint32(newV)
}
}
// Append src to the history buffer.
e.NextEmit = len(q.history)
q.history = append(q.history, src...)
if q.ChainLength > 0 {
q.chain = append(q.chain, make([]uint16, len(src))...)
}
src = q.history
// matches stores the matches that have been found but not emitted,
// in reverse order. (matches[0] is the most recent one.)
var matches [3]absoluteMatch
for i := e.NextEmit; i < len(src)-7; i++ {
if matches[0] != (absoluteMatch{}) && i >= matches[0].End {
// We have found some matches, and we're far enough along that we probably
// won't find overlapping matches, so we might as well emit them.
if matches[1] != (absoluteMatch{}) {
if matches[1].End > matches[0].Start {
matches[1].End = matches[0].Start
}
if matches[1].End-matches[1].Start >= q.MinLength && q.score(matches[1]) > 0 {
e.emit(matches[1])
}
}
e.emit(matches[0])
matches = [3]absoluteMatch{}
}
// Calculate and store the hash.
h := ((binary.LittleEndian.Uint64(src[i:]) & (1<<(8*q.HashLen) - 1)) * hashMul64) >> (64 - q.TableBits)
candidate := int(q.table[h])
q.table[h] = uint32(i)
if q.ChainLength > 0 && candidate != 0 {
delta := i - candidate
if delta < 1<<16 {
q.chain[i] = uint16(delta)
}
}
if i < matches[0].End && i != matches[0].End+2-q.HashLen {
continue
}
if candidate == 0 || i-candidate > q.MaxDistance {
continue
}
// Look for a match.
var currentMatch absoluteMatch
if binary.LittleEndian.Uint32(src[candidate:]) == binary.LittleEndian.Uint32(src[i:]) {
m := extendMatch2(src, i, candidate, e.NextEmit)
if m.End-m.Start > q.MinLength && q.score(m) > 0 {
currentMatch = m
}
}
for j := 0; j < q.ChainLength; j++ {
delta := q.chain[candidate]
if delta == 0 {
break
}
candidate -= int(delta)
if candidate <= 0 || i-candidate > q.MaxDistance {
break
}
if binary.LittleEndian.Uint32(src[candidate:]) == binary.LittleEndian.Uint32(src[i:]) {
m := extendMatch2(src, i, candidate, e.NextEmit)
if m.End-m.Start > q.MinLength && q.score(m) > q.score(currentMatch) {
currentMatch = m
}
}
}
if currentMatch.End-currentMatch.Start < q.MinLength {
continue
}
overlapPenalty := 0
if matches[0] != (absoluteMatch{}) {
overlapPenalty = 275
if currentMatch.Start <= matches[1].End {
// This match would completely replace the previous match,
// so there is no penalty for overlap.
overlapPenalty = 0
}
}
if q.score(currentMatch) <= q.score(matches[0])+overlapPenalty {
continue
}
matches = [3]absoluteMatch{
currentMatch,
matches[0],
matches[1],
}
if matches[2] == (absoluteMatch{}) {
continue
}
// We have three matches, so it's time to emit one and/or eliminate one.
switch {
case matches[0].Start < matches[2].End:
// The first and third matches overlap; discard the one in between.
matches = [3]absoluteMatch{
matches[0],
matches[2],
absoluteMatch{},
}
case matches[0].Start < matches[2].End+q.MinLength:
// The first and third matches don't overlap, but there's no room for
// another match between them. Emit the first match and discard the second.
e.emit(matches[2])
matches = [3]absoluteMatch{
matches[0],
absoluteMatch{},
absoluteMatch{},
}
default:
// Emit the first match, shortening it if necessary to avoid overlap with the second.
if matches[2].End > matches[1].Start {
matches[2].End = matches[1].Start
}
if matches[2].End-matches[2].Start >= q.MinLength && q.score(matches[2]) > 0 {
e.emit(matches[2])
}
matches[2] = absoluteMatch{}
}
}
// We've found all the matches now; emit the remaining ones.
if matches[1] != (absoluteMatch{}) {
if matches[1].End > matches[0].Start {
matches[1].End = matches[0].Start
}
if matches[1].End-matches[1].Start >= q.MinLength && q.score(matches[1]) > 0 {
e.emit(matches[1])
}
}
if matches[0] != (absoluteMatch{}) {
e.emit(matches[0])
}
dst = e.Dst
if e.NextEmit < len(src) {
dst = append(dst, Match{
Unmatched: len(src) - e.NextEmit,
})
}
return dst
}
const hashMul64 = 0x1E35A7BD1E35A7BD
// extendMatch returns the largest k such that k <= len(src) and that
// src[i:i+k-j] and src[j:k] have the same contents.
//
// It assumes that:
//
// 0 <= i && i < j && j <= len(src)
func extendMatch(src []byte, i, j int) int {
switch runtime.GOARCH {
case "amd64":
// As long as we are 8 or more bytes before the end of src, we can load and
// compare 8 bytes at a time. If those 8 bytes are equal, repeat.
for j+8 < len(src) {
iBytes := binary.LittleEndian.Uint64(src[i:])
jBytes := binary.LittleEndian.Uint64(src[j:])
if iBytes != jBytes {
// If those 8 bytes were not equal, XOR the two 8 byte values, and return
// the index of the first byte that differs. The BSF instruction finds the
// least significant 1 bit, the amd64 architecture is little-endian, and
// the shift by 3 converts a bit index to a byte index.
return j + bits.TrailingZeros64(iBytes^jBytes)>>3
}
i, j = i+8, j+8
}
case "386":
// On a 32-bit CPU, we do it 4 bytes at a time.
for j+4 < len(src) {
iBytes := binary.LittleEndian.Uint32(src[i:])
jBytes := binary.LittleEndian.Uint32(src[j:])
if iBytes != jBytes {
return j + bits.TrailingZeros32(iBytes^jBytes)>>3
}
i, j = i+4, j+4
}
}
for ; j < len(src) && src[i] == src[j]; i, j = i+1, j+1 {
}
return j
}
// Given a 4-byte match at src[start] and src[candidate], extendMatch2 extends it
// upward as far as possible, and downward no farther than to min.
func extendMatch2(src []byte, start, candidate, min int) absoluteMatch {
end := extendMatch(src, candidate+4, start+4)
for start > min && candidate > 0 && src[start-1] == src[candidate-1] {
start--
candidate--
}
return absoluteMatch{
Start: start,
End: end,
Match: candidate,
}
}
+103
View File
@@ -0,0 +1,103 @@
// The matchfinder package defines reusable components for data compression.
//
// Many compression libraries have two main parts:
// - Something that looks for repeated sequences of bytes
// - An encoder for the compressed data format (often an entropy coder)
//
// Although these are logically two separate steps, the implementations are
// usually closely tied together. You can't use flate's matcher with snappy's
// encoder, for example. This package defines interfaces and an intermediate
// representation to allow mixing and matching compression components.
package matchfinder
import "io"
// A Match is the basic unit of LZ77 compression.
type Match struct {
Unmatched int // the number of unmatched bytes since the previous match
Length int // the number of bytes in the matched string; it may be 0 at the end of the input
Distance int // how far back in the stream to copy from
}
// A MatchFinder performs the LZ77 stage of compression, looking for matches.
type MatchFinder interface {
// FindMatches looks for matches in src, appends them to dst, and returns dst.
FindMatches(dst []Match, src []byte) []Match
// Reset clears any internal state, preparing the MatchFinder to be used with
// a new stream.
Reset()
}
// An Encoder encodes the data in its final format.
type Encoder interface {
// Encode appends the encoded format of src to dst, using the match
// information from matches.
Encode(dst []byte, src []byte, matches []Match, lastBlock bool) []byte
// Reset clears any internal state, preparing the Encoder to be used with
// a new stream.
Reset()
}
// A Writer uses MatchFinder and Encoder to write compressed data to Dest.
type Writer struct {
Dest io.Writer
MatchFinder MatchFinder
Encoder Encoder
// BlockSize is the number of bytes to compress at a time. If it is zero,
// each Write operation will be treated as one block.
BlockSize int
err error
inBuf []byte
outBuf []byte
matches []Match
}
func (w *Writer) Write(p []byte) (n int, err error) {
if w.err != nil {
return 0, w.err
}
if w.BlockSize == 0 {
return w.writeBlock(p, false)
}
w.inBuf = append(w.inBuf, p...)
var pos int
for pos = 0; pos+w.BlockSize <= len(w.inBuf) && w.err == nil; pos += w.BlockSize {
w.writeBlock(w.inBuf[pos:pos+w.BlockSize], false)
}
if pos > 0 {
n := copy(w.inBuf, w.inBuf[pos:])
w.inBuf = w.inBuf[:n]
}
return len(p), w.err
}
func (w *Writer) writeBlock(p []byte, lastBlock bool) (n int, err error) {
w.outBuf = w.outBuf[:0]
w.matches = w.MatchFinder.FindMatches(w.matches[:0], p)
w.outBuf = w.Encoder.Encode(w.outBuf, p, w.matches, lastBlock)
_, w.err = w.Dest.Write(w.outBuf)
return len(p), w.err
}
func (w *Writer) Close() error {
w.writeBlock(w.inBuf, true)
w.inBuf = w.inBuf[:0]
return w.err
}
func (w *Writer) Reset(newDest io.Writer) {
w.MatchFinder.Reset()
w.Encoder.Reset()
w.err = nil
w.inBuf = w.inBuf[:0]
w.outBuf = w.outBuf[:0]
w.matches = w.matches[:0]
w.Dest = newDest
}
+53
View File
@@ -0,0 +1,53 @@
package matchfinder
import "fmt"
// A TextEncoder is an Encoder that produces a human-readable representation of
// the LZ77 compression. Matches are replaced with <Length,Distance> symbols.
type TextEncoder struct{}
func (t TextEncoder) Reset() {}
func (t TextEncoder) Encode(dst []byte, src []byte, matches []Match, lastBlock bool) []byte {
pos := 0
for _, m := range matches {
if m.Unmatched > 0 {
dst = append(dst, src[pos:pos+m.Unmatched]...)
pos += m.Unmatched
}
if m.Length > 0 {
dst = append(dst, []byte(fmt.Sprintf("<%d,%d>", m.Length, m.Distance))...)
pos += m.Length
}
}
if pos < len(src) {
dst = append(dst, src[pos:]...)
}
return dst
}
// A NoMatchFinder implements MatchFinder, but doesn't find any matches.
// It can be used to implement the equivalent of the standard library flate package's
// HuffmanOnly setting.
type NoMatchFinder struct{}
func (n NoMatchFinder) Reset() {}
func (n NoMatchFinder) FindMatches(dst []Match, src []byte) []Match {
return append(dst, Match{
Unmatched: len(src),
})
}
// AutoReset wraps a MatchFinder that can return references to data in previous
// blocks, and calls Reset before each block. It is useful for (e.g.) using a
// snappy Encoder with a MatchFinder designed for flate. (Snappy doesn't
// support references between blocks.)
type AutoReset struct {
MatchFinder
}
func (a AutoReset) FindMatches(dst []Match, src []byte) []Match {
a.Reset()
return a.MatchFinder.FindMatches(dst, src)
}
+66
View File
@@ -0,0 +1,66 @@
package brotli
/* Copyright 2016 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/*
Dynamically grows array capacity to at least the requested size
T: data type
A: array
C: capacity
R: requested size
*/
func brotli_ensure_capacity_uint8_t(a *[]byte, c *uint, r uint) {
if *c < r {
var new_size uint = *c
if new_size == 0 {
new_size = r
}
for new_size < r {
new_size *= 2
}
if cap(*a) < int(new_size) {
var new_array []byte = make([]byte, new_size)
if *c != 0 {
copy(new_array, (*a)[:*c])
}
*a = new_array
} else {
*a = (*a)[:new_size]
}
*c = new_size
}
}
func brotli_ensure_capacity_uint32_t(a *[]uint32, c *uint, r uint) {
var new_array []uint32
if *c < r {
var new_size uint = *c
if new_size == 0 {
new_size = r
}
for new_size < r {
new_size *= 2
}
if cap(*a) < int(new_size) {
new_array = make([]uint32, new_size)
if *c != 0 {
copy(new_array, (*a)[:*c])
}
*a = new_array
} else {
*a = (*a)[:new_size]
}
*c = new_size
}
}
+574
View File
@@ -0,0 +1,574 @@
package brotli
import (
"sync"
)
/* Copyright 2014 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Algorithms for distributing the literals and commands of a metablock between
block types and contexts. */
type metaBlockSplit struct {
literal_split blockSplit
command_split blockSplit
distance_split blockSplit
literal_context_map []uint32
literal_context_map_size uint
distance_context_map []uint32
distance_context_map_size uint
literal_histograms []histogramLiteral
literal_histograms_size uint
command_histograms []histogramCommand
command_histograms_size uint
distance_histograms []histogramDistance
distance_histograms_size uint
}
var metaBlockPool sync.Pool
func getMetaBlockSplit() *metaBlockSplit {
mb, _ := metaBlockPool.Get().(*metaBlockSplit)
if mb == nil {
mb = &metaBlockSplit{}
} else {
initBlockSplit(&mb.literal_split)
initBlockSplit(&mb.command_split)
initBlockSplit(&mb.distance_split)
mb.literal_context_map = mb.literal_context_map[:0]
mb.literal_context_map_size = 0
mb.distance_context_map = mb.distance_context_map[:0]
mb.distance_context_map_size = 0
mb.literal_histograms = mb.literal_histograms[:0]
mb.command_histograms = mb.command_histograms[:0]
mb.distance_histograms = mb.distance_histograms[:0]
}
return mb
}
func freeMetaBlockSplit(mb *metaBlockSplit) {
metaBlockPool.Put(mb)
}
func initDistanceParams(params *encoderParams, npostfix uint32, ndirect uint32) {
var dist_params *distanceParams = &params.dist
var alphabet_size uint32
var max_distance uint32
dist_params.distance_postfix_bits = npostfix
dist_params.num_direct_distance_codes = ndirect
alphabet_size = uint32(distanceAlphabetSize(uint(npostfix), uint(ndirect), maxDistanceBits))
max_distance = ndirect + (1 << (maxDistanceBits + npostfix + 2)) - (1 << (npostfix + 2))
if params.large_window {
var bound = [maxNpostfix + 1]uint32{0, 4, 12, 28}
var postfix uint32 = 1 << npostfix
alphabet_size = uint32(distanceAlphabetSize(uint(npostfix), uint(ndirect), largeMaxDistanceBits))
/* The maximum distance is set so that no distance symbol used can encode
a distance larger than BROTLI_MAX_ALLOWED_DISTANCE with all
its extra bits set. */
if ndirect < bound[npostfix] {
max_distance = maxAllowedDistance - (bound[npostfix] - ndirect)
} else if ndirect >= bound[npostfix]+postfix {
max_distance = (3 << 29) - 4 + (ndirect - bound[npostfix])
} else {
max_distance = maxAllowedDistance
}
}
dist_params.alphabet_size = alphabet_size
dist_params.max_distance = uint(max_distance)
}
func recomputeDistancePrefixes(cmds []command, orig_params *distanceParams, new_params *distanceParams) {
if orig_params.distance_postfix_bits == new_params.distance_postfix_bits && orig_params.num_direct_distance_codes == new_params.num_direct_distance_codes {
return
}
for i := range cmds {
var cmd *command = &cmds[i]
if commandCopyLen(cmd) != 0 && cmd.cmd_prefix_ >= 128 {
prefixEncodeCopyDistance(uint(commandRestoreDistanceCode(cmd, orig_params)), uint(new_params.num_direct_distance_codes), uint(new_params.distance_postfix_bits), &cmd.dist_prefix_, &cmd.dist_extra_)
}
}
}
func computeDistanceCost(cmds []command, orig_params *distanceParams, new_params *distanceParams, cost *float64) bool {
var equal_params bool = false
var dist_prefix uint16
var dist_extra uint32
var extra_bits float64 = 0.0
var histo histogramDistance
histogramClearDistance(&histo)
if orig_params.distance_postfix_bits == new_params.distance_postfix_bits && orig_params.num_direct_distance_codes == new_params.num_direct_distance_codes {
equal_params = true
}
for i := range cmds {
cmd := &cmds[i]
if commandCopyLen(cmd) != 0 && cmd.cmd_prefix_ >= 128 {
if equal_params {
dist_prefix = cmd.dist_prefix_
} else {
var distance uint32 = commandRestoreDistanceCode(cmd, orig_params)
if distance > uint32(new_params.max_distance) {
return false
}
prefixEncodeCopyDistance(uint(distance), uint(new_params.num_direct_distance_codes), uint(new_params.distance_postfix_bits), &dist_prefix, &dist_extra)
}
histogramAddDistance(&histo, uint(dist_prefix)&0x3FF)
extra_bits += float64(dist_prefix >> 10)
}
}
*cost = populationCostDistance(&histo) + extra_bits
return true
}
var buildMetaBlock_kMaxNumberOfHistograms uint = 256
func buildMetaBlock(ringbuffer []byte, pos uint, mask uint, params *encoderParams, prev_byte byte, prev_byte2 byte, cmds []command, literal_context_mode int, mb *metaBlockSplit) {
var distance_histograms []histogramDistance
var literal_histograms []histogramLiteral
var literal_context_modes []int = nil
var literal_histograms_size uint
var distance_histograms_size uint
var i uint
var literal_context_multiplier uint = 1
var npostfix uint32
var ndirect_msb uint32 = 0
var check_orig bool = true
var best_dist_cost float64 = 1e99
var orig_params encoderParams = *params
/* Histogram ids need to fit in one byte. */
var new_params encoderParams = *params
for npostfix = 0; npostfix <= maxNpostfix; npostfix++ {
for ; ndirect_msb < 16; ndirect_msb++ {
var ndirect uint32 = ndirect_msb << npostfix
var skip bool
var dist_cost float64
initDistanceParams(&new_params, npostfix, ndirect)
if npostfix == orig_params.dist.distance_postfix_bits && ndirect == orig_params.dist.num_direct_distance_codes {
check_orig = false
}
skip = !computeDistanceCost(cmds, &orig_params.dist, &new_params.dist, &dist_cost)
if skip || (dist_cost > best_dist_cost) {
break
}
best_dist_cost = dist_cost
params.dist = new_params.dist
}
if ndirect_msb > 0 {
ndirect_msb--
}
ndirect_msb /= 2
}
if check_orig {
var dist_cost float64
computeDistanceCost(cmds, &orig_params.dist, &orig_params.dist, &dist_cost)
if dist_cost < best_dist_cost {
/* NB: currently unused; uncomment when more param tuning is added. */
/* best_dist_cost = dist_cost; */
params.dist = orig_params.dist
}
}
recomputeDistancePrefixes(cmds, &orig_params.dist, &params.dist)
splitBlock(cmds, ringbuffer, pos, mask, params, &mb.literal_split, &mb.command_split, &mb.distance_split)
if !params.disable_literal_context_modeling {
literal_context_multiplier = 1 << literalContextBits
literal_context_modes = make([]int, (mb.literal_split.num_types))
for i = 0; i < mb.literal_split.num_types; i++ {
literal_context_modes[i] = literal_context_mode
}
}
literal_histograms_size = mb.literal_split.num_types * literal_context_multiplier
literal_histograms = make([]histogramLiteral, literal_histograms_size)
clearHistogramsLiteral(literal_histograms, literal_histograms_size)
distance_histograms_size = mb.distance_split.num_types << distanceContextBits
distance_histograms = make([]histogramDistance, distance_histograms_size)
clearHistogramsDistance(distance_histograms, distance_histograms_size)
mb.command_histograms_size = mb.command_split.num_types
if cap(mb.command_histograms) < int(mb.command_histograms_size) {
mb.command_histograms = make([]histogramCommand, (mb.command_histograms_size))
} else {
mb.command_histograms = mb.command_histograms[:mb.command_histograms_size]
}
clearHistogramsCommand(mb.command_histograms, mb.command_histograms_size)
buildHistogramsWithContext(cmds, &mb.literal_split, &mb.command_split, &mb.distance_split, ringbuffer, pos, mask, prev_byte, prev_byte2, literal_context_modes, literal_histograms, mb.command_histograms, distance_histograms)
literal_context_modes = nil
mb.literal_context_map_size = mb.literal_split.num_types << literalContextBits
if cap(mb.literal_context_map) < int(mb.literal_context_map_size) {
mb.literal_context_map = make([]uint32, (mb.literal_context_map_size))
} else {
mb.literal_context_map = mb.literal_context_map[:mb.literal_context_map_size]
}
mb.literal_histograms_size = mb.literal_context_map_size
if cap(mb.literal_histograms) < int(mb.literal_histograms_size) {
mb.literal_histograms = make([]histogramLiteral, (mb.literal_histograms_size))
} else {
mb.literal_histograms = mb.literal_histograms[:mb.literal_histograms_size]
}
clusterHistogramsLiteral(literal_histograms, literal_histograms_size, buildMetaBlock_kMaxNumberOfHistograms, mb.literal_histograms, &mb.literal_histograms_size, mb.literal_context_map)
literal_histograms = nil
if params.disable_literal_context_modeling {
/* Distribute assignment to all contexts. */
for i = mb.literal_split.num_types; i != 0; {
var j uint = 0
i--
for ; j < 1<<literalContextBits; j++ {
mb.literal_context_map[(i<<literalContextBits)+j] = mb.literal_context_map[i]
}
}
}
mb.distance_context_map_size = mb.distance_split.num_types << distanceContextBits
if cap(mb.distance_context_map) < int(mb.distance_context_map_size) {
mb.distance_context_map = make([]uint32, (mb.distance_context_map_size))
} else {
mb.distance_context_map = mb.distance_context_map[:mb.distance_context_map_size]
}
mb.distance_histograms_size = mb.distance_context_map_size
if cap(mb.distance_histograms) < int(mb.distance_histograms_size) {
mb.distance_histograms = make([]histogramDistance, (mb.distance_histograms_size))
} else {
mb.distance_histograms = mb.distance_histograms[:mb.distance_histograms_size]
}
clusterHistogramsDistance(distance_histograms, mb.distance_context_map_size, buildMetaBlock_kMaxNumberOfHistograms, mb.distance_histograms, &mb.distance_histograms_size, mb.distance_context_map)
distance_histograms = nil
}
const maxStaticContexts = 13
/* Greedy block splitter for one block category (literal, command or distance).
Gathers histograms for all context buckets. */
type contextBlockSplitter struct {
alphabet_size_ uint
num_contexts_ uint
max_block_types_ uint
min_block_size_ uint
split_threshold_ float64
num_blocks_ uint
split_ *blockSplit
histograms_ []histogramLiteral
histograms_size_ *uint
target_block_size_ uint
block_size_ uint
curr_histogram_ix_ uint
last_histogram_ix_ [2]uint
last_entropy_ [2 * maxStaticContexts]float64
merge_last_count_ uint
}
func initContextBlockSplitter(self *contextBlockSplitter, alphabet_size uint, num_contexts uint, min_block_size uint, split_threshold float64, num_symbols uint, split *blockSplit, histograms *[]histogramLiteral, histograms_size *uint) {
var max_num_blocks uint = num_symbols/min_block_size + 1
var max_num_types uint
assert(num_contexts <= maxStaticContexts)
self.alphabet_size_ = alphabet_size
self.num_contexts_ = num_contexts
self.max_block_types_ = maxNumberOfBlockTypes / num_contexts
self.min_block_size_ = min_block_size
self.split_threshold_ = split_threshold
self.num_blocks_ = 0
self.split_ = split
self.histograms_size_ = histograms_size
self.target_block_size_ = min_block_size
self.block_size_ = 0
self.curr_histogram_ix_ = 0
self.merge_last_count_ = 0
/* We have to allocate one more histogram than the maximum number of block
types for the current histogram when the meta-block is too big. */
max_num_types = brotli_min_size_t(max_num_blocks, self.max_block_types_+1)
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, max_num_blocks)
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, max_num_blocks)
split.num_blocks = max_num_blocks
*histograms_size = max_num_types * num_contexts
if histograms == nil || cap(*histograms) < int(*histograms_size) {
*histograms = make([]histogramLiteral, (*histograms_size))
} else {
*histograms = (*histograms)[:*histograms_size]
}
self.histograms_ = *histograms
/* Clear only current histogram. */
clearHistogramsLiteral(self.histograms_[0:], num_contexts)
self.last_histogram_ix_[1] = 0
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
}
/* Does either of three things:
(1) emits the current block with a new block type;
(2) emits the current block with the type of the second last block;
(3) merges the current block with the last block. */
func contextBlockSplitterFinishBlock(self *contextBlockSplitter, is_final bool) {
var split *blockSplit = self.split_
var num_contexts uint = self.num_contexts_
var last_entropy []float64 = self.last_entropy_[:]
var histograms []histogramLiteral = self.histograms_
if self.block_size_ < self.min_block_size_ {
self.block_size_ = self.min_block_size_
}
if self.num_blocks_ == 0 {
var i uint
/* Create first block. */
split.lengths[0] = uint32(self.block_size_)
split.types[0] = 0
for i = 0; i < num_contexts; i++ {
last_entropy[i] = bitsEntropy(histograms[i].data_[:], self.alphabet_size_)
last_entropy[num_contexts+i] = last_entropy[i]
}
self.num_blocks_++
split.num_types++
self.curr_histogram_ix_ += num_contexts
if self.curr_histogram_ix_ < *self.histograms_size_ {
clearHistogramsLiteral(self.histograms_[self.curr_histogram_ix_:], self.num_contexts_)
}
self.block_size_ = 0
} else if self.block_size_ > 0 {
var entropy [maxStaticContexts]float64
var combined_histo []histogramLiteral = make([]histogramLiteral, (2 * num_contexts))
var combined_entropy [2 * maxStaticContexts]float64
var diff = [2]float64{0.0}
/* Try merging the set of histograms for the current block type with the
respective set of histograms for the last and second last block types.
Decide over the split based on the total reduction of entropy across
all contexts. */
var i uint
for i = 0; i < num_contexts; i++ {
var curr_histo_ix uint = self.curr_histogram_ix_ + i
var j uint
entropy[i] = bitsEntropy(histograms[curr_histo_ix].data_[:], self.alphabet_size_)
for j = 0; j < 2; j++ {
var jx uint = j*num_contexts + i
var last_histogram_ix uint = self.last_histogram_ix_[j] + i
combined_histo[jx] = histograms[curr_histo_ix]
histogramAddHistogramLiteral(&combined_histo[jx], &histograms[last_histogram_ix])
combined_entropy[jx] = bitsEntropy(combined_histo[jx].data_[0:], self.alphabet_size_)
diff[j] += combined_entropy[jx] - entropy[i] - last_entropy[jx]
}
}
if split.num_types < self.max_block_types_ && diff[0] > self.split_threshold_ && diff[1] > self.split_threshold_ {
/* Create new block. */
split.lengths[self.num_blocks_] = uint32(self.block_size_)
split.types[self.num_blocks_] = byte(split.num_types)
self.last_histogram_ix_[1] = self.last_histogram_ix_[0]
self.last_histogram_ix_[0] = split.num_types * num_contexts
for i = 0; i < num_contexts; i++ {
last_entropy[num_contexts+i] = last_entropy[i]
last_entropy[i] = entropy[i]
}
self.num_blocks_++
split.num_types++
self.curr_histogram_ix_ += num_contexts
if self.curr_histogram_ix_ < *self.histograms_size_ {
clearHistogramsLiteral(self.histograms_[self.curr_histogram_ix_:], self.num_contexts_)
}
self.block_size_ = 0
self.merge_last_count_ = 0
self.target_block_size_ = self.min_block_size_
} else if diff[1] < diff[0]-20.0 {
split.lengths[self.num_blocks_] = uint32(self.block_size_)
split.types[self.num_blocks_] = split.types[self.num_blocks_-2]
/* Combine this block with second last block. */
var tmp uint = self.last_histogram_ix_[0]
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
self.last_histogram_ix_[1] = tmp
for i = 0; i < num_contexts; i++ {
histograms[self.last_histogram_ix_[0]+i] = combined_histo[num_contexts+i]
last_entropy[num_contexts+i] = last_entropy[i]
last_entropy[i] = combined_entropy[num_contexts+i]
histogramClearLiteral(&histograms[self.curr_histogram_ix_+i])
}
self.num_blocks_++
self.block_size_ = 0
self.merge_last_count_ = 0
self.target_block_size_ = self.min_block_size_
} else {
/* Combine this block with last block. */
split.lengths[self.num_blocks_-1] += uint32(self.block_size_)
for i = 0; i < num_contexts; i++ {
histograms[self.last_histogram_ix_[0]+i] = combined_histo[i]
last_entropy[i] = combined_entropy[i]
if split.num_types == 1 {
last_entropy[num_contexts+i] = last_entropy[i]
}
histogramClearLiteral(&histograms[self.curr_histogram_ix_+i])
}
self.block_size_ = 0
self.merge_last_count_++
if self.merge_last_count_ > 1 {
self.target_block_size_ += self.min_block_size_
}
}
combined_histo = nil
}
if is_final {
*self.histograms_size_ = split.num_types * num_contexts
split.num_blocks = self.num_blocks_
}
}
/* Adds the next symbol to the current block type and context. When the
current block reaches the target size, decides on merging the block. */
func contextBlockSplitterAddSymbol(self *contextBlockSplitter, symbol uint, context uint) {
histogramAddLiteral(&self.histograms_[self.curr_histogram_ix_+context], symbol)
self.block_size_++
if self.block_size_ == self.target_block_size_ {
contextBlockSplitterFinishBlock(self, false) /* is_final = */
}
}
func mapStaticContexts(num_contexts uint, static_context_map []uint32, mb *metaBlockSplit) {
var i uint
mb.literal_context_map_size = mb.literal_split.num_types << literalContextBits
if cap(mb.literal_context_map) < int(mb.literal_context_map_size) {
mb.literal_context_map = make([]uint32, (mb.literal_context_map_size))
} else {
mb.literal_context_map = mb.literal_context_map[:mb.literal_context_map_size]
}
for i = 0; i < mb.literal_split.num_types; i++ {
var offset uint32 = uint32(i * num_contexts)
var j uint
for j = 0; j < 1<<literalContextBits; j++ {
mb.literal_context_map[(i<<literalContextBits)+j] = offset + static_context_map[j]
}
}
}
func buildMetaBlockGreedyInternal(ringbuffer []byte, pos uint, mask uint, prev_byte byte, prev_byte2 byte, literal_context_lut contextLUT, num_contexts uint, static_context_map []uint32, commands []command, mb *metaBlockSplit) {
var lit_blocks struct {
plain blockSplitterLiteral
ctx contextBlockSplitter
}
var cmd_blocks blockSplitterCommand
var dist_blocks blockSplitterDistance
var num_literals uint = 0
for i := range commands {
num_literals += uint(commands[i].insert_len_)
}
if num_contexts == 1 {
initBlockSplitterLiteral(&lit_blocks.plain, 256, 512, 400.0, num_literals, &mb.literal_split, &mb.literal_histograms, &mb.literal_histograms_size)
} else {
initContextBlockSplitter(&lit_blocks.ctx, 256, num_contexts, 512, 400.0, num_literals, &mb.literal_split, &mb.literal_histograms, &mb.literal_histograms_size)
}
initBlockSplitterCommand(&cmd_blocks, numCommandSymbols, 1024, 500.0, uint(len(commands)), &mb.command_split, &mb.command_histograms, &mb.command_histograms_size)
initBlockSplitterDistance(&dist_blocks, 64, 512, 100.0, uint(len(commands)), &mb.distance_split, &mb.distance_histograms, &mb.distance_histograms_size)
for _, cmd := range commands {
var j uint
blockSplitterAddSymbolCommand(&cmd_blocks, uint(cmd.cmd_prefix_))
for j = uint(cmd.insert_len_); j != 0; j-- {
var literal byte = ringbuffer[pos&mask]
if num_contexts == 1 {
blockSplitterAddSymbolLiteral(&lit_blocks.plain, uint(literal))
} else {
var context uint = uint(getContext(prev_byte, prev_byte2, literal_context_lut))
contextBlockSplitterAddSymbol(&lit_blocks.ctx, uint(literal), uint(static_context_map[context]))
}
prev_byte2 = prev_byte
prev_byte = literal
pos++
}
pos += uint(commandCopyLen(&cmd))
if commandCopyLen(&cmd) != 0 {
prev_byte2 = ringbuffer[(pos-2)&mask]
prev_byte = ringbuffer[(pos-1)&mask]
if cmd.cmd_prefix_ >= 128 {
blockSplitterAddSymbolDistance(&dist_blocks, uint(cmd.dist_prefix_)&0x3FF)
}
}
}
if num_contexts == 1 {
blockSplitterFinishBlockLiteral(&lit_blocks.plain, true) /* is_final = */
} else {
contextBlockSplitterFinishBlock(&lit_blocks.ctx, true) /* is_final = */
}
blockSplitterFinishBlockCommand(&cmd_blocks, true) /* is_final = */
blockSplitterFinishBlockDistance(&dist_blocks, true) /* is_final = */
if num_contexts > 1 {
mapStaticContexts(num_contexts, static_context_map, mb)
}
}
func buildMetaBlockGreedy(ringbuffer []byte, pos uint, mask uint, prev_byte byte, prev_byte2 byte, literal_context_lut contextLUT, num_contexts uint, static_context_map []uint32, commands []command, mb *metaBlockSplit) {
if num_contexts == 1 {
buildMetaBlockGreedyInternal(ringbuffer, pos, mask, prev_byte, prev_byte2, literal_context_lut, 1, nil, commands, mb)
} else {
buildMetaBlockGreedyInternal(ringbuffer, pos, mask, prev_byte, prev_byte2, literal_context_lut, num_contexts, static_context_map, commands, mb)
}
}
func optimizeHistograms(num_distance_codes uint32, mb *metaBlockSplit) {
var good_for_rle [numCommandSymbols]byte
var i uint
for i = 0; i < mb.literal_histograms_size; i++ {
optimizeHuffmanCountsForRLE(256, mb.literal_histograms[i].data_[:], good_for_rle[:])
}
for i = 0; i < mb.command_histograms_size; i++ {
optimizeHuffmanCountsForRLE(numCommandSymbols, mb.command_histograms[i].data_[:], good_for_rle[:])
}
for i = 0; i < mb.distance_histograms_size; i++ {
optimizeHuffmanCountsForRLE(uint(num_distance_codes), mb.distance_histograms[i].data_[:], good_for_rle[:])
}
}
+165
View File
@@ -0,0 +1,165 @@
package brotli
/* Copyright 2015 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Greedy block splitter for one block category (literal, command or distance).
*/
type blockSplitterCommand struct {
alphabet_size_ uint
min_block_size_ uint
split_threshold_ float64
num_blocks_ uint
split_ *blockSplit
histograms_ []histogramCommand
histograms_size_ *uint
target_block_size_ uint
block_size_ uint
curr_histogram_ix_ uint
last_histogram_ix_ [2]uint
last_entropy_ [2]float64
merge_last_count_ uint
}
func initBlockSplitterCommand(self *blockSplitterCommand, alphabet_size uint, min_block_size uint, split_threshold float64, num_symbols uint, split *blockSplit, histograms *[]histogramCommand, histograms_size *uint) {
var max_num_blocks uint = num_symbols/min_block_size + 1
var max_num_types uint = brotli_min_size_t(max_num_blocks, maxNumberOfBlockTypes+1)
/* We have to allocate one more histogram than the maximum number of block
types for the current histogram when the meta-block is too big. */
self.alphabet_size_ = alphabet_size
self.min_block_size_ = min_block_size
self.split_threshold_ = split_threshold
self.num_blocks_ = 0
self.split_ = split
self.histograms_size_ = histograms_size
self.target_block_size_ = min_block_size
self.block_size_ = 0
self.curr_histogram_ix_ = 0
self.merge_last_count_ = 0
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, max_num_blocks)
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, max_num_blocks)
self.split_.num_blocks = max_num_blocks
*histograms_size = max_num_types
if histograms == nil || cap(*histograms) < int(*histograms_size) {
*histograms = make([]histogramCommand, (*histograms_size))
} else {
*histograms = (*histograms)[:*histograms_size]
}
self.histograms_ = *histograms
/* Clear only current histogram. */
histogramClearCommand(&self.histograms_[0])
self.last_histogram_ix_[1] = 0
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
}
/* Does either of three things:
(1) emits the current block with a new block type;
(2) emits the current block with the type of the second last block;
(3) merges the current block with the last block. */
func blockSplitterFinishBlockCommand(self *blockSplitterCommand, is_final bool) {
var split *blockSplit = self.split_
var last_entropy []float64 = self.last_entropy_[:]
var histograms []histogramCommand = self.histograms_
self.block_size_ = brotli_max_size_t(self.block_size_, self.min_block_size_)
if self.num_blocks_ == 0 {
/* Create first block. */
split.lengths[0] = uint32(self.block_size_)
split.types[0] = 0
last_entropy[0] = bitsEntropy(histograms[0].data_[:], self.alphabet_size_)
last_entropy[1] = last_entropy[0]
self.num_blocks_++
split.num_types++
self.curr_histogram_ix_++
if self.curr_histogram_ix_ < *self.histograms_size_ {
histogramClearCommand(&histograms[self.curr_histogram_ix_])
}
self.block_size_ = 0
} else if self.block_size_ > 0 {
var entropy float64 = bitsEntropy(histograms[self.curr_histogram_ix_].data_[:], self.alphabet_size_)
var combined_histo [2]histogramCommand
var combined_entropy [2]float64
var diff [2]float64
var j uint
for j = 0; j < 2; j++ {
var last_histogram_ix uint = self.last_histogram_ix_[j]
combined_histo[j] = histograms[self.curr_histogram_ix_]
histogramAddHistogramCommand(&combined_histo[j], &histograms[last_histogram_ix])
combined_entropy[j] = bitsEntropy(combined_histo[j].data_[0:], self.alphabet_size_)
diff[j] = combined_entropy[j] - entropy - last_entropy[j]
}
if split.num_types < maxNumberOfBlockTypes && diff[0] > self.split_threshold_ && diff[1] > self.split_threshold_ {
/* Create new block. */
split.lengths[self.num_blocks_] = uint32(self.block_size_)
split.types[self.num_blocks_] = byte(split.num_types)
self.last_histogram_ix_[1] = self.last_histogram_ix_[0]
self.last_histogram_ix_[0] = uint(byte(split.num_types))
last_entropy[1] = last_entropy[0]
last_entropy[0] = entropy
self.num_blocks_++
split.num_types++
self.curr_histogram_ix_++
if self.curr_histogram_ix_ < *self.histograms_size_ {
histogramClearCommand(&histograms[self.curr_histogram_ix_])
}
self.block_size_ = 0
self.merge_last_count_ = 0
self.target_block_size_ = self.min_block_size_
} else if diff[1] < diff[0]-20.0 {
split.lengths[self.num_blocks_] = uint32(self.block_size_)
split.types[self.num_blocks_] = split.types[self.num_blocks_-2]
/* Combine this block with second last block. */
var tmp uint = self.last_histogram_ix_[0]
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
self.last_histogram_ix_[1] = tmp
histograms[self.last_histogram_ix_[0]] = combined_histo[1]
last_entropy[1] = last_entropy[0]
last_entropy[0] = combined_entropy[1]
self.num_blocks_++
self.block_size_ = 0
histogramClearCommand(&histograms[self.curr_histogram_ix_])
self.merge_last_count_ = 0
self.target_block_size_ = self.min_block_size_
} else {
/* Combine this block with last block. */
split.lengths[self.num_blocks_-1] += uint32(self.block_size_)
histograms[self.last_histogram_ix_[0]] = combined_histo[0]
last_entropy[0] = combined_entropy[0]
if split.num_types == 1 {
last_entropy[1] = last_entropy[0]
}
self.block_size_ = 0
histogramClearCommand(&histograms[self.curr_histogram_ix_])
self.merge_last_count_++
if self.merge_last_count_ > 1 {
self.target_block_size_ += self.min_block_size_
}
}
}
if is_final {
*self.histograms_size_ = split.num_types
split.num_blocks = self.num_blocks_
}
}
/* Adds the next symbol to the current histogram. When the current histogram
reaches the target size, decides on merging the block. */
func blockSplitterAddSymbolCommand(self *blockSplitterCommand, symbol uint) {
histogramAddCommand(&self.histograms_[self.curr_histogram_ix_], symbol)
self.block_size_++
if self.block_size_ == self.target_block_size_ {
blockSplitterFinishBlockCommand(self, false) /* is_final = */
}
}
+165
View File
@@ -0,0 +1,165 @@
package brotli
/* Copyright 2015 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Greedy block splitter for one block category (literal, command or distance).
*/
type blockSplitterDistance struct {
alphabet_size_ uint
min_block_size_ uint
split_threshold_ float64
num_blocks_ uint
split_ *blockSplit
histograms_ []histogramDistance
histograms_size_ *uint
target_block_size_ uint
block_size_ uint
curr_histogram_ix_ uint
last_histogram_ix_ [2]uint
last_entropy_ [2]float64
merge_last_count_ uint
}
func initBlockSplitterDistance(self *blockSplitterDistance, alphabet_size uint, min_block_size uint, split_threshold float64, num_symbols uint, split *blockSplit, histograms *[]histogramDistance, histograms_size *uint) {
var max_num_blocks uint = num_symbols/min_block_size + 1
var max_num_types uint = brotli_min_size_t(max_num_blocks, maxNumberOfBlockTypes+1)
/* We have to allocate one more histogram than the maximum number of block
types for the current histogram when the meta-block is too big. */
self.alphabet_size_ = alphabet_size
self.min_block_size_ = min_block_size
self.split_threshold_ = split_threshold
self.num_blocks_ = 0
self.split_ = split
self.histograms_size_ = histograms_size
self.target_block_size_ = min_block_size
self.block_size_ = 0
self.curr_histogram_ix_ = 0
self.merge_last_count_ = 0
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, max_num_blocks)
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, max_num_blocks)
self.split_.num_blocks = max_num_blocks
*histograms_size = max_num_types
if histograms == nil || cap(*histograms) < int(*histograms_size) {
*histograms = make([]histogramDistance, *histograms_size)
} else {
*histograms = (*histograms)[:*histograms_size]
}
self.histograms_ = *histograms
/* Clear only current histogram. */
histogramClearDistance(&self.histograms_[0])
self.last_histogram_ix_[1] = 0
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
}
/* Does either of three things:
(1) emits the current block with a new block type;
(2) emits the current block with the type of the second last block;
(3) merges the current block with the last block. */
func blockSplitterFinishBlockDistance(self *blockSplitterDistance, is_final bool) {
var split *blockSplit = self.split_
var last_entropy []float64 = self.last_entropy_[:]
var histograms []histogramDistance = self.histograms_
self.block_size_ = brotli_max_size_t(self.block_size_, self.min_block_size_)
if self.num_blocks_ == 0 {
/* Create first block. */
split.lengths[0] = uint32(self.block_size_)
split.types[0] = 0
last_entropy[0] = bitsEntropy(histograms[0].data_[:], self.alphabet_size_)
last_entropy[1] = last_entropy[0]
self.num_blocks_++
split.num_types++
self.curr_histogram_ix_++
if self.curr_histogram_ix_ < *self.histograms_size_ {
histogramClearDistance(&histograms[self.curr_histogram_ix_])
}
self.block_size_ = 0
} else if self.block_size_ > 0 {
var entropy float64 = bitsEntropy(histograms[self.curr_histogram_ix_].data_[:], self.alphabet_size_)
var combined_histo [2]histogramDistance
var combined_entropy [2]float64
var diff [2]float64
var j uint
for j = 0; j < 2; j++ {
var last_histogram_ix uint = self.last_histogram_ix_[j]
combined_histo[j] = histograms[self.curr_histogram_ix_]
histogramAddHistogramDistance(&combined_histo[j], &histograms[last_histogram_ix])
combined_entropy[j] = bitsEntropy(combined_histo[j].data_[0:], self.alphabet_size_)
diff[j] = combined_entropy[j] - entropy - last_entropy[j]
}
if split.num_types < maxNumberOfBlockTypes && diff[0] > self.split_threshold_ && diff[1] > self.split_threshold_ {
/* Create new block. */
split.lengths[self.num_blocks_] = uint32(self.block_size_)
split.types[self.num_blocks_] = byte(split.num_types)
self.last_histogram_ix_[1] = self.last_histogram_ix_[0]
self.last_histogram_ix_[0] = uint(byte(split.num_types))
last_entropy[1] = last_entropy[0]
last_entropy[0] = entropy
self.num_blocks_++
split.num_types++
self.curr_histogram_ix_++
if self.curr_histogram_ix_ < *self.histograms_size_ {
histogramClearDistance(&histograms[self.curr_histogram_ix_])
}
self.block_size_ = 0
self.merge_last_count_ = 0
self.target_block_size_ = self.min_block_size_
} else if diff[1] < diff[0]-20.0 {
split.lengths[self.num_blocks_] = uint32(self.block_size_)
split.types[self.num_blocks_] = split.types[self.num_blocks_-2]
/* Combine this block with second last block. */
var tmp uint = self.last_histogram_ix_[0]
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
self.last_histogram_ix_[1] = tmp
histograms[self.last_histogram_ix_[0]] = combined_histo[1]
last_entropy[1] = last_entropy[0]
last_entropy[0] = combined_entropy[1]
self.num_blocks_++
self.block_size_ = 0
histogramClearDistance(&histograms[self.curr_histogram_ix_])
self.merge_last_count_ = 0
self.target_block_size_ = self.min_block_size_
} else {
/* Combine this block with last block. */
split.lengths[self.num_blocks_-1] += uint32(self.block_size_)
histograms[self.last_histogram_ix_[0]] = combined_histo[0]
last_entropy[0] = combined_entropy[0]
if split.num_types == 1 {
last_entropy[1] = last_entropy[0]
}
self.block_size_ = 0
histogramClearDistance(&histograms[self.curr_histogram_ix_])
self.merge_last_count_++
if self.merge_last_count_ > 1 {
self.target_block_size_ += self.min_block_size_
}
}
}
if is_final {
*self.histograms_size_ = split.num_types
split.num_blocks = self.num_blocks_
}
}
/* Adds the next symbol to the current histogram. When the current histogram
reaches the target size, decides on merging the block. */
func blockSplitterAddSymbolDistance(self *blockSplitterDistance, symbol uint) {
histogramAddDistance(&self.histograms_[self.curr_histogram_ix_], symbol)
self.block_size_++
if self.block_size_ == self.target_block_size_ {
blockSplitterFinishBlockDistance(self, false) /* is_final = */
}
}
+165
View File
@@ -0,0 +1,165 @@
package brotli
/* Copyright 2015 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Greedy block splitter for one block category (literal, command or distance).
*/
type blockSplitterLiteral struct {
alphabet_size_ uint
min_block_size_ uint
split_threshold_ float64
num_blocks_ uint
split_ *blockSplit
histograms_ []histogramLiteral
histograms_size_ *uint
target_block_size_ uint
block_size_ uint
curr_histogram_ix_ uint
last_histogram_ix_ [2]uint
last_entropy_ [2]float64
merge_last_count_ uint
}
func initBlockSplitterLiteral(self *blockSplitterLiteral, alphabet_size uint, min_block_size uint, split_threshold float64, num_symbols uint, split *blockSplit, histograms *[]histogramLiteral, histograms_size *uint) {
var max_num_blocks uint = num_symbols/min_block_size + 1
var max_num_types uint = brotli_min_size_t(max_num_blocks, maxNumberOfBlockTypes+1)
/* We have to allocate one more histogram than the maximum number of block
types for the current histogram when the meta-block is too big. */
self.alphabet_size_ = alphabet_size
self.min_block_size_ = min_block_size
self.split_threshold_ = split_threshold
self.num_blocks_ = 0
self.split_ = split
self.histograms_size_ = histograms_size
self.target_block_size_ = min_block_size
self.block_size_ = 0
self.curr_histogram_ix_ = 0
self.merge_last_count_ = 0
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, max_num_blocks)
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, max_num_blocks)
self.split_.num_blocks = max_num_blocks
*histograms_size = max_num_types
if histograms == nil || cap(*histograms) < int(*histograms_size) {
*histograms = make([]histogramLiteral, *histograms_size)
} else {
*histograms = (*histograms)[:*histograms_size]
}
self.histograms_ = *histograms
/* Clear only current histogram. */
histogramClearLiteral(&self.histograms_[0])
self.last_histogram_ix_[1] = 0
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
}
/* Does either of three things:
(1) emits the current block with a new block type;
(2) emits the current block with the type of the second last block;
(3) merges the current block with the last block. */
func blockSplitterFinishBlockLiteral(self *blockSplitterLiteral, is_final bool) {
var split *blockSplit = self.split_
var last_entropy []float64 = self.last_entropy_[:]
var histograms []histogramLiteral = self.histograms_
self.block_size_ = brotli_max_size_t(self.block_size_, self.min_block_size_)
if self.num_blocks_ == 0 {
/* Create first block. */
split.lengths[0] = uint32(self.block_size_)
split.types[0] = 0
last_entropy[0] = bitsEntropy(histograms[0].data_[:], self.alphabet_size_)
last_entropy[1] = last_entropy[0]
self.num_blocks_++
split.num_types++
self.curr_histogram_ix_++
if self.curr_histogram_ix_ < *self.histograms_size_ {
histogramClearLiteral(&histograms[self.curr_histogram_ix_])
}
self.block_size_ = 0
} else if self.block_size_ > 0 {
var entropy float64 = bitsEntropy(histograms[self.curr_histogram_ix_].data_[:], self.alphabet_size_)
var combined_histo [2]histogramLiteral
var combined_entropy [2]float64
var diff [2]float64
var j uint
for j = 0; j < 2; j++ {
var last_histogram_ix uint = self.last_histogram_ix_[j]
combined_histo[j] = histograms[self.curr_histogram_ix_]
histogramAddHistogramLiteral(&combined_histo[j], &histograms[last_histogram_ix])
combined_entropy[j] = bitsEntropy(combined_histo[j].data_[0:], self.alphabet_size_)
diff[j] = combined_entropy[j] - entropy - last_entropy[j]
}
if split.num_types < maxNumberOfBlockTypes && diff[0] > self.split_threshold_ && diff[1] > self.split_threshold_ {
/* Create new block. */
split.lengths[self.num_blocks_] = uint32(self.block_size_)
split.types[self.num_blocks_] = byte(split.num_types)
self.last_histogram_ix_[1] = self.last_histogram_ix_[0]
self.last_histogram_ix_[0] = uint(byte(split.num_types))
last_entropy[1] = last_entropy[0]
last_entropy[0] = entropy
self.num_blocks_++
split.num_types++
self.curr_histogram_ix_++
if self.curr_histogram_ix_ < *self.histograms_size_ {
histogramClearLiteral(&histograms[self.curr_histogram_ix_])
}
self.block_size_ = 0
self.merge_last_count_ = 0
self.target_block_size_ = self.min_block_size_
} else if diff[1] < diff[0]-20.0 {
split.lengths[self.num_blocks_] = uint32(self.block_size_)
split.types[self.num_blocks_] = split.types[self.num_blocks_-2]
/* Combine this block with second last block. */
var tmp uint = self.last_histogram_ix_[0]
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
self.last_histogram_ix_[1] = tmp
histograms[self.last_histogram_ix_[0]] = combined_histo[1]
last_entropy[1] = last_entropy[0]
last_entropy[0] = combined_entropy[1]
self.num_blocks_++
self.block_size_ = 0
histogramClearLiteral(&histograms[self.curr_histogram_ix_])
self.merge_last_count_ = 0
self.target_block_size_ = self.min_block_size_
} else {
/* Combine this block with last block. */
split.lengths[self.num_blocks_-1] += uint32(self.block_size_)
histograms[self.last_histogram_ix_[0]] = combined_histo[0]
last_entropy[0] = combined_entropy[0]
if split.num_types == 1 {
last_entropy[1] = last_entropy[0]
}
self.block_size_ = 0
histogramClearLiteral(&histograms[self.curr_histogram_ix_])
self.merge_last_count_++
if self.merge_last_count_ > 1 {
self.target_block_size_ += self.min_block_size_
}
}
}
if is_final {
*self.histograms_size_ = split.num_types
split.num_blocks = self.num_blocks_
}
}
/* Adds the next symbol to the current histogram. When the current histogram
reaches the target size, decides on merging the block. */
func blockSplitterAddSymbolLiteral(self *blockSplitterLiteral, symbol uint) {
histogramAddLiteral(&self.histograms_[self.curr_histogram_ix_], symbol)
self.block_size_++
if self.block_size_ == self.target_block_size_ {
blockSplitterFinishBlockLiteral(self, false) /* is_final = */
}
}
+37
View File
@@ -0,0 +1,37 @@
package brotli
/* Copyright 2017 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Parameters for the Brotli encoder with chosen quality levels. */
type hasherParams struct {
type_ int
bucket_bits int
block_bits int
hash_len int
num_last_distances_to_check int
}
type distanceParams struct {
distance_postfix_bits uint32
num_direct_distance_codes uint32
alphabet_size uint32
max_distance uint
}
/* Encoding parameters */
type encoderParams struct {
mode int
quality int
lgwin uint
lgblock int
size_hint uint
disable_literal_context_modeling bool
large_window bool
hasher hasherParams
dist distanceParams
dictionary encoderDictionary
}
+103
View File
@@ -0,0 +1,103 @@
package brotli
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
func brotli_min_double(a float64, b float64) float64 {
if a < b {
return a
} else {
return b
}
}
func brotli_max_double(a float64, b float64) float64 {
if a > b {
return a
} else {
return b
}
}
func brotli_min_float(a float32, b float32) float32 {
if a < b {
return a
} else {
return b
}
}
func brotli_max_float(a float32, b float32) float32 {
if a > b {
return a
} else {
return b
}
}
func brotli_min_int(a int, b int) int {
if a < b {
return a
} else {
return b
}
}
func brotli_max_int(a int, b int) int {
if a > b {
return a
} else {
return b
}
}
func brotli_min_size_t(a uint, b uint) uint {
if a < b {
return a
} else {
return b
}
}
func brotli_max_size_t(a uint, b uint) uint {
if a > b {
return a
} else {
return b
}
}
func brotli_min_uint32_t(a uint32, b uint32) uint32 {
if a < b {
return a
} else {
return b
}
}
func brotli_max_uint32_t(a uint32, b uint32) uint32 {
if a > b {
return a
} else {
return b
}
}
func brotli_min_uint8_t(a byte, b byte) byte {
if a < b {
return a
} else {
return b
}
}
func brotli_max_uint8_t(a byte, b byte) byte {
if a > b {
return a
} else {
return b
}
}
+30
View File
@@ -0,0 +1,30 @@
package brotli
/* Copyright 2013 Google Inc. All Rights Reserved.
Distributed under MIT license.
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
*/
/* Functions for encoding of integers into prefix codes the amount of extra
bits, and the actual values of the extra bits. */
/* Here distance_code is an intermediate code, i.e. one of the special codes or
the actual distance increased by BROTLI_NUM_DISTANCE_SHORT_CODES - 1. */
func prefixEncodeCopyDistance(distance_code uint, num_direct_codes uint, postfix_bits uint, code *uint16, extra_bits *uint32) {
if distance_code < numDistanceShortCodes+num_direct_codes {
*code = uint16(distance_code)
*extra_bits = 0
return
} else {
var dist uint = (uint(1) << (postfix_bits + 2)) + (distance_code - numDistanceShortCodes - num_direct_codes)
var bucket uint = uint(log2FloorNonZero(dist) - 1)
var postfix_mask uint = (1 << postfix_bits) - 1
var postfix uint = dist & postfix_mask
var prefix uint = (dist >> bucket) & 1
var offset uint = (2 + prefix) << bucket
var nbits uint = bucket - postfix_bits
*code = uint16(nbits<<10 | (numDistanceShortCodes + num_direct_codes + ((2*(nbits-1) + prefix) << postfix_bits) + postfix))
*extra_bits = uint32((dist - offset) >> postfix_bits)
}
}
Loaded 100 of 1085 files, more files were not shown because too many files have changed in this diff. Show more