Compare commits
4
Commits
ff27075f41
...
new
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7350331b3a | ||
|
|
191785021e | ||
|
|
1838c1d559 | ||
|
|
a6f82d5a0b |
No files matched your search
+12
@@ -0,0 +1,12 @@
|
||||
# llm-api-benchmark-tool Project Rules & Patterns
|
||||
|
||||
1. **TDD is MANDATORY.** Write tests first (`test/` dir).
|
||||
2. **Client Implementation:** Use `FastHTTPClient` (`pkg/client/fasthttp.go`).
|
||||
3. `FastHTTPClient` uses `fasthttp` for `Do` (non-stream) and `net/http` for `Stream` (SSE).
|
||||
4. Streaming stability relies on the internal `net/http` usage.
|
||||
5. Configuration via `config/config.yaml` drives benchmark parameters.
|
||||
6. Core metrics defined in `stats.RequestResult` (`pkg/stats/collector.go`).
|
||||
7. Concurrency managed via goroutine pool in `pkg/concurrency/manager.go`.
|
||||
8. Use `tiktoken-go/tokenizer` for token operations.
|
||||
9. Record results (`statsCollector.RecordResult`) immediately after request completion in `runWorker`.
|
||||
10. Prefer fixing core functionality/bugs before large refactors.
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
# Workspace Rules for `llm-api-benchmark-tool`
|
||||
|
||||
## 1. Project Structure
|
||||
- **Root Directory**: `llm-api-benchmark-tool`
|
||||
- **Subdirectories**:
|
||||
- `cmd/`: Entry point for the main application.
|
||||
- `pkg/`: Core functionality modules.
|
||||
- `client/`: HTTP client abstraction layer.
|
||||
- `tokenizer/`: Prompt generation using `github.com/tiktoken-go/tokenizer`.
|
||||
- `concurrency/`: Concurrency management with goroutine pools and `sync.Pool`.
|
||||
- `stats/`: Performance metrics collection with `sync.Map` and `gonum/stat`.
|
||||
- `report/`: Report generation with `go-echarts`.
|
||||
- `test/`: Test cases for all modules.
|
||||
- `config/`: Configuration files (e.g., `config.yaml`).
|
||||
- `docs/`: Documentation.
|
||||
- `scripts/`: Utility scripts.
|
||||
|
||||
## 2. Test-Driven Development (TDD)
|
||||
- **MANDATORY TDD**: The entire project must follow test-driven development. Write test cases first, lock them, then iteratively develop code until all tests pass.
|
||||
- **MANDATORY TDD**: The entire project must follow test-driven development. Write test cases first, lock them, then iteratively develop code until all tests pass.
|
||||
- **MANDATORY TDD**: The entire project must follow test-driven development. Write test cases first, lock them, then iteratively develop code until all tests pass.
|
||||
- **Process**:
|
||||
1. Write test cases defining expected behavior, including edge cases and error handling.
|
||||
2. Run tests to ensure they fail (red).
|
||||
3. Write minimal code to pass tests (green).
|
||||
4. Refactor code while keeping tests passing.
|
||||
5. Repeat for each feature.
|
||||
|
||||
## 3. Technical Stack
|
||||
- **Language**: Go.
|
||||
- **HTTP Client**:
|
||||
- Interface: `HTTPClient` defined in `pkg/client/interface.go`.
|
||||
- Implementations:
|
||||
- `fasthttp` (`github.com/valyala/fasthttp`): High-performance, default choice.
|
||||
- `net/http` (standard library): Fallback for compatibility.
|
||||
- Configurable via `client: "fasthttp"` or `client: "net-http"` in `config.yaml`.
|
||||
- **Tokenizer**: `github.com/tiktoken-go/tokenizer` for prompt generation.
|
||||
- **Concurrency**: Goroutine pool (e.g., `ants` or custom), `sync.Pool` for object reuse, `context.Context` for control.
|
||||
- **Data Processing**: `sync.Map` for response storage, `gonum/stat` for percentile calculations.
|
||||
- **Charts**: `go-echarts` for visualizing metrics.
|
||||
|
||||
## 4. HTTP Client Abstraction Layer
|
||||
- **Interface**: `HTTPClient` with:
|
||||
- `Do(req *Request) (*Response, error)`: Non-streaming requests.
|
||||
- `Stream(req *Request, callback func(chunk SSEChunk) error) error`: Streaming SSE responses.
|
||||
- **Implementations**:
|
||||
- `pkg/client/fasthttp.go`: Uses `fasthttp` for high throughput.
|
||||
- `pkg/client/nethttp.go`: Uses `net/http` for standard compatibility.
|
||||
- **Tests**: `test/client_test.go` verifies `Do` and `Stream`, including TTFT measurement with SSE.
|
||||
- **Switching**: Config-driven client selection at runtime.
|
||||
|
||||
## 5. Prompt Generation
|
||||
- **Library**: `github.com/tiktoken-go/tokenizer`.
|
||||
- **Features**: Generate prompts of 50 tokens (±5%) and 1000 tokens (±5%).
|
||||
- **Tests**: `test/tokenizer_test.go` ensures token counts are within tolerance.
|
||||
|
||||
## 6. Concurrency Management
|
||||
- **Goroutine Pool**: Limit to 600 goroutines max, using `ants` or custom implementation.
|
||||
- **Object Reuse**: `sync.Pool` for `Request` and `Response` objects.
|
||||
- **Control**: `context.WithTimeout` for test duration (default 300s).
|
||||
- **Tests**: `test/concurrency_test.go` validates concurrency levels, intervals, and termination.
|
||||
|
||||
## 7. Performance Metrics Collection
|
||||
- **Storage**: `sync.Map` for thread-safe response data.
|
||||
- **Calculations**: `gonum/stat` for P90/P95/P99 percentiles.
|
||||
- **Metrics**:
|
||||
- Request stats (total, successful, failed, timeout ratio).
|
||||
- Response time (avg, min/max, percentiles).
|
||||
- TTFT (min/max, percentiles) via SSE.
|
||||
- QPS (avg, max).
|
||||
- Token generation rate (avg, max) for `content` and `reasoning_content`.
|
||||
- **Tests**: `test/stats_test.go` ensures accurate metrics.
|
||||
|
||||
## 8. Report Generation
|
||||
- **Charts**: Use `go-echarts` for response time, TTFT, QPS, and token rate visualizations.
|
||||
- **Format**: HTML/PDF reports via templates.
|
||||
- **Tests**: `test/report_test.go` validates report content and charts.
|
||||
|
||||
## 9. Configuration Management
|
||||
- **File**: `config/config.yaml` for API, prompts, timeouts, etc.
|
||||
- **Parsing**: Use `viper` or `yaml` package.
|
||||
- **Tests**: `test/config_test.go` verifies loading and parsing.
|
||||
|
||||
## 10. Logging and Monitoring
|
||||
- **Logging**: Use `log` package, enable debug logs with `--debug`.
|
||||
- **Monitoring**: Use `runtime` for CPU, memory, and goroutine tracking.
|
||||
- **Tests**: `test/log_test.go` and `test/monitor_test.go` ensure proper output.
|
||||
|
||||
## 11. Continuous Integration
|
||||
- **CI/CD**: GitHub Actions for automated testing.
|
||||
- **Coverage**: Target 80%+ code coverage.
|
||||
- **Automation**: Run tests on every commit.
|
||||
|
||||
## 12. Version Control
|
||||
- **Tool**: Git.
|
||||
- **Branches**: `main`, `develop`, `feature/*`, `bugfix/*`.
|
||||
|
||||
## 13. Code Standards
|
||||
- **Formatting**: `gofmt` for consistency.
|
||||
- **Linting**: `golint` or `staticcheck` for quality checks.
|
||||
- **Naming**: Follow Go conventions.
|
||||
|
||||
## 14. Dependency Management
|
||||
- **Tool**: `go mod` for dependencies.
|
||||
- **Optional**: `go mod vendor` for offline builds.
|
||||
|
||||
## 15. Security
|
||||
- **API Keys**: Use environment variables or config files, never hardcode.
|
||||
- **Error Handling**: Catch and log all errors to prevent crashes.
|
||||
|
||||
## 16. Performance Optimization
|
||||
- **Benchmarks**: Use `testing` package for performance tests.
|
||||
- **Profiling**: Use `pprof` for analysis.
|
||||
|
||||
## 17. User Engagement
|
||||
- **Feedback**: Encourage issues and PRs via GitHub.
|
||||
- **Guidelines**: Provide `CONTRIBUTING.md`.
|
||||
|
||||
## 18. Release Process
|
||||
- **Versioning**: Semantic versioning (e.g., `v1.0.0`).
|
||||
- **Changelog**: Maintain `CHANGELOG.md`.
|
||||
|
||||
## 19. Community
|
||||
- **Discussion**: Use GitHub Discussions.
|
||||
- **License**: MIT.
|
||||
|
||||
## 20. TDD Workflow
|
||||
- **Steps**:
|
||||
1. Write and lock test cases in `test/`.
|
||||
2. Implement code in `pkg/` or `cmd/`.
|
||||
3. Iterate until tests pass.
|
||||
4. Refactor and retest.
|
||||
- **Examples**: See `test/client_test.go`, `test/tokenizer_test.go`, etc.
|
||||
|
||||
## Summary
|
||||
These rules ensure a robust, testable, and maintainable `llm-api-benchmark-tool`. The TDD approach—writing tests first, locking them, and coding until they pass—is enforced throughout. Windsurf can leverage this structure to generate code, tests, and documentation systematically.
|
||||
@@ -1,52 +0,0 @@
|
||||
# LLM API 基准测试工具
|
||||
|
||||
这是一个用于测试兼容OpenAI风格API的LLM性能基准测试工具。该工具支持动态生成提示词、高并发请求和详细的性能报告。
|
||||
|
||||
## 功能特点
|
||||
|
||||
- **API兼容性**:支持OpenAI Compatible风格的LLM API
|
||||
- **提示词生成**:
|
||||
- 集成tiktoken-go/tokenizer
|
||||
- 自动生成短咨询提示词(50 tokens ±5%)和长文档提示词(1000 tokens ±5%)
|
||||
- 基于模板动态调整至目标Token数
|
||||
- **并发请求**:支持高并发异步请求,模拟真实负载
|
||||
- **场景建模**:支持混合负载(短咨询70%,长文档30%)
|
||||
- **用户行为模拟**:请求间隔遵循泊松分布
|
||||
- **梯度增压**:支持阶梯式加压(50→200→500并发)
|
||||
- **性能指标收集**:响应时间、QPS、Token生成速率等
|
||||
- **报告生成**:生成包含性能指标统计和可视化图表的质量报告
|
||||
|
||||
## 安装
|
||||
|
||||
```bash
|
||||
git clone https://github.com/yourusername/llm-api-benchmark-tool.git
|
||||
cd llm-api-benchmark-tool
|
||||
go mod download
|
||||
```
|
||||
|
||||
## 使用方法
|
||||
|
||||
1. 配置 `config.yaml` 文件:
|
||||
|
||||
```yaml
|
||||
api:
|
||||
endpoint: "https://api.example.com/v1/completions"
|
||||
api_key: "your_api_key"
|
||||
model: "gpt-3.5-turbo"
|
||||
```
|
||||
|
||||
2. 运行测试:
|
||||
|
||||
```bash
|
||||
go run main.go -config config.yaml
|
||||
```
|
||||
|
||||
3. 查看生成的报告(位于 `reports` 目录)
|
||||
|
||||
## 配置说明
|
||||
|
||||
详细配置选项请参考 [配置文档](docs/configuration.md)。
|
||||
|
||||
## 许可证
|
||||
|
||||
MIT
|
||||
@@ -1,673 +0,0 @@
|
||||
package benchmark
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"math/rand"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/montanaflynn/stats"
|
||||
"github.com/valyala/fasthttp"
|
||||
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/config"
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/logger"
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/prompt"
|
||||
)
|
||||
|
||||
// Benchmark 表示基准测试执行器
|
||||
type Benchmark struct {
|
||||
config *config.Config
|
||||
generator *prompt.Generator
|
||||
client *fasthttp.Client
|
||||
rand *rand.Rand
|
||||
}
|
||||
|
||||
// Response 表示API响应
|
||||
type Response struct {
|
||||
ID string `json:"id"`
|
||||
Object string `json:"object"`
|
||||
Created int64 `json:"created"`
|
||||
Model string `json:"model"`
|
||||
Choices []Choice `json:"choices"`
|
||||
Usage Usage `json:"usage"`
|
||||
}
|
||||
|
||||
// StreamResponse 表示流式API响应
|
||||
type StreamResponse struct {
|
||||
ID string `json:"id"`
|
||||
Object string `json:"object"`
|
||||
Created int64 `json:"created"`
|
||||
Model string `json:"model"`
|
||||
Choices []StreamChoice `json:"choices"`
|
||||
}
|
||||
|
||||
// StreamChoice 表示流式响应中的选择
|
||||
type StreamChoice struct {
|
||||
Delta StreamDelta `json:"delta"`
|
||||
Index int `json:"index"`
|
||||
FinishReason string `json:"finish_reason"`
|
||||
}
|
||||
|
||||
// StreamDelta 表示流式响应中的增量内容
|
||||
type StreamDelta struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
// Choice 表示API响应中的选择
|
||||
type Choice struct {
|
||||
Message Message `json:"message"`
|
||||
Index int `json:"index"`
|
||||
FinishReason string `json:"finish_reason"`
|
||||
}
|
||||
|
||||
// Message 表示聊天消息
|
||||
type Message struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
// Usage 表示API响应中的使用情况
|
||||
type Usage struct {
|
||||
PromptTokens int `json:"prompt_tokens"`
|
||||
CompletionTokens int `json:"completion_tokens"`
|
||||
TotalTokens int `json:"total_tokens"`
|
||||
}
|
||||
|
||||
// Request 表示API请求
|
||||
type Request struct {
|
||||
Model string `json:"model"`
|
||||
Messages []Message `json:"messages"`
|
||||
MaxTokens int `json:"max_tokens"`
|
||||
Temperature float64 `json:"temperature"`
|
||||
Stream bool `json:"stream"`
|
||||
}
|
||||
|
||||
// Result 表示单个API请求的结果
|
||||
type Result struct {
|
||||
Timestamp time.Time // 请求时间戳
|
||||
Concurrency int // 并发级别
|
||||
PromptType string // 提示词类型 (short/long)
|
||||
PromptTokenCount int // 提示词token数量
|
||||
PromptTokens int // 提示词token数量(API返回)
|
||||
CompletionTokens int // 完成token数量
|
||||
TotalTokens int // 总token数量
|
||||
ResponseTime time.Duration // 响应时间
|
||||
Response string // 响应内容
|
||||
Error error // 错误信息
|
||||
}
|
||||
|
||||
// BenchmarkResults 表示基准测试结果
|
||||
type BenchmarkResults struct {
|
||||
Results []Result // 所有请求结果
|
||||
StartTime time.Time // 开始时间
|
||||
EndTime time.Time // 结束时间
|
||||
Config *config.Config // 配置信息
|
||||
ConcurrencyData map[int][]Result // 按并发步骤分组的结果
|
||||
TotalRequests int // 总请求数
|
||||
SuccessRequests int // 成功请求数
|
||||
FailedRequests int // 失败请求数
|
||||
AvgResponseTime time.Duration // 平均响应时间
|
||||
P50ResponseTime time.Duration // 响应时间50百分位
|
||||
P90ResponseTime time.Duration // 响应时间90百分位
|
||||
P99ResponseTime time.Duration // 响应时间99百分位
|
||||
QPS float64 // 每秒请求数
|
||||
AvgPromptTokens int // 平均提示词token数
|
||||
AvgCompletionTokens int // 平均完成token数
|
||||
AvgTotalTokens int // 平均总token数
|
||||
AvgTokenRate float64 // 平均token生成速率
|
||||
}
|
||||
|
||||
// NewBenchmark 创建新的基准测试执行器
|
||||
func NewBenchmark(cfg *config.Config) (*Benchmark, error) {
|
||||
// 初始化提示词生成器
|
||||
logger.Debug("初始化提示词生成器...")
|
||||
gen, err := prompt.NewGenerator(cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("初始化提示词生成器失败: %w", err)
|
||||
}
|
||||
logger.Debug("提示词生成器初始化成功")
|
||||
|
||||
// 初始化HTTP客户端
|
||||
logger.Debug("初始化HTTP客户端...")
|
||||
client := &fasthttp.Client{
|
||||
MaxConnsPerHost: 10000,
|
||||
ReadTimeout: time.Duration(cfg.Timeout) * time.Second,
|
||||
WriteTimeout: time.Duration(cfg.Timeout) * time.Second,
|
||||
}
|
||||
logger.Debug("HTTP客户端初始化成功,超时设置: %d秒", cfg.Timeout)
|
||||
|
||||
// 初始化随机数生成器
|
||||
source := rand.NewSource(time.Now().UnixNano())
|
||||
|
||||
return &Benchmark{
|
||||
config: cfg,
|
||||
generator: gen,
|
||||
client: client,
|
||||
rand: rand.New(source),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Run 执行基准测试
|
||||
func (b *Benchmark) Run() (*BenchmarkResults, error) {
|
||||
// 创建结果对象
|
||||
results := &BenchmarkResults{
|
||||
StartTime: time.Now(),
|
||||
Config: b.config,
|
||||
ConcurrencyData: make(map[int][]Result),
|
||||
}
|
||||
|
||||
// 执行每个并发步骤
|
||||
for _, concurrency := range b.config.Concurrency.Steps {
|
||||
logger.Info("开始执行并发步骤: %d 并发用户", concurrency)
|
||||
logger.Info("预计持续时间: %d秒", b.config.Concurrency.DurationPerStep)
|
||||
|
||||
// 执行单个并发步骤
|
||||
stepResults, err := b.runConcurrencyStep(concurrency)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("执行并发步骤 %d 失败: %w", concurrency, err)
|
||||
}
|
||||
|
||||
// 统计成功率
|
||||
successCount := 0
|
||||
for _, result := range stepResults {
|
||||
if result.Error == nil {
|
||||
successCount++
|
||||
}
|
||||
// 添加到总结果
|
||||
results.Results = append(results.Results, result)
|
||||
}
|
||||
|
||||
// 记录步骤结果
|
||||
results.ConcurrencyData[concurrency] = stepResults
|
||||
|
||||
// 输出步骤统计
|
||||
successRate := 0.0
|
||||
if len(stepResults) > 0 {
|
||||
successRate = float64(successCount) / float64(len(stepResults)) * 100
|
||||
}
|
||||
logger.Info("完成并发步骤: %d 并发用户,共 %d 个请求,成功率: %.2f%%",
|
||||
concurrency, len(stepResults), successRate)
|
||||
}
|
||||
|
||||
results.EndTime = time.Now()
|
||||
logger.Info("所有并发步骤测试完成")
|
||||
|
||||
return results, nil
|
||||
}
|
||||
|
||||
// runConcurrencyStep 执行单个并发步骤
|
||||
func (b *Benchmark) runConcurrencyStep(concurrencyStep int) ([]Result, error) {
|
||||
var results []Result
|
||||
var resultsMutex sync.Mutex
|
||||
|
||||
// 创建上下文,用于控制测试时间
|
||||
ctx, cancel := context.WithTimeout(
|
||||
context.Background(),
|
||||
time.Duration(b.config.Concurrency.DurationPerStep)*time.Second,
|
||||
)
|
||||
defer cancel()
|
||||
|
||||
// 创建工作通道
|
||||
workChan := make(chan struct{}, concurrencyStep)
|
||||
|
||||
// 统计变量
|
||||
var statsMutex sync.Mutex
|
||||
totalRequests := 0
|
||||
successRequests := 0
|
||||
failedRequests := 0
|
||||
totalResponseTime := time.Duration(0)
|
||||
|
||||
// 启动统计协程
|
||||
statsTicker := time.NewTicker(5 * time.Second)
|
||||
defer statsTicker.Stop()
|
||||
|
||||
go func() {
|
||||
startTime := time.Now()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-statsTicker.C:
|
||||
statsMutex.Lock()
|
||||
currentTotal := totalRequests
|
||||
currentSuccess := successRequests
|
||||
currentFailed := failedRequests
|
||||
var avgResponseTime time.Duration
|
||||
if successRequests > 0 {
|
||||
avgResponseTime = totalResponseTime / time.Duration(successRequests)
|
||||
}
|
||||
statsMutex.Unlock()
|
||||
|
||||
elapsedTime := time.Since(startTime).Seconds()
|
||||
qps := float64(currentTotal) / elapsedTime
|
||||
|
||||
logger.Info("实时统计 - 请求: %d (成功: %d, 失败: %d), QPS: %.2f, 平均响应时间: %v",
|
||||
currentTotal, currentSuccess, currentFailed, qps, avgResponseTime)
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
// 启动工作协程
|
||||
var wg sync.WaitGroup
|
||||
logger.Debug("启动 %d 个工作协程...", concurrencyStep)
|
||||
for i := 0; i < concurrencyStep; i++ {
|
||||
wg.Add(1)
|
||||
go func(workerID int) {
|
||||
defer wg.Done()
|
||||
logger.Debug("工作协程 #%d 已启动", workerID)
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
// 测试时间结束
|
||||
logger.Debug("工作协程 #%d 收到结束信号", workerID)
|
||||
return
|
||||
case workChan <- struct{}{}:
|
||||
// 执行请求前先更新计数器
|
||||
statsMutex.Lock()
|
||||
totalRequests++
|
||||
requestID := totalRequests
|
||||
statsMutex.Unlock()
|
||||
|
||||
logger.Debug("工作协程 #%d 开始执行请求 #%d", workerID, requestID)
|
||||
|
||||
// 执行请求
|
||||
result := b.executeRequest(concurrencyStep)
|
||||
|
||||
// 更新统计
|
||||
statsMutex.Lock()
|
||||
if result.Error == nil {
|
||||
successRequests++
|
||||
totalResponseTime += result.ResponseTime
|
||||
logger.Debug("请求 #%d 成功,响应时间: %v", requestID, result.ResponseTime)
|
||||
} else {
|
||||
failedRequests++
|
||||
logger.Debug("请求 #%d 失败: %v", requestID, result.Error)
|
||||
}
|
||||
statsMutex.Unlock()
|
||||
|
||||
// 添加结果
|
||||
resultsMutex.Lock()
|
||||
results = append(results, result)
|
||||
resultsMutex.Unlock()
|
||||
|
||||
// 根据泊松分布等待
|
||||
waitTime := b.getPoissonWaitTime()
|
||||
logger.Debug("请求 #%d 完成,等待 %v 后发送下一个请求", requestID, waitTime)
|
||||
time.Sleep(waitTime)
|
||||
|
||||
<-workChan
|
||||
}
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
|
||||
// 等待所有工作协程完成
|
||||
logger.Debug("等待所有工作协程完成...")
|
||||
wg.Wait()
|
||||
logger.Debug("所有工作协程已完成")
|
||||
|
||||
return results, nil
|
||||
}
|
||||
|
||||
// executeRequest 执行单个请求
|
||||
func (b *Benchmark) executeRequest(concurrencyStep int) Result {
|
||||
result := Result{
|
||||
Timestamp: time.Now(),
|
||||
Concurrency: concurrencyStep,
|
||||
}
|
||||
|
||||
// 选择提示词类型
|
||||
promptType := b.generator.SelectPromptType()
|
||||
result.PromptType = promptType
|
||||
|
||||
// 生成提示词
|
||||
prompt, err := b.generator.GeneratePrompt(promptType)
|
||||
if err != nil {
|
||||
result.Error = fmt.Errorf("生成提示词失败: %w", err)
|
||||
return result
|
||||
}
|
||||
|
||||
// 创建请求
|
||||
req := Request{
|
||||
Model: b.config.API.Model,
|
||||
Messages: []Message{
|
||||
{
|
||||
Role: "user",
|
||||
Content: prompt,
|
||||
},
|
||||
},
|
||||
MaxTokens: 2048,
|
||||
Temperature: 0.7,
|
||||
Stream: true, // 启用流式响应
|
||||
}
|
||||
|
||||
// 序列化请求
|
||||
reqBody, err := json.Marshal(req)
|
||||
if err != nil {
|
||||
result.Error = fmt.Errorf("序列化请求失败: %w", err)
|
||||
return result
|
||||
}
|
||||
|
||||
// 创建HTTP请求
|
||||
httpReq := fasthttp.AcquireRequest()
|
||||
httpResp := fasthttp.AcquireResponse()
|
||||
defer fasthttp.ReleaseRequest(httpReq)
|
||||
defer fasthttp.ReleaseResponse(httpResp)
|
||||
|
||||
httpReq.SetRequestURI(b.config.API.Endpoint)
|
||||
httpReq.Header.SetMethod("POST")
|
||||
httpReq.Header.Set("Content-Type", "application/json")
|
||||
httpReq.Header.Set("Authorization", "Bearer "+b.config.API.APIKey)
|
||||
httpReq.Header.Set("Accept", "text/event-stream") // 设置接受流式响应
|
||||
httpReq.SetBody(reqBody)
|
||||
|
||||
// 记录开始时间
|
||||
startTime := time.Now()
|
||||
|
||||
// 发送请求
|
||||
logger.Debug("发送请求到 %s,提示词类型: %s,提示词长度: %d字符",
|
||||
b.config.API.Endpoint, promptType, len(prompt))
|
||||
|
||||
err = b.client.Do(httpReq, httpResp)
|
||||
|
||||
// 计算响应时间
|
||||
responseTime := time.Since(startTime)
|
||||
result.ResponseTime = responseTime
|
||||
|
||||
// 处理错误
|
||||
if err != nil {
|
||||
result.Error = fmt.Errorf("发送请求失败: %w", err)
|
||||
logger.Error("请求失败: %v", err)
|
||||
return result
|
||||
}
|
||||
|
||||
// 检查状态码
|
||||
statusCode := httpResp.StatusCode()
|
||||
if statusCode != 200 {
|
||||
result.Error = fmt.Errorf("API返回错误状态码: %d, 响应: %s", statusCode, httpResp.Body())
|
||||
logger.Error("API返回错误状态码: %d, 响应: %s", statusCode, httpResp.Body())
|
||||
return result
|
||||
}
|
||||
|
||||
// 处理流式响应
|
||||
if req.Stream {
|
||||
return b.handleStreamResponse(httpResp, result, startTime)
|
||||
}
|
||||
|
||||
// 处理非流式响应
|
||||
return b.handleNormalResponse(httpResp, result)
|
||||
}
|
||||
|
||||
// handleStreamResponse 处理流式响应
|
||||
func (b *Benchmark) handleStreamResponse(httpResp *fasthttp.Response, result Result, startTime time.Time) Result {
|
||||
body := httpResp.Body()
|
||||
responseContent := ""
|
||||
tokenCount := 0
|
||||
|
||||
// 打印响应头和原始响应体的前100个字符,用于调试
|
||||
logger.Debug("响应头: %s", httpResp.Header.String())
|
||||
if len(body) > 0 {
|
||||
previewLen := 100
|
||||
if len(body) < previewLen {
|
||||
previewLen = len(body)
|
||||
}
|
||||
logger.Debug("响应体预览: %s", string(body[:previewLen]))
|
||||
} else {
|
||||
logger.Warn("收到空响应体")
|
||||
result.Error = fmt.Errorf("收到空响应体")
|
||||
return result
|
||||
}
|
||||
|
||||
// 解析流式响应
|
||||
// 流式响应格式为: data: {...}\n\ndata: {...}\n\ndata: [DONE]\n\n
|
||||
lines := bytes.Split(body, []byte("\n\n"))
|
||||
if len(lines) == 0 {
|
||||
logger.Warn("无法解析流式响应")
|
||||
result.Error = fmt.Errorf("无法解析流式响应")
|
||||
return result
|
||||
}
|
||||
|
||||
logger.Debug("解析到 %d 个数据块", len(lines))
|
||||
|
||||
// 超时控制
|
||||
timeout := time.After(30 * time.Second)
|
||||
|
||||
// 使用通道处理解析结果
|
||||
doneChan := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
for _, line := range lines {
|
||||
if len(line) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// 检查是否为数据行
|
||||
if !bytes.HasPrefix(line, []byte("data: ")) {
|
||||
continue
|
||||
}
|
||||
|
||||
// 移除 "data: " 前缀
|
||||
data := line[6:]
|
||||
|
||||
// 检查是否为结束标记
|
||||
if bytes.Equal(data, []byte("[DONE]")) {
|
||||
logger.Debug("收到流式响应结束标记")
|
||||
break
|
||||
}
|
||||
|
||||
// 解析JSON
|
||||
var chunk map[string]interface{}
|
||||
if err := json.Unmarshal(data, &chunk); err != nil {
|
||||
logger.Warn("解析JSON数据块失败: %v, 数据: %s", err, string(data))
|
||||
continue
|
||||
}
|
||||
|
||||
// 提取内容
|
||||
choices, ok := chunk["choices"].([]interface{})
|
||||
if !ok || len(choices) == 0 {
|
||||
logger.Warn("无法从数据块中提取choices")
|
||||
continue
|
||||
}
|
||||
|
||||
choice, ok := choices[0].(map[string]interface{})
|
||||
if !ok {
|
||||
logger.Warn("无法从choices中提取第一个元素")
|
||||
continue
|
||||
}
|
||||
|
||||
delta, ok := choice["delta"].(map[string]interface{})
|
||||
if !ok {
|
||||
logger.Warn("无法从choice中提取delta")
|
||||
continue
|
||||
}
|
||||
|
||||
content, ok := delta["content"].(string)
|
||||
if !ok {
|
||||
// 某些delta可能没有content,这是正常的
|
||||
continue
|
||||
}
|
||||
|
||||
// 累加内容
|
||||
responseContent += content
|
||||
tokenCount += len(strings.Split(content, " "))
|
||||
}
|
||||
|
||||
close(doneChan)
|
||||
}()
|
||||
|
||||
// 等待处理完成或超时
|
||||
select {
|
||||
case <-doneChan:
|
||||
// 处理完成
|
||||
case <-timeout:
|
||||
logger.Warn("解析流式响应超时")
|
||||
result.Error = fmt.Errorf("解析流式响应超时")
|
||||
return result
|
||||
}
|
||||
|
||||
// 计算响应时间
|
||||
result.ResponseTime = time.Since(startTime)
|
||||
|
||||
// 设置结果
|
||||
result.Response = responseContent
|
||||
result.PromptTokenCount = len(result.PromptType) / 2 // 简单估计
|
||||
result.PromptTokens = len(result.PromptType) / 2 // 简单估计
|
||||
result.CompletionTokens = tokenCount
|
||||
result.TotalTokens = result.PromptTokens + result.CompletionTokens
|
||||
|
||||
logger.Debug("流式响应处理完成,响应时间: %v, 提示词Token: %d, 完成Token: %d, 总Token: %d",
|
||||
result.ResponseTime, result.PromptTokens, result.CompletionTokens, result.TotalTokens)
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// handleNormalResponse 处理普通响应
|
||||
func (b *Benchmark) handleNormalResponse(httpResp *fasthttp.Response, result Result) Result {
|
||||
// 解析响应
|
||||
var response Response
|
||||
if err := json.Unmarshal(httpResp.Body(), &response); err != nil {
|
||||
result.Error = fmt.Errorf("解析响应失败: %w", err)
|
||||
logger.Error("解析响应失败: %v", err)
|
||||
return result
|
||||
}
|
||||
|
||||
// 记录Token数
|
||||
result.PromptTokens = response.Usage.PromptTokens
|
||||
result.CompletionTokens = response.Usage.CompletionTokens
|
||||
result.TotalTokens = response.Usage.TotalTokens
|
||||
|
||||
// 记录详细日志
|
||||
logger.Debug("请求完成 - 类型: %s, 提示词Token: %d, 响应Token: %d, 响应时间: %v",
|
||||
result.PromptType, result.PromptTokens, result.CompletionTokens, result.ResponseTime)
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// getPoissonWaitTime 根据泊松分布获取等待时间
|
||||
func (b *Benchmark) getPoissonWaitTime() time.Duration {
|
||||
// 生成泊松分布随机数
|
||||
lambda := b.config.PoissonLambda
|
||||
L := math.Exp(-lambda)
|
||||
k := 0
|
||||
p := 1.0
|
||||
|
||||
for p > L {
|
||||
k++
|
||||
p *= b.rand.Float64()
|
||||
}
|
||||
|
||||
// 转换为等待时间(毫秒)
|
||||
waitTime := time.Duration(k * 100) * time.Millisecond
|
||||
|
||||
// 限制最大等待时间为1秒
|
||||
if waitTime > time.Second {
|
||||
waitTime = time.Second
|
||||
}
|
||||
|
||||
return waitTime
|
||||
}
|
||||
|
||||
// convertToFloat64 将整数切片转换为浮点数切片
|
||||
func convertToFloat64(ints []int) []float64 {
|
||||
floats := make([]float64, len(ints))
|
||||
for i, v := range ints {
|
||||
floats[i] = float64(v)
|
||||
}
|
||||
return floats
|
||||
}
|
||||
|
||||
// calculateStats 计算统计数据
|
||||
func (b *Benchmark) calculateStats(results []Result) *BenchmarkResults {
|
||||
var stats BenchmarkResults
|
||||
|
||||
// 过滤出成功的请求
|
||||
var successResults []Result
|
||||
for _, r := range results {
|
||||
if r.Error == nil {
|
||||
successResults = append(successResults, r)
|
||||
}
|
||||
}
|
||||
|
||||
// 如果没有成功的请求,返回空结果
|
||||
if len(successResults) == 0 {
|
||||
return &stats
|
||||
}
|
||||
|
||||
// 计算基本统计数据
|
||||
stats.TotalRequests = len(results)
|
||||
stats.SuccessRequests = len(successResults)
|
||||
stats.FailedRequests = stats.TotalRequests - stats.SuccessRequests
|
||||
|
||||
// 计算响应时间统计
|
||||
var responseTimes []float64
|
||||
for _, r := range successResults {
|
||||
responseTimes = append(responseTimes, float64(r.ResponseTime.Milliseconds()))
|
||||
}
|
||||
|
||||
// 计算平均响应时间
|
||||
avg, _ := stats.Mean(responseTimes)
|
||||
stats.AvgResponseTime = time.Duration(avg) * time.Millisecond
|
||||
|
||||
// 计算响应时间百分位数
|
||||
p50, _ := stats.Percentile(responseTimes, 50)
|
||||
p90, _ := stats.Percentile(responseTimes, 90)
|
||||
p99, _ := stats.Percentile(responseTimes, 99)
|
||||
stats.P50ResponseTime = time.Duration(p50) * time.Millisecond
|
||||
stats.P90ResponseTime = time.Duration(p90) * time.Millisecond
|
||||
stats.P99ResponseTime = time.Duration(p99) * time.Millisecond
|
||||
|
||||
// 计算QPS
|
||||
totalDuration := successResults[len(successResults)-1].Timestamp.Sub(successResults[0].Timestamp).Seconds()
|
||||
if totalDuration > 0 {
|
||||
stats.QPS = float64(stats.SuccessRequests) / totalDuration
|
||||
}
|
||||
|
||||
// 计算Token统计
|
||||
var promptTokens, completionTokens, totalTokens []int
|
||||
for _, r := range successResults {
|
||||
promptTokens = append(promptTokens, r.PromptTokens)
|
||||
completionTokens = append(completionTokens, r.CompletionTokens)
|
||||
totalTokens = append(totalTokens, r.TotalTokens)
|
||||
}
|
||||
|
||||
// 计算平均Token数
|
||||
if len(promptTokens) > 0 {
|
||||
promptAvg, _ := stats.Mean(convertToFloat64(promptTokens))
|
||||
completionAvg, _ := stats.Mean(convertToFloat64(completionTokens))
|
||||
totalAvg, _ := stats.Mean(convertToFloat64(totalTokens))
|
||||
stats.AvgPromptTokens = int(promptAvg)
|
||||
stats.AvgCompletionTokens = int(completionAvg)
|
||||
stats.AvgTotalTokens = int(totalAvg)
|
||||
}
|
||||
|
||||
// 计算Token生成速率 (tokens/s)
|
||||
var tokenRates []float64
|
||||
for _, r := range successResults {
|
||||
if r.ResponseTime.Seconds() > 0 {
|
||||
tokenRate := float64(r.CompletionTokens) / r.ResponseTime.Seconds()
|
||||
tokenRates = append(tokenRates, tokenRate)
|
||||
}
|
||||
}
|
||||
|
||||
if len(tokenRates) > 0 {
|
||||
tokenRateAvg, _ := stats.Mean(tokenRates)
|
||||
stats.AvgTokenRate = tokenRateAvg
|
||||
}
|
||||
|
||||
return &stats
|
||||
}
|
||||
|
||||
func (b *BenchmarkResults) Mean(data []float64) (float64, error) {
|
||||
return stats.Mean(data)
|
||||
}
|
||||
|
||||
func (b *BenchmarkResults) Percentile(data []float64, p float64) (float64, error) {
|
||||
return stats.Percentile(data, p)
|
||||
}
|
||||
Executable
BIN
Binary file not shown.
@@ -0,0 +1,111 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"log"
|
||||
"time"
|
||||
"context"
|
||||
|
||||
"llm-api-benchmark-tool/pkg/client"
|
||||
"llm-api-benchmark-tool/pkg/config"
|
||||
"llm-api-benchmark-tool/pkg/concurrency"
|
||||
"llm-api-benchmark-tool/pkg/stats"
|
||||
"llm-api-benchmark-tool/pkg/tokenizer"
|
||||
"llm-api-benchmark-tool/pkg/report"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Define command-line flags
|
||||
configPath := flag.String("config", "config/config.yaml", "Path to the configuration YAML file")
|
||||
flag.Parse()
|
||||
|
||||
// Load configuration
|
||||
cfg, cfgErr := config.LoadConfig(*configPath)
|
||||
if cfgErr != nil {
|
||||
log.Fatalf("Failed to load config: %v", cfgErr)
|
||||
}
|
||||
|
||||
// Check Tokenizer Initialization Status (happens in init() func)
|
||||
tknErr := tokenizer.CheckInitStatus()
|
||||
if tknErr != nil {
|
||||
log.Fatalf("Tokenizer initialization failed: %v", tknErr)
|
||||
}
|
||||
log.Println("Tokenizer initialization checked successfully.")
|
||||
|
||||
// Initialize HTTP Client
|
||||
// Determine the effective request timeout
|
||||
requestTimeout := cfg.RequestTimeout
|
||||
if requestTimeout <= 0 {
|
||||
requestTimeout = cfg.Timeout // Fallback to global timeout
|
||||
}
|
||||
// Always create FastHTTPClient; it handles both streaming (net/http) and non-streaming (fasthttp)
|
||||
httpClient := client.NewFastHTTPClient(requestTimeout)
|
||||
|
||||
log.Println("Configuration loaded successfully.")
|
||||
log.Printf("API Endpoint: %s", cfg.APIEndpoint)
|
||||
// The actual client used (fasthttp for Do, net/http for Stream) is handled internally by FastHTTPClient
|
||||
log.Printf("Using HTTP client implementation: FastHTTPClient (non-stream timeout: %s)", requestTimeout)
|
||||
log.Println("HTTP Client initialized.")
|
||||
|
||||
// Initialize Stats Collector
|
||||
statsCollector := stats.NewStatsCollector()
|
||||
log.Println("Stats Collector initialized.")
|
||||
|
||||
// Initialize Concurrency Manager
|
||||
benchmarkParams := concurrency.BenchmarkParams{
|
||||
HTTPClient: httpClient,
|
||||
TargetURL: cfg.APIEndpoint,
|
||||
APIKey: cfg.APIKey,
|
||||
TotalDuration: cfg.Timeout,
|
||||
Concurrency: cfg.Concurrency,
|
||||
RequestInterval: time.Duration(cfg.RateLimit) * time.Millisecond, // Convert ms to time.Duration
|
||||
IsStreaming: cfg.Streaming,
|
||||
PromptTokens: cfg.PromptTokens,
|
||||
Model: cfg.Model,
|
||||
// TODO: Add other params like API Key, Model from cfg if needed
|
||||
}
|
||||
concurrencyManager := concurrency.NewManager(benchmarkParams, statsCollector)
|
||||
log.Println("Concurrency Manager initialized.")
|
||||
log.Printf("Benchmark settings: Duration=%s, Concurrency=%d, RateLimit=%s, Streaming=%v, PromptTokens=%d, Model=%s",
|
||||
benchmarkParams.TotalDuration, benchmarkParams.Concurrency, benchmarkParams.RequestInterval, benchmarkParams.IsStreaming, benchmarkParams.PromptTokens, benchmarkParams.Model)
|
||||
|
||||
// --- Run Benchmark ---
|
||||
log.Println("Starting benchmark...")
|
||||
ctx := context.Background()
|
||||
if err := concurrencyManager.RunBenchmark(ctx); err != nil {
|
||||
log.Fatalf("Benchmark run failed: %v", err)
|
||||
}
|
||||
log.Println("Benchmark finished.")
|
||||
|
||||
// --- Get Final Stats ---
|
||||
finalStats := statsCollector.CalculateStats(benchmarkParams.TotalDuration)
|
||||
log.Println("--- Benchmark Results ---")
|
||||
log.Printf("Total Duration: %.2fs", finalStats.TotalDuration.Seconds())
|
||||
log.Printf("Total Requests: %d", finalStats.TotalRequests)
|
||||
log.Printf("Successful Requests: %d", finalStats.SuccessfulRequests)
|
||||
log.Printf("Failed Requests: %d", finalStats.FailedRequests)
|
||||
if finalStats.TotalRequests > 0 {
|
||||
log.Printf("Success Rate: %.2f%%", float64(finalStats.SuccessfulRequests)/float64(finalStats.TotalRequests)*100)
|
||||
}
|
||||
log.Printf("Average QPS: %.2f", finalStats.AvgQPS)
|
||||
if finalStats.SuccessfulRequests > 0 {
|
||||
log.Printf("Average Latency (Successful): %s", finalStats.AvgLatency)
|
||||
log.Printf("Latency (Min/Max): %s / %s", finalStats.MinLatency, finalStats.MaxLatency)
|
||||
log.Printf("Latency (P90/P95/P99): %s / %s / %s", finalStats.P90Latency, finalStats.P95Latency, finalStats.P99Latency)
|
||||
log.Printf("Average TTFT (Successful): %s", finalStats.AvgTimeToFirstToken)
|
||||
log.Printf("TTFT (Min/Max): %s / %s", finalStats.MinTimeToFirstToken, finalStats.MaxTimeToFirstToken)
|
||||
log.Printf("TTFT (P90/P95/P99): %s / %s / %s", finalStats.P90TimeToFirstToken, finalStats.P95TimeToFirstToken, finalStats.P99TimeToFirstToken)
|
||||
log.Printf("Average Tokens/Second (Successful): %.2f", finalStats.AvgTokensPerSecond)
|
||||
}
|
||||
log.Println("-------------------------")
|
||||
|
||||
// --- Generate Report ---
|
||||
if cfg.ReportFile != "" {
|
||||
log.Printf("Generating HTML report to: %s", cfg.ReportFile)
|
||||
if err := report.GenerateHTMLReport(finalStats, cfg.ReportFile); err != nil {
|
||||
log.Printf("Error generating report: %v", err)
|
||||
} else {
|
||||
log.Println("Report generated successfully.")
|
||||
}
|
||||
}
|
||||
}
|
||||
-38
@@ -1,38 +0,0 @@
|
||||
api:
|
||||
endpoint: "http://139.9.229.200:8080/v1/chat/completions"
|
||||
api_key: "devserver-9aB3-7fG5q2Yd-8C4T-1R6k-Ew9p-3MnL7bZx4Wv2"
|
||||
model: "DeepSeek-R1"
|
||||
|
||||
prompt_templates:
|
||||
short:
|
||||
target_tokens: 50
|
||||
templates:
|
||||
- "什么是{concept}?"
|
||||
- "请简要解释{concept}的基本原理。"
|
||||
- "简述{topic}的主要观点。"
|
||||
- "{country}的首都是哪里?"
|
||||
- "如何快速入门{concept}?"
|
||||
long:
|
||||
target_tokens: 1000
|
||||
templates:
|
||||
- "请撰写一篇关于{country}历史的详细介绍,包括主要历史事件和文化发展。"
|
||||
- "请对{topic}进行深入分析,包括其背景、现状、问题和未来发展趋势。"
|
||||
- "请撰写一份关于{event}影响的综合报告,包括社会、经济和政治层面的分析。"
|
||||
- "请详细介绍{concept}的技术原理、应用场景和未来发展方向。"
|
||||
- "请撰写一篇关于{topic}的学术论文,包括研究背景、方法、结果和讨论。"
|
||||
|
||||
requests:
|
||||
- type: "short"
|
||||
weight: 0.7
|
||||
- type: "long"
|
||||
weight: 0.3
|
||||
|
||||
concurrency:
|
||||
steps: [1,2]
|
||||
duration_per_step: 60 # 秒
|
||||
|
||||
timeout: 60 # 秒
|
||||
poisson_lambda: 1.0 # 请求间隔泊松分布参数
|
||||
|
||||
tokenizer:
|
||||
model: "gpt-3.5-turbo" # 用于tiktoken-go的分词器模型
|
||||
@@ -1,103 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/spf13/viper"
|
||||
)
|
||||
|
||||
// Config 表示应用程序的配置
|
||||
type Config struct {
|
||||
API APIConfig `mapstructure:"api"`
|
||||
PromptTemplates PromptTemplatesConfig `mapstructure:"prompt_templates"`
|
||||
Requests []RequestConfig `mapstructure:"requests"`
|
||||
Concurrency ConcurrencyConfig `mapstructure:"concurrency"`
|
||||
Timeout int `mapstructure:"timeout"`
|
||||
PoissonLambda float64 `mapstructure:"poisson_lambda"`
|
||||
Tokenizer TokenizerConfig `mapstructure:"tokenizer"`
|
||||
}
|
||||
|
||||
// APIConfig 表示API相关配置
|
||||
type APIConfig struct {
|
||||
Endpoint string `mapstructure:"endpoint"`
|
||||
APIKey string `mapstructure:"api_key"`
|
||||
Model string `mapstructure:"model"`
|
||||
}
|
||||
|
||||
// PromptTemplatesConfig 表示提示词模板配置
|
||||
type PromptTemplatesConfig struct {
|
||||
Short PromptTypeConfig `mapstructure:"short"`
|
||||
Long PromptTypeConfig `mapstructure:"long"`
|
||||
}
|
||||
|
||||
// PromptTypeConfig 表示特定类型提示词的配置
|
||||
type PromptTypeConfig struct {
|
||||
TargetTokens int `mapstructure:"target_tokens"`
|
||||
Templates []string `mapstructure:"templates"`
|
||||
}
|
||||
|
||||
// RequestConfig 表示请求配置
|
||||
type RequestConfig struct {
|
||||
Type string `mapstructure:"type"`
|
||||
Weight float64 `mapstructure:"weight"`
|
||||
}
|
||||
|
||||
// ConcurrencyConfig 表示并发配置
|
||||
type ConcurrencyConfig struct {
|
||||
Steps []int `mapstructure:"steps"`
|
||||
DurationPerStep int `mapstructure:"duration_per_step"`
|
||||
}
|
||||
|
||||
// TokenizerConfig 表示分词器配置
|
||||
type TokenizerConfig struct {
|
||||
Model string `mapstructure:"model"`
|
||||
}
|
||||
|
||||
// LoadConfig 从指定路径加载配置
|
||||
func LoadConfig(path string) (*Config, error) {
|
||||
viper.SetConfigFile(path)
|
||||
|
||||
if err := viper.ReadInConfig(); err != nil {
|
||||
return nil, fmt.Errorf("读取配置文件失败: %w", err)
|
||||
}
|
||||
|
||||
var config Config
|
||||
if err := viper.Unmarshal(&config); err != nil {
|
||||
return nil, fmt.Errorf("解析配置失败: %w", err)
|
||||
}
|
||||
|
||||
// 验证配置
|
||||
if err := validateConfig(&config); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &config, nil
|
||||
}
|
||||
|
||||
// validateConfig 验证配置的有效性
|
||||
func validateConfig(config *Config) error {
|
||||
if config.API.Endpoint == "" {
|
||||
return fmt.Errorf("API端点不能为空")
|
||||
}
|
||||
|
||||
if config.API.APIKey == "" {
|
||||
return fmt.Errorf("API密钥不能为空")
|
||||
}
|
||||
|
||||
if config.API.Model == "" {
|
||||
return fmt.Errorf("模型名称不能为空")
|
||||
}
|
||||
|
||||
if len(config.Requests) == 0 {
|
||||
return fmt.Errorf("请求配置不能为空")
|
||||
}
|
||||
|
||||
if len(config.Concurrency.Steps) == 0 {
|
||||
return fmt.Errorf("并发步骤不能为空")
|
||||
}
|
||||
|
||||
if config.Timeout <= 0 {
|
||||
return fmt.Errorf("超时时间必须大于0")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
# Basic Benchmark Configuration
|
||||
apiEndpoint: "http://139.9.229.200:8080/v1/chat/completions" # Replace with your actual API endpoint
|
||||
apiKey: "devserver-9aB3-7fG5q2Yd-8C4T-1R6k-Ew9p-3MnL7bZx4Wv2" # Replace with your actual API key or load from environment
|
||||
model: "DeepSeek-R1" # Replace with the model you want to benchmark
|
||||
|
||||
requests: 10 # Total number of requests to send
|
||||
concurrency: 5 # Number of concurrent workers
|
||||
rateLimit: 0 # Requests per second (0 means no limit)
|
||||
duration: 10s # Short duration for testing
|
||||
|
||||
promptTokens: 50 # Target number of tokens for the generated prompt (e.g., 50 or 1000)
|
||||
maxTokens: 150 # Maximum tokens allowed in the response (if applicable)
|
||||
streaming: true # Enable streaming responses
|
||||
|
||||
timeout: 30s # Timeout for each individual request
|
||||
|
||||
outputReport: "benchmark_report.html" # Filename for the generated HTML report
|
||||
|
||||
client: "fasthttp" # Choose the HTTP client: "fasthttp" (default) or "nethttp"
|
||||
|
||||
headers: # Optional: Custom HTTP headers
|
||||
# Authorization: "Bearer YOUR_OTHER_TOKEN" # Example
|
||||
# Custom-Header: "Value"
|
||||
|
||||
payload: # Optional: Custom fields to add/override in the request payload
|
||||
# temperature: 0.7
|
||||
# top_p: 0.9
|
||||
@@ -1,130 +0,0 @@
|
||||
# 配置文档
|
||||
|
||||
本文档详细说明了LLM API基准测试工具的配置选项。
|
||||
|
||||
## 配置文件格式
|
||||
|
||||
配置文件使用YAML格式,默认文件名为`config.yaml`。您可以通过命令行参数`-config`指定其他配置文件路径。
|
||||
|
||||
## 配置选项
|
||||
|
||||
### API配置
|
||||
|
||||
```yaml
|
||||
api:
|
||||
endpoint: "https://api.example.com/v1/completions" # API端点URL
|
||||
api_key: "your_api_key" # API密钥
|
||||
model: "gpt-3.5-turbo" # 模型名称
|
||||
```
|
||||
|
||||
### 提示词模板配置
|
||||
|
||||
```yaml
|
||||
prompt_templates:
|
||||
short: # 短咨询提示词配置
|
||||
target_tokens: 50 # 目标Token数(±5%)
|
||||
templates: # 提示词模板列表
|
||||
- "什么是{concept}?"
|
||||
- "请简要解释{concept}的基本原理。"
|
||||
- "简述{topic}的主要观点。"
|
||||
|
||||
long: # 长文档提示词配置
|
||||
target_tokens: 1000 # 目标Token数(±5%)
|
||||
templates: # 提示词模板列表
|
||||
- "请撰写一篇关于{country}历史的详细介绍,包括主要历史事件和文化发展。"
|
||||
- "请对{topic}进行深入分析,包括其背景、现状、问题和未来发展趋势。"
|
||||
```
|
||||
|
||||
提示词模板中可以使用以下占位符:
|
||||
- `{country}`: 国家名称
|
||||
- `{concept}`: 概念名称
|
||||
- `{topic}`: 主题名称
|
||||
- `{event}`: 事件名称
|
||||
|
||||
系统会自动替换这些占位符,并根据目标Token数动态调整提示词长度。
|
||||
|
||||
### 请求配置
|
||||
|
||||
```yaml
|
||||
requests:
|
||||
- type: "short" # 提示词类型
|
||||
weight: 0.7 # 权重(占比)
|
||||
- type: "long"
|
||||
weight: 0.3
|
||||
```
|
||||
|
||||
权重总和应为1.0,表示各类型提示词的比例。
|
||||
|
||||
### 并发配置
|
||||
|
||||
```yaml
|
||||
concurrency:
|
||||
steps: [50, 200, 500] # 并发步骤
|
||||
duration_per_step: 300 # 每个步骤的持续时间(秒)
|
||||
```
|
||||
|
||||
系统会按照指定的并发步骤逐步增加并发用户数,每个步骤持续指定的时间。
|
||||
|
||||
### 超时配置
|
||||
|
||||
```yaml
|
||||
timeout: 60 # 请求超时时间(秒)
|
||||
```
|
||||
|
||||
### 泊松分布参数
|
||||
|
||||
```yaml
|
||||
poisson_lambda: 1.0 # 请求间隔泊松分布参数
|
||||
```
|
||||
|
||||
该参数控制请求间隔的随机性,模拟真实用户行为。
|
||||
|
||||
### 分词器配置
|
||||
|
||||
```yaml
|
||||
tokenizer:
|
||||
model: "gpt-3.5-turbo" # 用于tiktoken-go的分词器模型
|
||||
```
|
||||
|
||||
## 完整配置示例
|
||||
|
||||
```yaml
|
||||
api:
|
||||
endpoint: "https://api.example.com/v1/completions"
|
||||
api_key: "your_api_key"
|
||||
model: "gpt-3.5-turbo"
|
||||
|
||||
prompt_templates:
|
||||
short:
|
||||
target_tokens: 50
|
||||
templates:
|
||||
- "什么是{concept}?"
|
||||
- "请简要解释{concept}的基本原理。"
|
||||
- "简述{topic}的主要观点。"
|
||||
- "{country}的首都是哪里?"
|
||||
- "如何快速入门{concept}?"
|
||||
long:
|
||||
target_tokens: 1000
|
||||
templates:
|
||||
- "请撰写一篇关于{country}历史的详细介绍,包括主要历史事件和文化发展。"
|
||||
- "请对{topic}进行深入分析,包括其背景、现状、问题和未来发展趋势。"
|
||||
- "请撰写一份关于{event}影响的综合报告,包括社会、经济和政治层面的分析。"
|
||||
- "请详细介绍{concept}的技术原理、应用场景和未来发展方向。"
|
||||
- "请撰写一篇关于{topic}的学术论文,包括研究背景、方法、结果和讨论。"
|
||||
|
||||
requests:
|
||||
- type: "short"
|
||||
weight: 0.7
|
||||
- type: "long"
|
||||
weight: 0.3
|
||||
|
||||
concurrency:
|
||||
steps: [50, 200, 500]
|
||||
duration_per_step: 300 # 秒
|
||||
|
||||
timeout: 60 # 秒
|
||||
poisson_lambda: 1.0 # 请求间隔泊松分布参数
|
||||
|
||||
tokenizer:
|
||||
model: "gpt-3.5-turbo" # 用于tiktoken-go的分词器模型
|
||||
```
|
||||
@@ -0,0 +1,231 @@
|
||||
### 技术选型
|
||||
- **语言**:Go。
|
||||
- **HTTP客户端**:
|
||||
- 抽象接口:`HTTPClient`。
|
||||
- 实现:
|
||||
- `fasthttp`(`github.com/valyala/fasthttp`):高性能,优先选择。
|
||||
- `net/http`(标准库):兼容性强,备选。
|
||||
- **Tokenizer 库**:`github.com/tiktoken-go/tokenizer`。
|
||||
- **并发管理**:goroutine池、`sync.Pool`、`context.Context`。
|
||||
- **数据处理**:`sync.Map`存储响应数据,`gonum/stat`计算百分位数。
|
||||
- **图表生成**:`go-echarts`。
|
||||
|
||||
### 产品需求设计
|
||||
|
||||
#### 功能需求
|
||||
- **API兼容性**:
|
||||
- 支持OpenAI Compatible风格的LLM API(包括`stream=true`的SSE流式响应)。
|
||||
- 兼容推理模型(如DeepSeek-R1)的响应结构,包含`reasoning_content`字段(与`content`同级)。
|
||||
- **HTTP客户端抽象层**:
|
||||
- 定义`HTTPClient`接口,包含以下方法:
|
||||
- `Do(req *Request) (*Response, error)`:发送非流式请求。
|
||||
- `Stream(req *Request, callback func(chunk SSEChunk) error) error`:处理流式响应,逐块回调SSE数据。
|
||||
- 实现:
|
||||
- `fasthttp`:基于`fasthttp.RequestCtx`和`Response.BodyStream`。
|
||||
- `net/http`:基于`http.Client`和`Response.Body`。
|
||||
- 配置项`client: "fasthttp"`或`client: "net-http"`切换客户端。
|
||||
- **提示词生成**:
|
||||
- 使用`tiktoken-go/tokenizer`动态生成短咨询提示词(50 tokens ±5%)和长文档提示词(1000 tokens ±5%)。
|
||||
- **并发请求**:
|
||||
- 支持最大500并发,持续5分钟(300秒)或更长时间。
|
||||
- 使用goroutine池(最大600 goroutine),`sync.Pool`重用请求对象。
|
||||
- 通过`context.Context`支持任务终止。
|
||||
- **场景建模**:
|
||||
- 混合负载:短咨询(50 tokens,70%比例),长文档生成(1000 tokens,30%比例)。
|
||||
- **用户行为模拟**:请求间隔遵循泊松分布(`math/rand`实现)。
|
||||
- **梯度增压**:支持阶梯式加压(50→200→500并发),每阶段持续300秒。
|
||||
- **性能指标收集**:
|
||||
- **请求统计**:总请求数、成功请求数、失败请求数、超时比率(目标:<1%,默认超时30-60秒)。
|
||||
- **响应时间**:平均、最小/最大、P90/P95/P99(目标:<2秒至<10秒,<3秒至<15秒,<5秒至<20秒)。
|
||||
- **TTFT**:
|
||||
- 最小/最大TTFT,P90/P95/P99 TTFT。
|
||||
- 通过SSE流式响应测量,精度达毫秒,兼容`content`和`reasoning_content`。
|
||||
- **QPS**:平均QPS(目标:50-200+),最大QPS。
|
||||
- **Token生成速率**:平均速率(目标:>100 tokens/秒),最大速率,兼容`content`和`reasoning_content`。
|
||||
- **最大有效并发用户数**:记录仍有请求成功的最大并发数(目标:50-500+)。
|
||||
- **报告生成**:
|
||||
- 生成质量报告,包含所有指标统计。
|
||||
- 提供图表(如响应时间分布、TTFT分布、QPS曲线)。
|
||||
|
||||
#### 非功能需求
|
||||
- **高性能**:
|
||||
- 支持500并发,持续5分钟以上,`fasthttp`优化吞吐量,goroutine池降低GC压力。
|
||||
- **统计准确性**:
|
||||
- TTFT和响应时间测量精度达毫秒,使用`time.Now().UnixNano()`。
|
||||
- 使用`sync.Mutex`或`sync/atomic`确保并发统计线程安全。
|
||||
- **可扩展性**:支持新服务提供商、提示词模板、响应字段。
|
||||
- **易用性**:配置文件指定API、提示词、超时、客户端类型(`fasthttp`或`net-http`)。
|
||||
- **稳定性**:长时间运行无崩溃、死锁或内存泄漏。
|
||||
- **可观测性**:
|
||||
- 使用`log`包记录关键事件和错误,通过`--debug`参数,开启调试日志,记录详细信息。
|
||||
- 使用`runtime`包监控CPU、内存、goroutine数。
|
||||
- 使用`sync.Map`存储响应数据,`gonum/stat`计算百分位数。
|
||||
|
||||
#### 技术实现建议
|
||||
- **HTTP客户端抽象层**:
|
||||
- 定义`HTTPClient`接口:
|
||||
```go
|
||||
type HTTPClient interface {
|
||||
Do(req *Request) (*Response, error)
|
||||
Stream(req *Request, callback func(chunk SSEChunk) error) error
|
||||
}
|
||||
type Request struct {
|
||||
Method string
|
||||
URL string
|
||||
Body []byte
|
||||
Headers map[string]string
|
||||
}
|
||||
type Response struct {
|
||||
StatusCode int
|
||||
Body []byte
|
||||
Headers map[string]string
|
||||
}
|
||||
type SSEChunk struct {
|
||||
Data []byte // JSON数据
|
||||
Timestamp int64 // 接收时间戳(UnixNano)
|
||||
IsDone bool // 是否为[data: [DONE]]
|
||||
}
|
||||
```
|
||||
- **fasthttp实现**:
|
||||
- `Do`:使用`fasthttp.Do`,构造OpenAI Compatible JSON请求,解析响应。
|
||||
- `Stream`:使用`fasthttp.Response.BodyStream`,逐行解析SSE(`data: {...}`),调用回调函数传递`SSEChunk`。
|
||||
- 使用`sync.Pool`重用`fasthttp.Request`和`fasthttp.Response`。
|
||||
- **net/http实现**:
|
||||
- `Do`:使用`http.Client.Do`,构造请求,解析响应。
|
||||
- `Stream`:使用`bufio.Scanner`读取`http.Response.Body`,解析SSE,调用回调函数。
|
||||
- **切换逻辑**:
|
||||
- 解析配置项`client`,实例化对应客户端:
|
||||
```go
|
||||
func NewHTTPClient(clientType string) (HTTPClient, error) {
|
||||
switch clientType {
|
||||
case "fasthttp":
|
||||
return &FastHTTPClient{}, nil
|
||||
case "net-http":
|
||||
return &NetHTTPClient{}, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported client: %s", clientType)
|
||||
}
|
||||
}
|
||||
```
|
||||
- **SSE与TTFT处理**:
|
||||
- `Stream`方法逐块解析SSE,提取`choices[0].delta.content`或`choices[0].delta.reasoning_content`。
|
||||
- 记录请求发送时间(`t0 = time.Now().UnixNano()`)。
|
||||
- 检测首个非空Token块,记录时间(`t1`),TTFT = (`t1 - t0`) / 1e6(毫秒)。
|
||||
- 兼容推理模型:优先检查`reasoning_content`,若为空则使用`content`。
|
||||
- **Go性能优化**:
|
||||
- **Goroutine池**:使用`ants`库或自定义池,限制最大600 goroutine。
|
||||
- **对象池**:`sync.Pool`重用请求/响应对象。
|
||||
- **上下文控制**:`context.WithTimeout`设置压测时长(默认300秒)。
|
||||
- **资源监控**:使用`runtime`包记录CPU、内存、goroutine数。
|
||||
- **统计准确性**:
|
||||
- 使用`sync/atomic`累加请求计数。
|
||||
- 使用`sync.Mutex`保护响应时间和TTFT切片。
|
||||
- 使用`gonum/stat`计算P90/P95/P99。
|
||||
- **提示词生成**:`tiktoken-go/tokenizer`动态调整模板至50或1000 tokens。
|
||||
- **报告生成**:`go-echarts`生成图表,模板引擎生成HTML/PDF。
|
||||
|
||||
#### 压测流程
|
||||
1. **配置加载**:读取API端点、客户端类型(`fasthttp`或`net-http`)、并发数、提示词模板等。
|
||||
2. **初始化HTTP客户端**:根据配置项`client`实例化`fasthttp`或`net/http`客户端。
|
||||
3. **提示词生成**:使用`tiktoken-go/tokenizer`生成短咨询(50 tokens)和长文档(1000 tokens)提示词。
|
||||
4. **并发执行**:
|
||||
- 初始化goroutine池(最大600)。
|
||||
- 使用`HTTPClient.Stream`发送请求(`stream=true`),间隔符合泊松分布。
|
||||
- 处理SSE流,记录TTFT。
|
||||
5. **响应处理**:
|
||||
- 解析`content`和`reasoning_content`,计算Token数。
|
||||
- 记录响应时间、TTFT、成功/失败状态。
|
||||
6. **指标计算**:
|
||||
- 计算请求统计、响应时间、TTFT、QPS、Token生成速率、最大并发用户数。
|
||||
7. **报告生成**:输出统计数据和图表。
|
||||
|
||||
#### 配置示例与报告内容
|
||||
以下是更新后的配置和报告示例,包含HTTP客户端抽象层和切换逻辑:
|
||||
|
||||
```yaml
|
||||
# 配置示例
|
||||
api:
|
||||
endpoint: "https://api.example.com/v1/completions"
|
||||
api_key: "your_api_key"
|
||||
model: "deepseek-r1" # 支持推理模型
|
||||
streaming: true # 启用流式响应
|
||||
client: "fasthttp" # HTTP客户端:fasthttp 或 net-http
|
||||
|
||||
prompt_templates:
|
||||
short:
|
||||
target_tokens: 50
|
||||
templates:
|
||||
- "What is the capital of {country}?"
|
||||
- "Please briefly explain the concept of {concept}."
|
||||
- "Summarize the main idea of {topic} in one sentence."
|
||||
long:
|
||||
target_tokens: 1000
|
||||
templates:
|
||||
- "Write a detailed history of {country} covering major events."
|
||||
- "Compose an in-depth analysis of {topic} with examples."
|
||||
- "Generate a comprehensive report on the impact of {event}."
|
||||
|
||||
requests:
|
||||
- type: "short"
|
||||
weight: 0.7
|
||||
- type: "long"
|
||||
weight: 0.3
|
||||
|
||||
concurrency:
|
||||
steps: [50, 200, 500]
|
||||
duration_per_step: 300 # 秒,5分钟
|
||||
max_goroutines: 600 # 最大goroutine数
|
||||
|
||||
timeout: 60 # 秒
|
||||
poisson_lambda: 1.0 # 请求间隔泊松分布参数
|
||||
|
||||
tokenizer:
|
||||
model: "gpt-3.5-turbo" # 用于tiktoken-go的分词器模型
|
||||
|
||||
# 报告内容示例
|
||||
report:
|
||||
overview:
|
||||
api_endpoint: "https://api.example.com/v1/completions"
|
||||
model: "deepseek-r1"
|
||||
concurrency_steps: [50, 200, 500]
|
||||
duration_per_step: "300s"
|
||||
request_mix: "70% short (50 tokens), 30% long (1000 tokens)"
|
||||
tokenizer: "tiktoken-go (gpt-3.5-turbo)"
|
||||
streaming_enabled: true
|
||||
http_client: "fasthttp"
|
||||
metrics:
|
||||
total_requests: 15000
|
||||
successful_requests: 14850
|
||||
failed_requests: 150
|
||||
timeout_ratio: "1.0%"
|
||||
response_time:
|
||||
avg: "1.8s"
|
||||
min: "0.5s"
|
||||
max: "10.2s"
|
||||
p90: "2.5s"
|
||||
p95: "3.0s"
|
||||
p99: "4.5s"
|
||||
ttft:
|
||||
min: "0.1s"
|
||||
max: "2.0s"
|
||||
p90: "0.3s"
|
||||
p95: "0.4s"
|
||||
p99: "0.6s"
|
||||
qps:
|
||||
avg: 180
|
||||
max: 220
|
||||
token_rate:
|
||||
avg: "120 tokens/s"
|
||||
max: "150 tokens/s"
|
||||
max_concurrent_users: 500
|
||||
resource_usage:
|
||||
avg_cpu: "60%"
|
||||
max_memory: "1.1GB"
|
||||
max_goroutines: 600
|
||||
charts:
|
||||
- "response_time_distribution.png"
|
||||
- "ttft_distribution.png"
|
||||
- "qps_over_time.png"
|
||||
- "token_rate_over_time.png"
|
||||
- "concurrency_vs_response_time.png"
|
||||
```
|
||||
@@ -1,38 +1,38 @@
|
||||
module github.com/acmestudio/llm-api-benchmark-tool
|
||||
module llm-api-benchmark-tool
|
||||
|
||||
go 1.21
|
||||
go 1.24.1
|
||||
|
||||
require (
|
||||
github.com/go-echarts/go-echarts/v2 v2.3.3
|
||||
github.com/montanaflynn/stats v0.7.1
|
||||
github.com/pkoukk/tiktoken-go v0.1.6
|
||||
github.com/spf13/viper v1.18.2
|
||||
github.com/valyala/fasthttp v1.52.0
|
||||
github.com/go-echarts/go-echarts/v2 v2.5.3
|
||||
github.com/stretchr/testify v1.10.0
|
||||
github.com/tiktoken-go/tokenizer v0.6.1
|
||||
github.com/valyala/fasthttp v1.61.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/andybalholm/brotli v1.1.0 // indirect
|
||||
github.com/dlclark/regexp2 v1.10.0 // indirect
|
||||
github.com/fsnotify/fsnotify v1.7.0 // indirect
|
||||
github.com/google/uuid v1.4.0 // indirect
|
||||
github.com/hashicorp/hcl v1.0.0 // indirect
|
||||
github.com/klauspost/compress v1.17.6 // indirect
|
||||
github.com/magiconair/properties v1.8.7 // indirect
|
||||
github.com/mitchellh/mapstructure v1.5.0 // indirect
|
||||
github.com/pelletier/go-toml/v2 v2.1.0 // indirect
|
||||
github.com/sagikazarmark/locafero v0.4.0 // indirect
|
||||
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
|
||||
github.com/andybalholm/brotli v1.1.1 // indirect
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/dlclark/regexp2 v1.11.5 // indirect
|
||||
github.com/fsnotify/fsnotify v1.8.0 // indirect
|
||||
github.com/go-viper/mapstructure/v2 v2.2.1 // indirect
|
||||
github.com/klauspost/compress v1.18.0 // indirect
|
||||
github.com/kr/text v0.2.0 // indirect
|
||||
github.com/pelletier/go-toml/v2 v2.2.3 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
github.com/sagikazarmark/locafero v0.7.0 // indirect
|
||||
github.com/sourcegraph/conc v0.3.0 // indirect
|
||||
github.com/spf13/afero v1.11.0 // indirect
|
||||
github.com/spf13/cast v1.6.0 // indirect
|
||||
github.com/spf13/pflag v1.0.5 // indirect
|
||||
github.com/spf13/afero v1.12.0 // indirect
|
||||
github.com/spf13/cast v1.7.1 // indirect
|
||||
github.com/spf13/pflag v1.0.6 // indirect
|
||||
github.com/spf13/viper v1.20.1 // indirect
|
||||
github.com/subosito/gotenv v1.6.0 // indirect
|
||||
github.com/valyala/bytebufferpool v1.0.0 // indirect
|
||||
go.uber.org/atomic v1.9.0 // indirect
|
||||
go.uber.org/multierr v1.9.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
|
||||
golang.org/x/sys v0.17.0 // indirect
|
||||
golang.org/x/text v0.14.0 // indirect
|
||||
gopkg.in/ini.v1 v1.67.0 // indirect
|
||||
golang.org/x/sys v0.32.0 // indirect
|
||||
golang.org/x/text v0.24.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
)
|
||||
|
||||
// Explicitly map the module path to the local directory
|
||||
replace llm-api-benchmark-tool => ./
|
||||
@@ -1,87 +1,63 @@
|
||||
github.com/andybalholm/brotli v1.1.0 h1:eLKJA0d02Lf0mVpIDgYnqXcUn0GqVmEFny3VuID1U3M=
|
||||
github.com/andybalholm/brotli v1.1.0/go.mod h1:sms7XGricyQI9K10gOSf56VKKWS4oLer58Q+mhRPtnY=
|
||||
github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7XdTA=
|
||||
github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
|
||||
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dlclark/regexp2 v1.10.0 h1:+/GIL799phkJqYW+3YbOd8LCcbHzT0Pbo8zl70MHsq0=
|
||||
github.com/dlclark/regexp2 v1.10.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
|
||||
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
|
||||
github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA=
|
||||
github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM=
|
||||
github.com/go-echarts/go-echarts/v2 v2.3.3 h1:uImZAk6qLkC6F9ju6mZ5SPBqTyK8xjZKwSmwnCg4bxg=
|
||||
github.com/go-echarts/go-echarts/v2 v2.3.3/go.mod h1:56YlvzhW/a+du15f3S2qUGNDfKnFOeJSThBIrVFHDtI=
|
||||
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
|
||||
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||
github.com/google/uuid v1.4.0 h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4=
|
||||
github.com/google/uuid v1.4.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
|
||||
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
|
||||
github.com/klauspost/compress v1.17.6 h1:60eq2E/jlfwQXtvZEeBUYADs+BwKBWURIY+Gj2eRGjI=
|
||||
github.com/klauspost/compress v1.17.6/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM=
|
||||
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
|
||||
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
|
||||
github.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ=
|
||||
github.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/8M=
|
||||
github.com/fsnotify/fsnotify v1.8.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
||||
github.com/go-echarts/go-echarts/v2 v2.5.3 h1:5SFAA6bAIWz52VnVGlCM1UZXo8nSdN+H9E8Ysi4m5ec=
|
||||
github.com/go-echarts/go-echarts/v2 v2.5.3/go.mod h1:56YlvzhW/a+du15f3S2qUGNDfKnFOeJSThBIrVFHDtI=
|
||||
github.com/go-viper/mapstructure/v2 v2.2.1 h1:ZAaOCxANMuZx5RCeg0mBdEZk7DZasvvZIxtHqx8aGss=
|
||||
github.com/go-viper/mapstructure/v2 v2.2.1/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
|
||||
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
|
||||
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
|
||||
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
|
||||
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY=
|
||||
github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
|
||||
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
|
||||
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
|
||||
github.com/montanaflynn/stats v0.7.1 h1:etflOAAHORrCC44V+aR6Ftzort912ZU+YLiSTuV8eaE=
|
||||
github.com/montanaflynn/stats v0.7.1/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt6R8Bnaayow=
|
||||
github.com/pelletier/go-toml/v2 v2.1.0 h1:FnwAJ4oYMvbT/34k9zzHuZNrhlz48GB3/s6at6/MHO4=
|
||||
github.com/pelletier/go-toml/v2 v2.1.0/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc=
|
||||
github.com/pkoukk/tiktoken-go v0.1.6 h1:JF0TlJzhTbrI30wCvFuiw6FzP2+/bR+FIxUdgEAcUsw=
|
||||
github.com/pkoukk/tiktoken-go v0.1.6/go.mod h1:9NiV+i9mJKGj1rYOT+njbv+ZwA/zJxYdewGl6qVatpg=
|
||||
github.com/pelletier/go-toml/v2 v2.2.3 h1:YmeHyLY8mFWbdkNWwpr+qIL2bEqT0o95WSdkNHvL12M=
|
||||
github.com/pelletier/go-toml/v2 v2.2.3/go.mod h1:MfCQTFTvCcUyyvvwm1+G6H/jORL20Xlb6rzQu9GuUkc=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
|
||||
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
|
||||
github.com/sagikazarmark/locafero v0.4.0 h1:HApY1R9zGo4DBgr7dqsTH/JJxLTTsOt7u6keLGt6kNQ=
|
||||
github.com/sagikazarmark/locafero v0.4.0/go.mod h1:Pe1W6UlPYUk/+wc/6KFhbORCfqzgYEpgQ3O5fPuL3H4=
|
||||
github.com/sagikazarmark/slog-shim v0.1.0 h1:diDBnUNK9N/354PgrxMywXnAwEr1QZcOr6gto+ugjYE=
|
||||
github.com/sagikazarmark/slog-shim v0.1.0/go.mod h1:SrcSrq8aKtyuqEI1uvTDTK1arOWRIczQRv+GVI1AkeQ=
|
||||
github.com/sagikazarmark/locafero v0.7.0 h1:5MqpDsTGNDhY8sGp0Aowyf0qKsPrhewaLSsFaodPcyo=
|
||||
github.com/sagikazarmark/locafero v0.7.0/go.mod h1:2za3Cg5rMaTMoG/2Ulr9AwtFaIppKXTRYnozin4aB5k=
|
||||
github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo=
|
||||
github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0=
|
||||
github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8=
|
||||
github.com/spf13/afero v1.11.0/go.mod h1:GH9Y3pIexgf1MTIWtNGyogA5MwRIDXGUr+hbWNoBjkY=
|
||||
github.com/spf13/cast v1.6.0 h1:GEiTHELF+vaR5dhz3VqZfFSzZjYbgeKDpBxQVS4GYJ0=
|
||||
github.com/spf13/cast v1.6.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
|
||||
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
|
||||
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/spf13/viper v1.18.2 h1:LUXCnvUvSM6FXAsj6nnfc8Q2tp1dIgUfY9Kc8GsSOiQ=
|
||||
github.com/spf13/viper v1.18.2/go.mod h1:EKmWIqdnk5lOcmR72yw6hS+8OPYcwD0jteitLMVB+yk=
|
||||
github.com/spf13/afero v1.12.0 h1:UcOPyRBYczmFn6yvphxkn9ZEOY65cpwGKb5mL36mrqs=
|
||||
github.com/spf13/afero v1.12.0/go.mod h1:ZTlWwG4/ahT8W7T0WQ5uYmjI9duaLQGy3Q2OAl4sk/4=
|
||||
github.com/spf13/cast v1.7.1 h1:cuNEagBQEHWN1FnbGEjCXL2szYEXqfJPbP2HNUaca9Y=
|
||||
github.com/spf13/cast v1.7.1/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
|
||||
github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
|
||||
github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/spf13/viper v1.20.1 h1:ZMi+z/lvLyPSCoNtFCpqjy0S4kPbirhpTMwl8BkW9X4=
|
||||
github.com/spf13/viper v1.20.1/go.mod h1:P9Mdzt1zoHIG8m2eZQinpiBjo6kCmZSKBClNNqjJvu4=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
|
||||
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
|
||||
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
|
||||
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
|
||||
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
|
||||
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
|
||||
github.com/tiktoken-go/tokenizer v0.6.1 h1:iuhZJUEYY4Fn5vYbDX0Clgqzus8w8rM410YbovUzyLM=
|
||||
github.com/tiktoken-go/tokenizer v0.6.1/go.mod h1:6UCYI/DtOallbmL7sSy30p6YQv60qNyU/4aVigPOx6w=
|
||||
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
|
||||
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
|
||||
github.com/valyala/fasthttp v1.52.0 h1:wqBQpxH71XW0e2g+Og4dzQM8pk34aFYlA1Ga8db7gU0=
|
||||
github.com/valyala/fasthttp v1.52.0/go.mod h1:hf5C4QnVMkNXMspnsUlfM3WitlgYflyhHYoKol/szxQ=
|
||||
github.com/valyala/fasthttp v1.61.0 h1:VV08V0AfoRaFurP1EWKvQQdPTZHiUzaVoulX1aBDgzU=
|
||||
github.com/valyala/fasthttp v1.61.0/go.mod h1:wRIV/4cMwUPWnRcDno9hGnYZGh78QzODFfo1LTUhBog=
|
||||
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
|
||||
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
|
||||
go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE=
|
||||
go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
|
||||
go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI=
|
||||
go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ=
|
||||
golang.org/x/exp v0.0.0-20230905200255-921286631fa9 h1:GoHiUyI/Tp2nVkLI2mCxVkOjsbSXD66ic0XW0js0R9g=
|
||||
golang.org/x/exp v0.0.0-20230905200255-921286631fa9/go.mod h1:S2oDrQGGwySpoQPVqRShND87VCbxmc6bL1Yd2oYrm6k=
|
||||
golang.org/x/sys v0.17.0 h1:25cE3gD+tdBA7lp7QfhuV+rJiE9YXTcS3VG1SqssI/Y=
|
||||
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
|
||||
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
golang.org/x/sys v0.32.0 h1:s77OFDvIQeibCmezSnk/q6iAfkdiQaJi4VzroCFrN20=
|
||||
golang.org/x/sys v0.32.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||
golang.org/x/text v0.24.0 h1:dd5Bzh4yt5KYA8f9CJHCP4FB4D51c2c6JvN37xJJkJ0=
|
||||
golang.org/x/text v0.24.0/go.mod h1:L8rBsPeo2pSS+xqN0d5u2ikmjtmoJbDBT1b7nHvFCdU=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
|
||||
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
|
||||
gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
|
||||
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
Binary file not shown.
@@ -1,200 +0,0 @@
|
||||
package logger
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// 日志级别
|
||||
const (
|
||||
DEBUG = iota
|
||||
INFO
|
||||
WARN
|
||||
ERROR
|
||||
FATAL
|
||||
)
|
||||
|
||||
// 日志级别名称
|
||||
var levelNames = []string{
|
||||
"调试",
|
||||
"信息",
|
||||
"警告",
|
||||
"错误",
|
||||
"致命",
|
||||
}
|
||||
|
||||
// 日志级别颜色
|
||||
var levelColors = []string{
|
||||
"\033[36m", // 青色 - 调试
|
||||
"\033[32m", // 绿色 - 信息
|
||||
"\033[33m", // 黄色 - 警告
|
||||
"\033[31m", // 红色 - 错误
|
||||
"\033[35m", // 紫色 - 致命
|
||||
}
|
||||
|
||||
// 重置颜色
|
||||
const resetColor = "\033[0m"
|
||||
|
||||
// Logger 日志记录器
|
||||
type Logger struct {
|
||||
level int
|
||||
logger *log.Logger
|
||||
useColors bool
|
||||
timeFormat string
|
||||
}
|
||||
|
||||
// 全局日志记录器
|
||||
var defaultLogger *Logger
|
||||
|
||||
// 初始化默认日志记录器
|
||||
func init() {
|
||||
defaultLogger = NewLogger(INFO, true)
|
||||
}
|
||||
|
||||
// NewLogger 创建新的日志记录器
|
||||
func NewLogger(level int, useColors bool) *Logger {
|
||||
return &Logger{
|
||||
level: level,
|
||||
logger: log.New(os.Stdout, "", 0),
|
||||
useColors: useColors,
|
||||
timeFormat: "2006-01-02 15:04:05.000",
|
||||
}
|
||||
}
|
||||
|
||||
// SetLevel 设置日志级别
|
||||
func SetLevel(level int) {
|
||||
defaultLogger.level = level
|
||||
}
|
||||
|
||||
// SetUseColors 设置是否使用颜色
|
||||
func SetUseColors(useColors bool) {
|
||||
defaultLogger.useColors = useColors
|
||||
}
|
||||
|
||||
// formatLog 格式化日志消息
|
||||
func (l *Logger) formatLog(level int, format string, args ...interface{}) string {
|
||||
// 格式化消息
|
||||
var msg string
|
||||
if len(args) > 0 {
|
||||
msg = fmt.Sprintf(format, args...)
|
||||
} else {
|
||||
msg = format
|
||||
}
|
||||
|
||||
// 获取当前时间
|
||||
now := time.Now().Format(l.timeFormat)
|
||||
|
||||
// 获取级别名称
|
||||
levelName := levelNames[level]
|
||||
|
||||
// 构建日志前缀
|
||||
prefix := fmt.Sprintf("[%s] [%s] ", now, levelName)
|
||||
|
||||
// 添加颜色
|
||||
if l.useColors {
|
||||
prefix = levelColors[level] + prefix
|
||||
msg = msg + resetColor
|
||||
}
|
||||
|
||||
// 处理多行消息,确保每行都有正确的前缀
|
||||
lines := strings.Split(msg, "\n")
|
||||
for i := 1; i < len(lines); i++ {
|
||||
if len(lines[i]) > 0 {
|
||||
padding := strings.Repeat(" ", len(now)+3) // +3 for "[] "
|
||||
lines[i] = fmt.Sprintf("%s%s", padding, lines[i])
|
||||
}
|
||||
}
|
||||
|
||||
return prefix + strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// log 记录日志
|
||||
func (l *Logger) log(level int, format string, args ...interface{}) {
|
||||
if level >= l.level {
|
||||
l.logger.Println(l.formatLog(level, format, args...))
|
||||
}
|
||||
}
|
||||
|
||||
// Debug 调试日志
|
||||
func (l *Logger) Debug(format string, args ...interface{}) {
|
||||
l.log(DEBUG, format, args...)
|
||||
}
|
||||
|
||||
// Info 信息日志
|
||||
func (l *Logger) Info(format string, args ...interface{}) {
|
||||
l.log(INFO, format, args...)
|
||||
}
|
||||
|
||||
// Warn 警告日志
|
||||
func (l *Logger) Warn(format string, args ...interface{}) {
|
||||
l.log(WARN, format, args...)
|
||||
}
|
||||
|
||||
// Error 错误日志
|
||||
func (l *Logger) Error(format string, args ...interface{}) {
|
||||
l.log(ERROR, format, args...)
|
||||
}
|
||||
|
||||
// Fatal 致命日志
|
||||
func (l *Logger) Fatal(format string, args ...interface{}) {
|
||||
l.log(FATAL, format, args...)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Debug 调试日志
|
||||
func Debug(format string, args ...interface{}) {
|
||||
defaultLogger.Debug(format, args...)
|
||||
}
|
||||
|
||||
// Info 信息日志
|
||||
func Info(format string, args ...interface{}) {
|
||||
defaultLogger.Info(format, args...)
|
||||
}
|
||||
|
||||
// Warn 警告日志
|
||||
func Warn(format string, args ...interface{}) {
|
||||
defaultLogger.Warn(format, args...)
|
||||
}
|
||||
|
||||
// Error 错误日志
|
||||
func Error(format string, args ...interface{}) {
|
||||
defaultLogger.Error(format, args...)
|
||||
}
|
||||
|
||||
// Fatal 致命日志
|
||||
func Fatal(format string, args ...interface{}) {
|
||||
defaultLogger.Fatal(format, args...)
|
||||
}
|
||||
|
||||
// 进度条相关
|
||||
const (
|
||||
progressBarWidth = 50
|
||||
)
|
||||
|
||||
// PrintProgressBar 打印进度条
|
||||
func PrintProgressBar(current, total int, prefix string) {
|
||||
percentage := float64(current) / float64(total)
|
||||
filled := int(progressBarWidth * percentage)
|
||||
empty := progressBarWidth - filled
|
||||
|
||||
// 构建进度条
|
||||
bar := fmt.Sprintf("\r%s [%s%s] %.1f%% (%d/%d)",
|
||||
prefix,
|
||||
strings.Repeat("█", filled),
|
||||
strings.Repeat("░", empty),
|
||||
percentage*100,
|
||||
current,
|
||||
total,
|
||||
)
|
||||
|
||||
// 打印进度条(不换行)
|
||||
fmt.Print(bar)
|
||||
|
||||
// 如果完成,打印换行
|
||||
if current == total {
|
||||
fmt.Println()
|
||||
}
|
||||
}
|
||||
@@ -1,85 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/config"
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/benchmark"
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/report"
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/logger"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// 解析命令行参数
|
||||
configPath := flag.String("config", "config.yaml", "配置文件路径")
|
||||
outputDir := flag.String("output", "reports", "报告输出目录")
|
||||
debugMode := flag.Bool("debug", false, "启用调试模式")
|
||||
noColor := flag.Bool("no-color", false, "禁用彩色日志输出")
|
||||
flag.Parse()
|
||||
|
||||
// 配置日志级别
|
||||
if *debugMode {
|
||||
logger.SetLevel(logger.DEBUG)
|
||||
} else {
|
||||
logger.SetLevel(logger.INFO)
|
||||
}
|
||||
|
||||
// 配置日志颜色
|
||||
logger.SetUseColors(!*noColor)
|
||||
|
||||
logger.Info("LLM API基准测试工具启动")
|
||||
logger.Info("版本: v0.1.0")
|
||||
logger.Debug("调试模式已启用")
|
||||
|
||||
// 加载配置
|
||||
logger.Info("正在加载配置文件: %s", *configPath)
|
||||
cfg, err := config.LoadConfig(*configPath)
|
||||
if err != nil {
|
||||
logger.Fatal("加载配置失败: %v", err)
|
||||
}
|
||||
logger.Info("配置加载成功")
|
||||
logger.Debug("API端点: %s", cfg.API.Endpoint)
|
||||
logger.Debug("模型: %s", cfg.API.Model)
|
||||
logger.Debug("并发步骤: %v", cfg.Concurrency.Steps)
|
||||
logger.Debug("每步持续时间: %d秒", cfg.Concurrency.DurationPerStep)
|
||||
|
||||
// 创建输出目录
|
||||
if err := os.MkdirAll(*outputDir, 0755); err != nil {
|
||||
logger.Fatal("创建输出目录失败: %v", err)
|
||||
}
|
||||
logger.Debug("输出目录已创建: %s", *outputDir)
|
||||
|
||||
// 初始化基准测试
|
||||
logger.Info("正在初始化基准测试...")
|
||||
bench, err := benchmark.NewBenchmark(cfg)
|
||||
if err != nil {
|
||||
logger.Fatal("初始化基准测试失败: %v", err)
|
||||
}
|
||||
logger.Info("基准测试初始化成功")
|
||||
|
||||
// 运行基准测试
|
||||
logger.Info("开始运行LLM API基准测试...")
|
||||
startTime := time.Now()
|
||||
results, err := bench.Run()
|
||||
if err != nil {
|
||||
logger.Fatal("运行基准测试失败: %v", err)
|
||||
}
|
||||
duration := time.Since(startTime)
|
||||
logger.Info("基准测试完成,总耗时: %v", duration)
|
||||
|
||||
// 生成报告
|
||||
logger.Info("正在生成性能报告...")
|
||||
reportTime := time.Now().Format("2006-01-02_15-04-05")
|
||||
reportPath := fmt.Sprintf("%s/report_%s", *outputDir, reportTime)
|
||||
|
||||
if err := report.GenerateReport(results, cfg, reportPath); err != nil {
|
||||
logger.Fatal("生成报告失败: %v", err)
|
||||
}
|
||||
|
||||
logger.Info("基准测试完成!报告已生成: %s", reportPath)
|
||||
logger.Info("HTML报告: %s/report.html", reportPath)
|
||||
logger.Info("CSV数据: %s/results.csv", reportPath)
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Active Context (As of 2025-04-23 ~08:04 UTC+8)
|
||||
|
||||
## 1. Current Focus
|
||||
Refining the core benchmarking logic and ensuring accurate metric collection, particularly after implementing and debugging streaming support.
|
||||
|
||||
## 2. Recent Changes
|
||||
- **Fixed Stats Recording:** Correctly implemented the recording of `stats.RequestResult` after each request (both streaming and non-streaming) completes within the `runWorker` function in `pkg/concurrency/manager.go`. This involved moving the `RecordResult` call inside the appropriate conditional blocks and using the correct field names (`IsSuccess`, `TimeToFirstToken`).
|
||||
- **Corrected Client Initialization:** Updated `cmd/benchmark/main.go` to correctly initialize the `client.FastHTTPClient`. Removed the flawed logic that attempted to switch client types based on the `streaming` flag, recognizing that `FastHTTPClient` internally handles both streaming (using `net/http`) and non-streaming (using `fasthttp`). Ensured the correct request timeout value is passed during initialization.
|
||||
- **Lint Fixes:** Addressed several lint warnings, including unused imports and tautological conditions.
|
||||
|
||||
## 3. Next Steps & Decisions
|
||||
- The core functionality of running concurrent streaming requests and recording basic results (Success, Latency, TTFT) is working.
|
||||
- The immediate next step is to choose which refinement to tackle next from the following options:
|
||||
1. Add detailed error recording to `RequestResult`.
|
||||
2. Add token statistics (e.g., average tokens/request, total tokens) to the final report.
|
||||
3. Improve/clarify TTFT calculation for non-streaming requests.
|
||||
4. Refactor `runWorker` in `pkg/concurrency/manager.go` for better readability.
|
||||
|
||||
## 4. Active Considerations
|
||||
- Ensuring the `net/http` client used internally for streaming has appropriate timeout settings (currently hardcoded in `Stream` method).
|
||||
- Planning the structure and implementation of the final report generation (console and potentially HTML).
|
||||
@@ -0,0 +1,21 @@
|
||||
# Product Context: LLM API Benchmark Tool
|
||||
|
||||
## 1. Problem Solved
|
||||
Evaluating and comparing the performance characteristics (speed, throughput, responsiveness) of different LLM APIs or different configurations of the same API can be difficult and time-consuming. This tool aims to provide a standardized and automated way to perform these benchmarks.
|
||||
|
||||
## 2. Target Users
|
||||
- Developers integrating LLM APIs who need to choose the best performing option.
|
||||
- Operations teams monitoring LLM API performance and reliability.
|
||||
- Researchers comparing different LLM models or hosting infrastructures.
|
||||
|
||||
## 3. How it Should Work
|
||||
- The user provides a `config.yaml` file specifying the target API, credentials, benchmark parameters (duration, concurrency, etc.), and desired prompt characteristics.
|
||||
- The tool runs the benchmark according to the configuration, sending requests and collecting performance data.
|
||||
- Upon completion (or interruption), the tool calculates and displays summary statistics for the key metrics.
|
||||
- Streaming requests should accurately measure TTFT and token throughput.
|
||||
|
||||
## 4. User Experience Goals
|
||||
- **Easy Configuration:** Simple and clear YAML configuration.
|
||||
- **Clear Output:** Understandable summary report of key metrics.
|
||||
- **Reliable Measurement:** Accurate and consistent performance data collection, especially for streaming TTFT.
|
||||
- **Informative Logging:** Useful logs during the benchmark run for diagnostics.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Progress (As of 2025-04-23 ~08:04 UTC+8)
|
||||
|
||||
## 1. What Works
|
||||
- **Core Benchmark Execution:** Can run concurrent requests (both streaming and non-streaming modes selectable via config) for a specified duration.
|
||||
- **Configuration Loading:** Reads parameters from `config.yaml`.
|
||||
- **Concurrency Control:** Limits active workers based on the `concurrency` setting.
|
||||
- **HTTP Client:** `FastHTTPClient` handles both request types:
|
||||
- Non-streaming (`Do`) uses `fasthttp`.
|
||||
- Streaming (`Stream`) uses `net/http` internally and processes SSE events.
|
||||
- **Tokenizer Integration:** Generates prompts and counts response tokens.
|
||||
- **Basic Stats Collection:**
|
||||
- Records individual `RequestResult` (IsSuccess, Latency, TimeToFirstToken, TotalTokens) correctly for both streaming and non-streaming requests.
|
||||
- Calculates and displays aggregate stats upon completion: Total Requests, Success/Fail counts, Success Rate, Avg QPS, Latency (Avg/Min/Max/Percentiles), TTFT (Avg/Min/Max/Percentiles), Avg Tokens/Second.
|
||||
|
||||
## 2. What's Left / Needs Refinement
|
||||
- **Error Reporting:** Currently only tracks success/failure boolean. Need to capture and potentially report specific error messages (`RequestResult.Error` field).
|
||||
- **Token Statistics:** While individual token counts are recorded, aggregate token stats (e.g., average tokens per successful request, total tokens generated) are not yet calculated or displayed in the final report.
|
||||
- **Non-Streaming TTFT:** The current TTFT for non-streaming requests is approximated as the total latency. This might need refinement or clearer definition.
|
||||
- **Streaming Timeout:** The timeout for the entire streaming request is hardcoded in `client.Stream`. Consider making this configurable.
|
||||
- **Report Generation:** Final report is currently basic console output. Planning to implement HTML report generation (possibly using `go-echarts`).
|
||||
- **Code Refinement:** `pkg/concurrency/manager.go`'s `runWorker` could be refactored for better readability/maintainability.
|
||||
- **Testing:** Need comprehensive integration tests covering various config scenarios (especially different prompt sizes, rate limits, etc.). Existing unit tests cover client and potentially other modules.
|
||||
|
||||
## 3. Current Status
|
||||
- The main bug related to recording streaming results has been fixed.
|
||||
- The client initialization logic is corrected.
|
||||
- The tool is functional for basic benchmarking scenarios, especially streaming.
|
||||
- Ready to proceed with further feature enhancements or refinements.
|
||||
|
||||
## 4. Known Issues
|
||||
- None critical at the moment. Previous lint warnings have been addressed.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Project Brief: LLM API Benchmark Tool
|
||||
|
||||
## 1. Core Goal
|
||||
Develop a command-line tool in Go to benchmark the performance of Large Language Model (LLM) APIs.
|
||||
|
||||
## 2. Key Features
|
||||
- Send configurable concurrent requests (streaming and non-streaming) to a specified LLM API endpoint.
|
||||
- Measure and report key performance metrics:
|
||||
- Request Success/Failure Rate
|
||||
- Queries Per Second (QPS)
|
||||
- Latency (min, max, average, percentiles)
|
||||
- Time To First Token (TTFT) for streaming and non-streaming (min, max, average, percentiles)
|
||||
- Token generation rate (tokens/second, average)
|
||||
- Support configurable parameters via a YAML file (API endpoint, key, model, concurrency, duration, rate limit, prompt size, streaming mode, timeouts, etc.).
|
||||
- Generate a summary report (initially console output, potentially HTML later).
|
||||
|
||||
## 3. Scope
|
||||
- Focus on HTTP-based LLM APIs (e.g., OpenAI compatible).
|
||||
- Initial implementation targets core metrics.
|
||||
- Extensibility for different client implementations or reporting formats is desirable.
|
||||
@@ -0,0 +1,65 @@
|
||||
# System Patterns
|
||||
|
||||
## 1. Overall Architecture
|
||||
- Modular Go application following standard Go project layout (`cmd`, `pkg`, `test`).
|
||||
- Configuration-driven behavior using YAML.
|
||||
- Clear separation of concerns between packages:
|
||||
- `cmd/benchmark`: Main application entry point, CLI flags, orchestrates initialization.
|
||||
- `pkg/config`: Loading and validation of `config.yaml`.
|
||||
- `pkg/client`: Abstraction layer for HTTP requests.
|
||||
- `pkg/tokenizer`: Handles prompt generation and token counting.
|
||||
- `pkg/concurrency`: Manages concurrent workers and request scheduling.
|
||||
- `pkg/stats`: Collects and aggregates performance metrics.
|
||||
- `pkg/report`: (Planned) Generates final benchmark reports.
|
||||
|
||||
## 2. Key Technical Decisions & Patterns
|
||||
- **HTTP Client Abstraction:**
|
||||
- `HTTPClient` interface (`pkg/client/interface.go`) defines `Do` (non-streaming) and `Stream` methods.
|
||||
- `FastHTTPClient` (`pkg/client/fasthttp.go`) is the primary implementation:
|
||||
- Uses `fasthttp` library for high performance in the `Do` method.
|
||||
- Uses Go's standard `net/http` library internally for the `Stream` method for better SSE handling stability.
|
||||
- Uses `sync.Pool` for `fasthttp` request/response object reuse.
|
||||
- **Concurrency Management:**
|
||||
- `pkg/concurrency/manager.go` uses a goroutine pool (implemented via channels/semaphore pattern) to limit concurrent workers based on `config.Concurrency`.
|
||||
- Uses `context.Context` for managing benchmark duration and worker cancellation.
|
||||
- **Statistics Collection:**
|
||||
- `pkg/stats/collector.go` uses a `sync.Map` for thread-safe storage of individual `RequestResult` structs.
|
||||
- Calculates final aggregate statistics (average, min/max, percentiles) upon benchmark completion.
|
||||
- Percentiles calculated using `gonum/stat`.
|
||||
- **Token Handling:**
|
||||
- Uses `github.com/tiktoken-go/tokenizer` for generating prompts of specific token lengths and counting tokens in responses.
|
||||
- **Configuration:**
|
||||
- Uses `viper` (or similar YAML parsing library) within `pkg/config` to load settings.
|
||||
|
||||
## 3. Component Relationships
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[cmd/benchmark/main.go] --> B(pkg/config)
|
||||
A --> C{client.HTTPClient}
|
||||
A --> D(pkg/stats)
|
||||
A --> E(pkg/concurrency)
|
||||
A --> F(pkg/tokenizer)
|
||||
|
||||
C -->|Do uses| G[fasthttp]
|
||||
C -->|Stream uses| H[net/http]
|
||||
|
||||
E --> C
|
||||
E --> F
|
||||
E --> D
|
||||
|
||||
subgraph pkg/client
|
||||
C
|
||||
G
|
||||
H
|
||||
end
|
||||
|
||||
subgraph pkg/stats
|
||||
D
|
||||
I[gonum/stat]
|
||||
end
|
||||
|
||||
subgraph pkg/tokenizer
|
||||
F
|
||||
J[tiktoken-go]
|
||||
end
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
# Technical Context
|
||||
|
||||
## 1. Core Technology
|
||||
- **Language:** Go (Golang)
|
||||
|
||||
## 2. Key Libraries & Dependencies
|
||||
- **HTTP:**
|
||||
- `github.com/valyala/fasthttp`: For high-performance non-streaming requests.
|
||||
- `net/http`: Go standard library, used for streaming (SSE) requests within the client implementation.
|
||||
- **Tokenization:**
|
||||
- `github.com/tiktoken-go/tokenizer`: For OpenAI-compatible token counting and prompt generation helper functions.
|
||||
- **Concurrency:**
|
||||
- Go standard library (`sync`, `context`, goroutines, channels).
|
||||
- **Statistics:**
|
||||
- `gonum.org/v1/gonum/stat`: For percentile calculations.
|
||||
- `sync/atomic`: Used within stats collector for thread-safe counters.
|
||||
- `sync.Map`: For thread-safe storage of request results.
|
||||
- **Configuration:**
|
||||
- `github.com/spf13/viper`: Likely used for loading YAML configuration (verify import).
|
||||
- `gopkg.in/yaml.v3` (or similar): For YAML parsing.
|
||||
- **CLI Flags:**
|
||||
- Go standard library `flag`.
|
||||
|
||||
## 3. Development Environment
|
||||
- **OS:** macOS (as per user information)
|
||||
- **Build/Dependency Management:** Go Modules (`go mod`)
|
||||
|
||||
## 4. Technical Constraints & Considerations
|
||||
- **HTTP Client Choice:** Balancing performance (`fasthttp`) and streaming stability (`net/http`). The current approach uses a hybrid within `FastHTTPClient`.
|
||||
- **Concurrency Limits:** Managing goroutine lifecycle and preventing resource exhaustion.
|
||||
- **Accurate Timing:** Measuring latency and TTFT precisely, especially across concurrent operations.
|
||||
- **Memory Management:** Using `sync.Pool` for `fasthttp` objects; being mindful of memory usage when handling large responses or many results.
|
||||
- **TDD:** Workspace rules mandate Test-Driven Development.
|
||||
@@ -0,0 +1,196 @@
|
||||
package client
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/valyala/fasthttp"
|
||||
)
|
||||
|
||||
// FastHTTPClient implements the HTTPClient interface using fasthttp for non-streaming
|
||||
// and net/http for streaming, with configurable request timeout for non-streaming.
|
||||
type FastHTTPClient struct {
|
||||
reqPool sync.Pool
|
||||
respPool sync.Pool
|
||||
stdClient *http.Client // Standard client for streaming
|
||||
requestTimeout time.Duration
|
||||
}
|
||||
|
||||
// NewFastHTTPClient creates a new FastHTTPClient.
|
||||
func NewFastHTTPClient(requestTimeout time.Duration) *FastHTTPClient {
|
||||
return &FastHTTPClient{
|
||||
reqPool: sync.Pool{
|
||||
New: func() interface{} {
|
||||
return fasthttp.AcquireRequest()
|
||||
},
|
||||
},
|
||||
respPool: sync.Pool{
|
||||
New: func() interface{} {
|
||||
return fasthttp.AcquireResponse()
|
||||
},
|
||||
},
|
||||
stdClient: &http.Client{
|
||||
// Configure standard client (e.g., transport, timeout)
|
||||
Timeout: 30 * time.Second, // Example timeout
|
||||
},
|
||||
requestTimeout: requestTimeout,
|
||||
}
|
||||
}
|
||||
|
||||
// Do sends a non-streaming request using fasthttp.
|
||||
func (c *FastHTTPClient) Do(req *Request) (*Response, error) {
|
||||
// 1. Acquire fasthttp request from pool
|
||||
freq := c.reqPool.Get().(*fasthttp.Request)
|
||||
defer c.reqPool.Put(freq) // Ensure request is put back even on error
|
||||
freq.Reset()
|
||||
|
||||
// 2. Populate fasthttp request from input *Request
|
||||
freq.SetRequestURI(req.URL)
|
||||
freq.Header.SetMethod(req.Method)
|
||||
if req.Body != nil {
|
||||
freq.SetBodyRaw(req.Body) // Use SetBodyRaw to avoid copying
|
||||
}
|
||||
for k, v := range req.Headers {
|
||||
freq.Header.Set(k, v)
|
||||
}
|
||||
|
||||
// 3. Acquire fasthttp response from pool
|
||||
fresp := c.respPool.Get().(*fasthttp.Response)
|
||||
defer c.respPool.Put(fresp) // Ensure response is put back
|
||||
fresp.Reset()
|
||||
|
||||
// 4. Perform the request
|
||||
// Ensure request/response are released even if DoTimeout panics (though unlikely)
|
||||
// Place deferred release calls *before* the operation that might fail
|
||||
defer fasthttp.ReleaseRequest(freq)
|
||||
defer fasthttp.ReleaseResponse(fresp)
|
||||
|
||||
// Perform the request using DoTimeout
|
||||
err := fasthttp.DoTimeout(freq, fresp, c.requestTimeout)
|
||||
if err != nil {
|
||||
// Don't release again here, defer handles it
|
||||
return nil, fmt.Errorf("fasthttp client timeout or connection error for %s %s: %w", req.Method, req.URL, err)
|
||||
}
|
||||
|
||||
// 5. Populate output *Response from fasthttp response
|
||||
resp := &Response{
|
||||
StatusCode: fresp.StatusCode(),
|
||||
Body: append([]byte(nil), fresp.Body()...), // Important: Copy body as fresp will be reused
|
||||
Headers: make(map[string]string),
|
||||
}
|
||||
fresp.Header.VisitAll(func(key, value []byte) {
|
||||
resp.Headers[string(key)] = string(value)
|
||||
})
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// Stream handles streaming SSE requests using the standard net/http client.
|
||||
func (c *FastHTTPClient) Stream(req *Request, callback func(chunk SSEChunk) error) error {
|
||||
// 1. Create standard http request
|
||||
var bodyReader io.Reader
|
||||
if req.Body != nil {
|
||||
bodyReader = bytes.NewReader(req.Body)
|
||||
}
|
||||
// Use context for potential cancellation/timeout
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) // Example: 60s timeout for the whole stream
|
||||
defer cancel()
|
||||
|
||||
stdReq, err := http.NewRequestWithContext(ctx, req.Method, req.URL, bodyReader)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create standard http request for %s: %w", req.URL, err)
|
||||
}
|
||||
// Copy headers
|
||||
for k, v := range req.Headers {
|
||||
stdReq.Header.Set(k, v)
|
||||
}
|
||||
// Ensure Content-Type is set if needed (and not already set)
|
||||
if bodyReader != nil && stdReq.Header.Get("Content-Type") == "" {
|
||||
stdReq.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
// Ensure Accept header is set for SSE
|
||||
if stdReq.Header.Get("Accept") == "" {
|
||||
stdReq.Header.Set("Accept", "text/event-stream")
|
||||
}
|
||||
// Ensure Connection header for streaming
|
||||
if stdReq.Header.Get("Connection") == "" {
|
||||
stdReq.Header.Set("Connection", "keep-alive")
|
||||
}
|
||||
|
||||
// 2. Perform the request
|
||||
resp, err := c.stdClient.Do(stdReq)
|
||||
if err != nil {
|
||||
return fmt.Errorf("standard http client Do failed for %s: %w", req.URL, err)
|
||||
}
|
||||
defer resp.Body.Close() // Ensure body is closed
|
||||
|
||||
// 3. Check status code
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
bodyBytes, readErr := io.ReadAll(resp.Body)
|
||||
if readErr != nil {
|
||||
return fmt.Errorf("http stream error status %d for %s, failed to read body: %w", resp.StatusCode, req.URL, readErr)
|
||||
}
|
||||
return fmt.Errorf("http stream error status %d for %s: %s", resp.StatusCode, req.URL, string(bodyBytes))
|
||||
}
|
||||
|
||||
// 4. Process the stream
|
||||
scanner := bufio.NewScanner(resp.Body)
|
||||
ssePrefix := []byte("data: ")
|
||||
var processingErr error
|
||||
|
||||
for scanner.Scan() {
|
||||
line := scanner.Bytes()
|
||||
if !bytes.HasPrefix(line, ssePrefix) {
|
||||
continue
|
||||
}
|
||||
|
||||
data := bytes.TrimPrefix(line, ssePrefix)
|
||||
ts := time.Now().UnixNano()
|
||||
var isDone bool
|
||||
|
||||
if bytes.Equal(data, []byte("[DONE]")) {
|
||||
isDone = true
|
||||
}
|
||||
|
||||
dataCopy := make([]byte, len(data))
|
||||
copy(dataCopy, data)
|
||||
|
||||
chunk := SSEChunk{
|
||||
Data: dataCopy,
|
||||
Timestamp: ts,
|
||||
IsDone: isDone,
|
||||
}
|
||||
|
||||
if err := callback(chunk); err != nil {
|
||||
if processingErr == nil {
|
||||
processingErr = fmt.Errorf("sse stream user callback error: %w", err)
|
||||
}
|
||||
// Stop processing loop on user callback error
|
||||
break
|
||||
}
|
||||
|
||||
if isDone {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Check for scanner errors
|
||||
if err := scanner.Err(); err != nil {
|
||||
if processingErr == nil {
|
||||
processingErr = fmt.Errorf("error reading sse stream for %s: %w", req.URL, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Return processing error if any occurred during scan or callback
|
||||
if processingErr != nil {
|
||||
return processingErr
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package client
|
||||
|
||||
// HTTPClient defines the interface for sending HTTP requests.
|
||||
// It supports both standard non-streaming and streaming (SSE) requests.
|
||||
type HTTPClient interface {
|
||||
// Do sends a standard non-streaming HTTP request.
|
||||
Do(req *Request) (*Response, error)
|
||||
// Stream sends an HTTP request and processes the response as a stream of Server-Sent Events (SSE).
|
||||
// It calls the callback function for each received SSE chunk.
|
||||
Stream(req *Request, callback func(chunk SSEChunk) error) error
|
||||
}
|
||||
|
||||
// Request represents an HTTP request to be sent.
|
||||
type Request struct {
|
||||
Method string // HTTP method (e.g., "POST", "GET")
|
||||
URL string // Request URL
|
||||
Body []byte // Request body
|
||||
Headers map[string]string // Request headers
|
||||
IsStream bool // Indicates if this request expects a streaming response
|
||||
}
|
||||
|
||||
// Response represents an HTTP response received from the server.
|
||||
type Response struct {
|
||||
StatusCode int // HTTP status code (e.g., 200, 404)
|
||||
Body []byte // Response body (for non-streaming requests)
|
||||
Headers map[string]string // Response headers
|
||||
}
|
||||
|
||||
// SSEChunk represents a single chunk of data received in an SSE stream.
|
||||
type SSEChunk struct {
|
||||
Data []byte // Raw data content of the SSE event (typically JSON)
|
||||
Timestamp int64 // Timestamp when the chunk was received (UnixNano)
|
||||
IsDone bool // Flag indicating if this chunk represents the end of the stream (e.g., "data: [DONE]")
|
||||
}
|
||||
@@ -0,0 +1,308 @@
|
||||
package concurrency
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"llm-api-benchmark-tool/pkg/client"
|
||||
"llm-api-benchmark-tool/pkg/stats"
|
||||
"llm-api-benchmark-tool/pkg/tokenizer"
|
||||
)
|
||||
|
||||
// BenchmarkParams holds the parameters for the benchmark run.
|
||||
type BenchmarkParams struct {
|
||||
HTTPClient client.HTTPClient // The HTTP client to use for requests
|
||||
TargetURL string // The target API endpoint
|
||||
APIKey string // API Key for authorization
|
||||
Model string // Model name to use
|
||||
PromptTokens int // Number of tokens for the generated prompt
|
||||
TotalDuration time.Duration // How long the benchmark should run
|
||||
Concurrency int // Number of concurrent workers/requests allowed simultaneously
|
||||
RequestInterval time.Duration // Minimum delay between starting *any* two requests globally
|
||||
IsStreaming bool // Indicates if the request expects a streaming response
|
||||
Payload map[string]interface{} // Optional: Custom payload fields
|
||||
}
|
||||
|
||||
// Manager coordinates the benchmark execution based on the parameters.
|
||||
type Manager struct {
|
||||
params BenchmarkParams
|
||||
statsCollector *stats.StatsCollector
|
||||
}
|
||||
|
||||
// NewManager creates a new concurrency manager.
|
||||
func NewManager(params BenchmarkParams, statsCollector *stats.StatsCollector) *Manager {
|
||||
return &Manager{
|
||||
params: params,
|
||||
statsCollector: statsCollector,
|
||||
}
|
||||
}
|
||||
|
||||
// RunBenchmark executes the benchmark according to the configured parameters.
|
||||
// It uses the provided context for cancellation.
|
||||
func (m *Manager) RunBenchmark(ctx context.Context) error {
|
||||
log.Printf("RunBenchmark called. RateLimit: %d", m.params.RequestInterval)
|
||||
if m.params.RequestInterval > 0 {
|
||||
return m.runBenchmarkWithTicker(ctx)
|
||||
}
|
||||
return m.runBenchmarkWithoutTicker(ctx)
|
||||
}
|
||||
|
||||
// runBenchmarkWithTicker executes the benchmark with a fixed rate limit.
|
||||
func (m *Manager) runBenchmarkWithTicker(ctx context.Context) error {
|
||||
log.Printf("Starting runBenchmarkWithTicker. Duration: %s, RateLimit: %d, Concurrency: %d",
|
||||
m.params.TotalDuration, m.params.RequestInterval, m.params.Concurrency)
|
||||
|
||||
runCtx, cancel := context.WithTimeout(ctx, m.params.TotalDuration)
|
||||
defer cancel()
|
||||
|
||||
var wg sync.WaitGroup
|
||||
concurrencyLimiter := make(chan struct{}, m.params.Concurrency)
|
||||
|
||||
ticker := time.NewTicker(m.params.RequestInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
log.Printf("Ticker interval: %s", m.params.RequestInterval)
|
||||
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
requestCounter := 0
|
||||
for {
|
||||
select {
|
||||
case <-runCtx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
select {
|
||||
case concurrencyLimiter <- struct{}{}:
|
||||
wg.Add(1)
|
||||
go func(workerID int) {
|
||||
defer wg.Done()
|
||||
defer func() { <-concurrencyLimiter }()
|
||||
|
||||
if runCtx.Err() != nil {
|
||||
return
|
||||
}
|
||||
|
||||
promptText, err := tokenizer.GeneratePrompt(m.params.PromptTokens)
|
||||
if err != nil {
|
||||
log.Printf("Worker: Failed to generate prompt: %v\n", err)
|
||||
return
|
||||
}
|
||||
|
||||
req := m.createRequest(promptText)
|
||||
startTime := time.Now()
|
||||
var result stats.RequestResult
|
||||
|
||||
log.Printf("Worker %d making request (Streaming: %v)...", workerID, m.params.IsStreaming)
|
||||
if m.params.IsStreaming {
|
||||
var firstChunkTime time.Time
|
||||
var totalTokens int
|
||||
err = m.params.HTTPClient.Stream(req, func(chunk client.SSEChunk) error {
|
||||
if firstChunkTime.IsZero() {
|
||||
firstChunkTime = time.Now()
|
||||
}
|
||||
chunkTokens, err := tokenizer.CountTokensInText(string(chunk.Data))
|
||||
if err != nil {
|
||||
log.Printf("Worker %d: Error counting stream chunk tokens: %v", workerID, err)
|
||||
} else {
|
||||
totalTokens += chunkTokens
|
||||
}
|
||||
return nil
|
||||
})
|
||||
latency := time.Since(startTime)
|
||||
result = stats.RequestResult{
|
||||
IsSuccess: err == nil,
|
||||
Latency: latency,
|
||||
TimeToFirstToken: firstChunkTime.Sub(startTime),
|
||||
TotalTokens: totalTokens,
|
||||
}
|
||||
log.Printf("Worker %d finished streaming request. Success: %v", workerID, result.IsSuccess)
|
||||
m.statsCollector.RecordResult(result)
|
||||
} else {
|
||||
resp, err := m.params.HTTPClient.Do(req)
|
||||
latency := time.Since(startTime)
|
||||
var statusCode int
|
||||
var totalTokens int
|
||||
if resp != nil {
|
||||
statusCode = resp.StatusCode
|
||||
bodyBytes := resp.Body
|
||||
bodyString := string(bodyBytes)
|
||||
totalTokens, err = tokenizer.CountTokensInText(bodyString)
|
||||
if err != nil {
|
||||
log.Printf("Worker %d: Error counting non-stream response tokens: %v", workerID, err)
|
||||
}
|
||||
}
|
||||
result = stats.RequestResult{
|
||||
IsSuccess: err == nil && statusCode >= 200 && statusCode < 300,
|
||||
Latency: latency,
|
||||
TimeToFirstToken: latency,
|
||||
TotalTokens: totalTokens,
|
||||
}
|
||||
log.Printf("Worker %d finished non-streaming request. Success: %v, Status: %d", workerID, result.IsSuccess, statusCode)
|
||||
m.statsCollector.RecordResult(result)
|
||||
}
|
||||
|
||||
}(requestCounter)
|
||||
requestCounter++
|
||||
default:
|
||||
log.Printf("Skipping ticker event, concurrency limiter full (Size: %d)", len(concurrencyLimiter))
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
wg.Wait()
|
||||
|
||||
if runCtx.Err() == context.DeadlineExceeded {
|
||||
return nil
|
||||
}
|
||||
log.Printf("Ticker stopped. Waiting for %d workers to finish outstanding requests...", m.params.Concurrency)
|
||||
wg.Wait()
|
||||
return runCtx.Err()
|
||||
}
|
||||
|
||||
// runBenchmarkWithoutTicker handles the benchmark execution when no rate limiting (RequestInterval <= 0) is needed.
|
||||
// It runs requests as fast as possible, respecting the concurrency limit.
|
||||
func (m *Manager) runBenchmarkWithoutTicker(ctx context.Context) error {
|
||||
log.Printf("Starting runBenchmarkWithoutTicker. Duration: %s, Concurrency: %d",
|
||||
m.params.TotalDuration, m.params.Concurrency)
|
||||
|
||||
runCtx, cancel := context.WithTimeout(ctx, m.params.TotalDuration)
|
||||
defer cancel()
|
||||
|
||||
var wg sync.WaitGroup
|
||||
concurrencyLimiter := make(chan struct{}, m.params.Concurrency)
|
||||
|
||||
for i := 0; i < m.params.Concurrency; i++ {
|
||||
concurrencyLimiter <- struct{}{}
|
||||
}
|
||||
|
||||
log.Printf("Starting %d workers", m.params.Concurrency)
|
||||
for i := 0; i < m.params.Concurrency; i++ {
|
||||
wg.Add(1)
|
||||
go func(workerID int) {
|
||||
defer wg.Done()
|
||||
log.Printf("Worker %d started", workerID)
|
||||
for {
|
||||
select {
|
||||
case <-runCtx.Done():
|
||||
log.Printf("Worker %d exiting (context done before acquiring slot).", workerID)
|
||||
return
|
||||
case <-concurrencyLimiter:
|
||||
log.Printf("Worker %d acquired concurrency slot.", workerID)
|
||||
defer func() {
|
||||
log.Printf("Worker %d releasing concurrency slot.", workerID)
|
||||
concurrencyLimiter <- struct{}{}
|
||||
}()
|
||||
|
||||
if runCtx.Err() != nil {
|
||||
log.Printf("Worker %d exiting (context done after acquiring slot).", workerID)
|
||||
return
|
||||
}
|
||||
|
||||
promptText, err := tokenizer.GeneratePrompt(m.params.PromptTokens)
|
||||
if err != nil {
|
||||
log.Printf("Worker %d failed to generate prompt: %v. Skipping request.", workerID, err)
|
||||
continue
|
||||
}
|
||||
|
||||
req := m.createRequest(promptText)
|
||||
startTime := time.Now()
|
||||
var result stats.RequestResult
|
||||
var totalTokens int
|
||||
firstChunkTime := time.Time{}
|
||||
|
||||
log.Printf("Worker %d making request (Streaming: %v)...", workerID, m.params.IsStreaming)
|
||||
if m.params.IsStreaming {
|
||||
err = m.params.HTTPClient.Stream(req, func(chunk client.SSEChunk) error {
|
||||
if firstChunkTime.IsZero() {
|
||||
firstChunkTime = time.Now()
|
||||
}
|
||||
chunkTokens, err := tokenizer.CountTokensInText(string(chunk.Data))
|
||||
if err != nil {
|
||||
log.Printf("Worker %d: Error counting stream chunk tokens: %v", workerID, err)
|
||||
} else {
|
||||
totalTokens += chunkTokens
|
||||
}
|
||||
return nil
|
||||
})
|
||||
latency := time.Since(startTime)
|
||||
result = stats.RequestResult{
|
||||
IsSuccess: err == nil,
|
||||
Latency: latency,
|
||||
TimeToFirstToken: firstChunkTime.Sub(startTime),
|
||||
TotalTokens: totalTokens,
|
||||
}
|
||||
log.Printf("Worker %d finished streaming request. Success: %v", workerID, result.IsSuccess)
|
||||
m.statsCollector.RecordResult(result)
|
||||
} else {
|
||||
resp, doErr := m.params.HTTPClient.Do(req)
|
||||
endTime := time.Now()
|
||||
latency := endTime.Sub(startTime)
|
||||
var statusCode int
|
||||
var totalTokens int
|
||||
if doErr != nil {
|
||||
log.Printf("Worker %d failed to send request (non-stream): %v", workerID, doErr)
|
||||
result = stats.RequestResult{
|
||||
IsSuccess: false,
|
||||
Latency: latency,
|
||||
TimeToFirstToken: latency,
|
||||
TotalTokens: 0,
|
||||
}
|
||||
} else {
|
||||
statusCode = resp.StatusCode
|
||||
bodyBytes := resp.Body
|
||||
bodyString := string(bodyBytes)
|
||||
totalTokens, err = tokenizer.CountTokensInText(bodyString)
|
||||
if err != nil {
|
||||
log.Printf("Worker %d: Error counting non-stream response tokens: %v", workerID, err)
|
||||
}
|
||||
result = stats.RequestResult{
|
||||
IsSuccess: resp != nil && resp.StatusCode >= 200 && resp.StatusCode < 300,
|
||||
Latency: latency,
|
||||
TimeToFirstToken: latency,
|
||||
TotalTokens: totalTokens,
|
||||
}
|
||||
log.Printf("Worker %d finished non-streaming request. Success: %v, Status: %d", workerID, result.IsSuccess, statusCode)
|
||||
}
|
||||
m.statsCollector.RecordResult(result)
|
||||
}
|
||||
}
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
|
||||
<-runCtx.Done()
|
||||
log.Printf("Benchmark duration (%s) finished or context cancelled.", m.params.TotalDuration)
|
||||
|
||||
wg.Wait()
|
||||
log.Println("runBenchmarkWithoutTicker finished.")
|
||||
return nil
|
||||
}
|
||||
|
||||
// createRequest creates a new client.Request object for the benchmark.
|
||||
// It generates a prompt using the tokenizer and constructs the request body.
|
||||
func (m *Manager) createRequest(prompt string) *client.Request {
|
||||
payload := map[string]interface{}{
|
||||
"model": m.params.Model,
|
||||
"messages": []map[string]string{{"role": "user", "content": prompt}},
|
||||
"stream": m.params.IsStreaming,
|
||||
}
|
||||
|
||||
body, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
log.Printf("Error marshaling request body: %v", err)
|
||||
return nil
|
||||
}
|
||||
|
||||
return &client.Request{
|
||||
URL: m.params.TargetURL,
|
||||
Method: "POST",
|
||||
Headers: map[string]string{"Content-Type": "application/json", "Authorization": "Bearer " + m.params.APIKey},
|
||||
Body: body,
|
||||
IsStream: m.params.IsStreaming,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
)
|
||||
|
||||
// Config holds the benchmark configuration.
|
||||
type Config struct {
|
||||
APIEndpoint string `yaml:"apiEndpoint" mapstructure:"apiEndpoint"`
|
||||
APIKey string `yaml:"apiKey" mapstructure:"apiKey"` // Consider loading from env or secure store
|
||||
Model string `yaml:"model" mapstructure:"model"`
|
||||
Requests int `yaml:"requests"`
|
||||
Concurrency int `yaml:"concurrency"`
|
||||
RateLimit int `yaml:"rateLimit"` // Requests per second
|
||||
Duration time.Duration `yaml:"duration"`
|
||||
PromptTokens int `yaml:"promptTokens"`
|
||||
MaxTokens int `yaml:"maxTokens"`
|
||||
Streaming bool `yaml:"streaming" mapstructure:"streaming"`
|
||||
Timeout time.Duration `yaml:"timeout"` // Request timeout
|
||||
RequestTimeout time.Duration `yaml:"requestTimeout" mapstructure:"request_timeout"` // Timeout for individual HTTP requests (defaults to global Timeout if 0)
|
||||
OutputReport string `yaml:"outputReport"` // Path to save the HTML report
|
||||
Headers map[string]string `yaml:"headers"` // Custom HTTP headers
|
||||
Payload map[string]any `yaml:"payload"` // Custom payload fields
|
||||
Client string `yaml:"client"` // "fasthttp" or "nethttp"
|
||||
ReportFile string `yaml:"reportFile" mapstructure:"report_file"` // Output file for the HTML report
|
||||
// Add other configuration parameters as needed
|
||||
}
|
||||
|
||||
// LoadConfig loads configuration from the specified file path.
|
||||
func LoadConfig(filePath string) (*Config, error) {
|
||||
// Check if the file exists
|
||||
if _, err := os.Stat(filePath); os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("config file not found at path: %s", filePath)
|
||||
}
|
||||
|
||||
v := viper.New()
|
||||
v.SetConfigFile(filePath)
|
||||
v.SetConfigType("yaml")
|
||||
|
||||
// Read the configuration file
|
||||
if err := v.ReadInConfig(); err != nil {
|
||||
return nil, fmt.Errorf("failed to read config file '%s': %w", filePath, err)
|
||||
}
|
||||
|
||||
var config Config
|
||||
// Unmarshal the configuration into the Config struct
|
||||
if err := v.Unmarshal(&config); err != nil {
|
||||
return nil, fmt.Errorf("failed to unmarshal config from '%s': %w", filePath, err)
|
||||
}
|
||||
|
||||
// Set default RequestTimeout if not provided
|
||||
if config.RequestTimeout <= 0 {
|
||||
config.RequestTimeout = config.Timeout
|
||||
}
|
||||
|
||||
// TODO: Add validation for required fields if necessary
|
||||
return &config, nil
|
||||
}
|
||||
@@ -0,0 +1,346 @@
|
||||
package report
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"html/template"
|
||||
"log"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/go-echarts/go-echarts/v2/charts"
|
||||
"github.com/go-echarts/go-echarts/v2/opts"
|
||||
"github.com/go-echarts/go-echarts/v2/render"
|
||||
"llm-api-benchmark-tool/pkg/stats"
|
||||
)
|
||||
|
||||
// ReportData holds the data passed to the HTML template.
|
||||
type ReportData struct {
|
||||
Stats stats.FinalStats
|
||||
LatencyChart template.HTML // Already safe HTML
|
||||
TTFTChart template.HTML // Already safe HTML
|
||||
// Add other charts here (QPSChart, etc.)
|
||||
}
|
||||
|
||||
// Basic HTML template string - No |safeHTML filter needed
|
||||
const htmlTemplate = `<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>LLM API Benchmark Report</title>
|
||||
<!-- Include ECharts JS directly for simplicity in this example -->
|
||||
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/echarts@5.4.1/dist/echarts.min.js"></script>
|
||||
<style> body { font-family: sans-serif; margin: 20px; } table { border-collapse: collapse; margin-bottom: 20px; } th, td { border: 1px solid #ddd; padding: 8px; text-align: left; } th { background-color: #f2f2f2; } .chart-container { width: 80%; min-height: 400px; margin: auto; margin-bottom: 30px; border: 1px solid #eee; padding: 15px; } </style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>LLM API Benchmark Report</h1>
|
||||
|
||||
<h2>Summary</h2>
|
||||
<table>
|
||||
<tr><th>Metric</th><th>Value</th></tr>
|
||||
<tr><td>Total Duration:</td><td>{{ .Stats.TotalDuration }}</td></tr>
|
||||
<tr><td>Total Requests:</td><td>{{ .Stats.TotalRequests }}</td></tr>
|
||||
<tr><td>Successful Requests:</td><td>{{ .Stats.SuccessfulRequests }}</td></tr>
|
||||
<tr><td>Failed Requests:</td><td>{{ .Stats.FailedRequests }}</td></tr>
|
||||
<tr><td>Avg QPS:</td><td>{{ printf "%.2f" .Stats.AvgQPS }}</td></tr>
|
||||
<tr><td>Avg Latency:</td><td>{{ .Stats.AvgLatency }}</td></tr>
|
||||
<tr><td>Min Latency:</td><td>{{ .Stats.MinLatency }}</td></tr>
|
||||
<tr><td>Max Latency:</td><td>{{ .Stats.MaxLatency }}</td></tr>
|
||||
<tr><td>P90 Latency:</td><td>{{ .Stats.P90Latency }}</td></tr>
|
||||
<tr><td>P95 Latency:</td><td>{{ .Stats.P95Latency }}</td></tr>
|
||||
<tr><td>P99 Latency:</td><td>{{ .Stats.P99Latency }}</td></tr>
|
||||
<tr><td>Avg TTFT:</td><td>{{ .Stats.AvgTimeToFirstToken }}</td></tr>
|
||||
<tr><td>Min TTFT:</td><td>{{ .Stats.MinTimeToFirstToken }}</td></tr>
|
||||
<tr><td>Max TTFT:</td><td>{{ .Stats.MaxTimeToFirstToken }}</td></tr>
|
||||
<tr><td>P90 TTFT:</td><td>{{ .Stats.P90TimeToFirstToken }}</td></tr>
|
||||
<tr><td>P95 TTFT:</td><td>{{ .Stats.P95TimeToFirstToken }}</td></tr>
|
||||
<tr><td>P99 TTFT:</td><td>{{ .Stats.P99TimeToFirstToken }}</td></tr>
|
||||
<tr><td>Avg Tokens/Second:</td><td>{{ printf "%.2f" .Stats.AvgTokensPerSecond }}</td></tr>
|
||||
<!-- Add more stats as needed -->
|
||||
</table>
|
||||
|
||||
<h2>Charts</h2>
|
||||
<!-- Latency Histogram Chart -->
|
||||
<div class="chart-container">
|
||||
{{ .LatencyChart }}
|
||||
</div>
|
||||
|
||||
<!-- TTFT Histogram Chart -->
|
||||
<div class="chart-container">
|
||||
{{ .TTFTChart }}
|
||||
</div>
|
||||
|
||||
<!-- Add more chart containers here -->
|
||||
|
||||
</body>
|
||||
</html>`
|
||||
|
||||
// GenerateHTMLReport generates an HTML report from the benchmark statistics.
|
||||
func GenerateHTMLReport(finalStats stats.FinalStats, outputFilePath string) error {
|
||||
// --- Create Charts ---
|
||||
latencyChart := createLatencyHistogram(finalStats)
|
||||
ttftChart := createTTFTHistogram(finalStats)
|
||||
|
||||
// --- Prepare Template Data ---
|
||||
// Render charts to temporary buffers to get HTML content
|
||||
latencyChartHTML, err := renderChartToHTMLFragment(latencyChart)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to render latency chart: %w", err)
|
||||
}
|
||||
ttftChartHTML, err := renderChartToHTMLFragment(ttftChart)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to render TTFT chart: %w", err)
|
||||
}
|
||||
|
||||
summaryStr := fmt.Sprintf(`
|
||||
Total Requests: %d
|
||||
Successful Requests: %d
|
||||
Failed Requests: %d
|
||||
Total Duration: %s
|
||||
Average Latency: %s
|
||||
Min Latency: %s
|
||||
Max Latency: %s
|
||||
P90 Latency: %s
|
||||
P95 Latency: %s
|
||||
P99 Latency: %s
|
||||
Average TTFT: %s
|
||||
Min TTFT: %s
|
||||
Max TTFT: %s
|
||||
P90 TTFT: %s
|
||||
P95 TTFT: %s
|
||||
P99 TTFT: %s
|
||||
Average QPS: %.2f
|
||||
Average Tokens/Second: %.2f
|
||||
`,
|
||||
finalStats.TotalRequests,
|
||||
finalStats.SuccessfulRequests,
|
||||
finalStats.FailedRequests,
|
||||
finalStats.TotalDuration,
|
||||
finalStats.AvgLatency,
|
||||
finalStats.MinLatency,
|
||||
finalStats.MaxLatency,
|
||||
finalStats.P90Latency,
|
||||
finalStats.P95Latency,
|
||||
finalStats.P99Latency,
|
||||
finalStats.AvgTimeToFirstToken,
|
||||
finalStats.MinTimeToFirstToken,
|
||||
finalStats.MaxTimeToFirstToken,
|
||||
finalStats.P90TimeToFirstToken,
|
||||
finalStats.P95TimeToFirstToken,
|
||||
finalStats.P99TimeToFirstToken,
|
||||
finalStats.AvgQPS,
|
||||
finalStats.AvgTokensPerSecond,
|
||||
)
|
||||
|
||||
templateData := map[string]interface{}{
|
||||
"Stats": finalStats,
|
||||
"Summary": summaryStr,
|
||||
"LatencyChart": latencyChartHTML,
|
||||
"TTFTChart": ttftChartHTML,
|
||||
}
|
||||
|
||||
// --- Parse and Execute Template ---
|
||||
// No Funcs map needed as renderChartToHTMLFragment returns template.HTML
|
||||
tmpl, err := template.New("report").Parse(htmlTemplate)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to parse HTML template: %w", err)
|
||||
}
|
||||
|
||||
f, err := os.Create(outputFilePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create report file '%s': %w", outputFilePath, err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
// log.Printf("DEBUG: Template Data Before Execution: %+v", templateData)
|
||||
|
||||
// Execute the template
|
||||
err = tmpl.Execute(f, templateData)
|
||||
if err != nil {
|
||||
// Include the underlying template execution error for better debugging
|
||||
return fmt.Errorf("failed to execute HTML template: %w", err)
|
||||
}
|
||||
|
||||
fmt.Printf("HTML report generated successfully: %s\n", outputFilePath)
|
||||
return nil
|
||||
}
|
||||
|
||||
// getChartBaseConfig safely extracts the BaseConfiguration from a render.Renderer.
|
||||
func getChartBaseConfig(chart render.Renderer) (*charts.BaseConfiguration, string, error) {
|
||||
var baseConfig *charts.BaseConfiguration
|
||||
var chartID string
|
||||
|
||||
switch c := chart.(type) {
|
||||
case *charts.Bar:
|
||||
baseConfig = &c.RectChart.RectConfiguration.BaseConfiguration
|
||||
chartID = c.Initialization.ChartID // Access ChartID from embedded Initialization
|
||||
case *charts.Line: // Add cases for other chart types if needed
|
||||
baseConfig = &c.RectChart.RectConfiguration.BaseConfiguration
|
||||
chartID = c.Initialization.ChartID
|
||||
// Add other chart types here (e.g., Pie, Scatter, etc.)
|
||||
default:
|
||||
return nil, "", fmt.Errorf("unsupported chart type: %T", chart)
|
||||
}
|
||||
|
||||
if chartID == "" {
|
||||
// Generate a fallback ID if not set in Initialization
|
||||
chartID = fmt.Sprintf("chart%d", time.Now().UnixNano())
|
||||
log.Printf("Warning: ChartID not set for %T, using generated ID: %s", chart, chartID)
|
||||
}
|
||||
|
||||
return baseConfig, chartID, nil
|
||||
}
|
||||
|
||||
// renderChartToHTMLFragment renders a chart configuration to an HTML fragment.
|
||||
func renderChartToHTMLFragment(chart render.Renderer) (template.HTML, error) {
|
||||
baseConfig, chartID, err := getChartBaseConfig(chart)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// Serialize the BaseConfiguration which holds the options.
|
||||
optJSONBytes, err := json.Marshal(baseConfig)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to marshal chart options for %s (%T): %w", chartID, chart, err)
|
||||
}
|
||||
optJSON := string(optJSONBytes)
|
||||
|
||||
// Construct the HTML fragment (div and script)
|
||||
script := fmt.Sprintf(`
|
||||
<div id="%s" style="width:100%%; height:400px;"></div>
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
// Check if element exists before initializing
|
||||
if (document.getElementById('%s')) {
|
||||
let chart_%s = echarts.init(document.getElementById('%s'), null, {renderer: '%s'});
|
||||
let option_%s = %s;
|
||||
chart_%s.setOption(option_%s);
|
||||
} else {
|
||||
console.error("Chart element not found: %s");
|
||||
}
|
||||
</script>`,
|
||||
chartID, // div id
|
||||
chartID, // Check element id
|
||||
chartID, chartID, // script variable and element id
|
||||
baseConfig.Initialization.Renderer, // Use renderer from initialization options
|
||||
chartID, optJSON, // options variable and json content
|
||||
chartID, chartID, // setOption call
|
||||
chartID, // Error log element id
|
||||
)
|
||||
|
||||
return template.HTML(script), nil
|
||||
}
|
||||
|
||||
// calculateHistogram calculates histogram data (categories and counts) from a slice of durations.
|
||||
func calculateHistogram(data []time.Duration, numBins int) ([]string, []opts.BarData) {
|
||||
if len(data) == 0 || numBins <= 0 {
|
||||
return []string{"No Data"}, []opts.BarData{{Value: 0}}
|
||||
}
|
||||
|
||||
// Find min and max values
|
||||
minVal, maxVal := data[0], data[0]
|
||||
for _, d := range data {
|
||||
if d < minVal {
|
||||
minVal = d
|
||||
}
|
||||
if d > maxVal {
|
||||
maxVal = d
|
||||
}
|
||||
}
|
||||
|
||||
// Handle case where all data points are the same
|
||||
if minVal == maxVal {
|
||||
return []string{fmt.Sprintf("%.2f ms", float64(minVal)/float64(time.Millisecond))},
|
||||
[]opts.BarData{{Value: len(data)}}
|
||||
}
|
||||
|
||||
binWidth := (maxVal - minVal) / time.Duration(numBins)
|
||||
if binWidth == 0 {
|
||||
binWidth = 1 // Avoid division by zero if range is very small relative to numBins
|
||||
}
|
||||
|
||||
bins := make([]int, numBins) // Stores counts for each bin
|
||||
categories := make([]string, numBins)
|
||||
|
||||
for i := 0; i < numBins; i++ {
|
||||
binStart := minVal + time.Duration(i)*binWidth
|
||||
binEnd := binStart + binWidth
|
||||
// Ensure the last bin's end includes the max value exactly
|
||||
if i == numBins-1 {
|
||||
binEnd = maxVal
|
||||
}
|
||||
|
||||
// Format category label (e.g., "50.00-100.00 ms")
|
||||
// Use slightly different formatting for clarity: [start, end) or [start, end] for the last bin
|
||||
labelFormat := "[%.2f, %.2f) ms"
|
||||
if i == numBins-1 {
|
||||
labelFormat = "[%.2f, %.2f] ms"
|
||||
}
|
||||
categories[i] = fmt.Sprintf(labelFormat,
|
||||
float64(binStart)/float64(time.Millisecond),
|
||||
float64(binEnd)/float64(time.Millisecond),
|
||||
)
|
||||
}
|
||||
|
||||
for _, d := range data {
|
||||
// Determine the correct bin index
|
||||
var binIndex int
|
||||
if d == maxVal {
|
||||
// Special case: max value goes into the last bin
|
||||
binIndex = numBins - 1
|
||||
} else {
|
||||
// Calculate bin index based on width
|
||||
binIndex = int((d - minVal) / binWidth)
|
||||
// Clamp index to valid range (should not happen with maxVal check, but defensive)
|
||||
if binIndex < 0 {
|
||||
binIndex = 0
|
||||
} else if binIndex >= numBins {
|
||||
binIndex = numBins - 1
|
||||
}
|
||||
}
|
||||
bins[binIndex]++
|
||||
}
|
||||
|
||||
barData := make([]opts.BarData, numBins)
|
||||
for i := 0; i < numBins; i++ {
|
||||
barData[i] = opts.BarData{Value: bins[i]}
|
||||
}
|
||||
|
||||
return categories, barData
|
||||
}
|
||||
|
||||
// createLatencyHistogram creates a Bar chart for latency distribution.
|
||||
func createLatencyHistogram(finalStats stats.FinalStats) *charts.Bar {
|
||||
bar := charts.NewBar()
|
||||
bar.SetGlobalOptions(
|
||||
charts.WithTitleOpts(opts.Title{Title: "Latency Distribution"}),
|
||||
// Ensure ID is set correctly via InitializationOpts
|
||||
charts.WithInitializationOpts(opts.Initialization{ChartID: "latencyHistogram"}),
|
||||
// Add other global options if needed, e.g., Toolbox
|
||||
charts.WithToolboxOpts(opts.Toolbox{Show: opts.Bool(true)}),
|
||||
)
|
||||
|
||||
// Calculate and set histogram data
|
||||
categories, data := calculateHistogram(finalStats.LatencyData, 10) // Use 10 bins for example
|
||||
bar.SetXAxis(categories).AddSeries("Latency", data)
|
||||
|
||||
return bar
|
||||
}
|
||||
|
||||
// createTTFTHistogram creates a Bar chart for Time To First Token distribution.
|
||||
func createTTFTHistogram(finalStats stats.FinalStats) *charts.Bar {
|
||||
bar := charts.NewBar()
|
||||
bar.SetGlobalOptions(
|
||||
charts.WithTitleOpts(opts.Title{Title: "TTFT Distribution"}),
|
||||
// Ensure ID is set correctly via InitializationOpts
|
||||
charts.WithInitializationOpts(opts.Initialization{ChartID: "ttftHistogram"}),
|
||||
// Add other global options if needed
|
||||
charts.WithToolboxOpts(opts.Toolbox{Show: opts.Bool(true)}),
|
||||
)
|
||||
|
||||
// Calculate and set histogram data (assuming TTFTData exists in FinalStats)
|
||||
categories, data := calculateHistogram(finalStats.TTFTData, 10) // Use actual TTFT data
|
||||
bar.SetXAxis(categories).AddSeries("TTFT", data)
|
||||
|
||||
return bar
|
||||
}
|
||||
@@ -0,0 +1,175 @@
|
||||
package report
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llm-api-benchmark-tool/pkg/stats"
|
||||
"github.com/go-echarts/go-echarts/v2/opts"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestCalculateHistogram(t *testing.T) {
|
||||
// Helper function to create durations from milliseconds
|
||||
dms := func(ms int) time.Duration {
|
||||
return time.Duration(ms) * time.Millisecond
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
data []time.Duration
|
||||
numBins int
|
||||
wantCats []string
|
||||
wantDataVals []int // Just check the values
|
||||
isErrorCase bool // Indicates if we expect the "No Data" case
|
||||
}{
|
||||
{
|
||||
name: "Empty data",
|
||||
data: []time.Duration{},
|
||||
numBins: 5,
|
||||
wantCats: []string{"No Data"},
|
||||
wantDataVals: []int{0},
|
||||
isErrorCase: true,
|
||||
},
|
||||
{
|
||||
name: "Zero bins",
|
||||
data: []time.Duration{dms(100), dms(200)},
|
||||
numBins: 0,
|
||||
wantCats: []string{"No Data"},
|
||||
wantDataVals: []int{0},
|
||||
isErrorCase: true,
|
||||
},
|
||||
{
|
||||
name: "Single data point",
|
||||
data: []time.Duration{dms(150)},
|
||||
numBins: 5,
|
||||
wantCats: []string{"150.00 ms"}, // Special case format
|
||||
wantDataVals: []int{1},
|
||||
},
|
||||
{
|
||||
name: "All data points same",
|
||||
data: []time.Duration{dms(100), dms(100), dms(100)},
|
||||
numBins: 3,
|
||||
wantCats: []string{"100.00 ms"},
|
||||
wantDataVals: []int{3},
|
||||
},
|
||||
{
|
||||
name: "Simple case, 2 bins",
|
||||
data: []time.Duration{dms(50), dms(60), dms(110), dms(120), dms(130)},
|
||||
// Range: 50 to 130 (80ms)
|
||||
// Bin width: 80 / 2 = 40
|
||||
// Bins: [50, 90), [90, 130]
|
||||
numBins: 2,
|
||||
wantCats: []string{"[50.00, 90.00) ms", "[90.00, 130.00] ms"},
|
||||
wantDataVals: []int{2, 3},
|
||||
},
|
||||
{
|
||||
name: "Even distribution, 5 bins",
|
||||
data: []time.Duration{dms(10), dms(25), dms(35), dms(45), dms(55), dms(65), dms(75), dms(85), dms(95), dms(100)},
|
||||
// Range: 10 to 100 (90ms) -> max should be 100? Let's make max 110 for simpler bins
|
||||
// Let's adjust data slightly for easier math: min 10, max 110
|
||||
// data: {10, 25, 35, 45, 55, 65, 75, 85, 95, 110}
|
||||
// Range: 10 to 110 (100ms)
|
||||
// Bin width: 100 / 5 = 20
|
||||
// Bins: [10,30), [30,50), [50,70), [70,90), [90,110]
|
||||
numBins: 5,
|
||||
wantCats: []string{"[10.00, 28.00) ms", "[28.00, 46.00) ms", "[46.00, 64.00) ms", "[64.00, 82.00) ms", "[82.00, 100.00] ms"},
|
||||
wantDataVals: []int{2, 2, 1, 2, 3},
|
||||
},
|
||||
{
|
||||
name: "Data clustered at ends",
|
||||
data: []time.Duration{dms(10), dms(15), dms(20), dms(90), dms(95), dms(100)},
|
||||
// Range: 10 to 100 (90). Width (3 bins): 30.
|
||||
// Bins: [10, 40), [40, 70), [70, 100]
|
||||
numBins: 3,
|
||||
wantCats: []string{"[10.00, 40.00) ms", "[40.00, 70.00) ms", "[70.00, 100.00] ms"},
|
||||
wantDataVals: []int{3, 0, 3},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
gcats, gdata := calculateHistogram(tt.data, tt.numBins)
|
||||
|
||||
require.Equal(t, len(tt.wantCats), len(gcats), "Number of categories mismatch")
|
||||
assert.Equal(t, tt.wantCats, gcats, "Category labels mismatch")
|
||||
|
||||
require.Equal(t, len(tt.wantDataVals), len(gdata), "Number of data points mismatch")
|
||||
|
||||
// Extract values from opts.BarData
|
||||
gotDataVals := make([]int, len(gdata))
|
||||
for i, item := range gdata {
|
||||
// Check if item.Value is an int or can be converted
|
||||
if val, ok := item.Value.(int); ok {
|
||||
gotDataVals[i] = val
|
||||
} else if valF, ok := item.Value.(float64); ok {
|
||||
gotDataVals[i] = int(valF) // Or handle float differently if needed
|
||||
} else {
|
||||
t.Fatalf("Unexpected data type in BarData: %T", item.Value)
|
||||
}
|
||||
}
|
||||
assert.Equal(t, tt.wantDataVals, gotDataVals, "Data values mismatch")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Helper to create durations
|
||||
func dms(ms int) time.Duration {
|
||||
return time.Duration(ms) * time.Millisecond
|
||||
}
|
||||
|
||||
// TestCreateLatencyHistogram checks if the latency histogram chart is created correctly.
|
||||
func TestCreateLatencyHistogram(t *testing.T) {
|
||||
mockStats := stats.FinalStats{
|
||||
LatencyData: []time.Duration{dms(50), dms(100), dms(110), dms(150), dms(200), dms(210), dms(220)},
|
||||
}
|
||||
|
||||
chart := createLatencyHistogram(mockStats)
|
||||
|
||||
require.NotNil(t, chart, "Chart should not be nil")
|
||||
require.NotNil(t, chart.BaseConfiguration.Title, "Chart title configuration should not be nil")
|
||||
|
||||
// Check title
|
||||
assert.Equal(t, "Latency Distribution", chart.BaseConfiguration.Title.Title, "Chart title mismatch")
|
||||
|
||||
// Check series data (basic check - assuming one series with correct name)
|
||||
require.Len(t, chart.MultiSeries, 1, "Expected one series")
|
||||
assert.Equal(t, "Latency", chart.MultiSeries[0].Name, "Series name mismatch")
|
||||
|
||||
// Optionally, check the number of data points generated by calculateHistogram via the chart
|
||||
// This depends on the numBins used internally by createLatencyHistogram (currently 10)
|
||||
// For the mock data {50, 100, 110, 150, 200, 210, 220}, range=170, 10 bins -> width=17
|
||||
// Bins: [50, 67), [67, 84), [84, 101), [101, 118), [118, 135), [135, 152), [152, 169), [169, 186), [186, 203), [203, 220]
|
||||
// Counts: 1, 0, 1, 1, 0, 1, 0, 0, 1, 2 -> Total 7 points, 10 bars
|
||||
seriesData, ok := chart.MultiSeries[0].Data.([]opts.BarData)
|
||||
require.True(t, ok, "Series data should be of type []opts.BarData")
|
||||
assert.Len(t, seriesData, 10, "Expected 10 data points (bins) in the series")
|
||||
}
|
||||
|
||||
// TestCreateTTFTHistogram checks if the TTFT histogram chart is created correctly.
|
||||
func TestCreateTTFTHistogram(t *testing.T) {
|
||||
mockStats := stats.FinalStats{
|
||||
TTFTData: []time.Duration{dms(20), dms(30), dms(35), dms(40), dms(60)},
|
||||
}
|
||||
|
||||
chart := createTTFTHistogram(mockStats)
|
||||
|
||||
require.NotNil(t, chart, "Chart should not be nil")
|
||||
require.NotNil(t, chart.BaseConfiguration.Title, "Chart title configuration should not be nil")
|
||||
|
||||
// Check title
|
||||
assert.Equal(t, "TTFT Distribution", chart.BaseConfiguration.Title.Title, "Chart title mismatch")
|
||||
|
||||
// Check series data
|
||||
require.Len(t, chart.MultiSeries, 1, "Expected one series")
|
||||
assert.Equal(t, "TTFT", chart.MultiSeries[0].Name, "Series name mismatch")
|
||||
|
||||
// Optionally, check the number of data points
|
||||
// Data: {20, 30, 35, 40, 60}, Range=40, 10 bins -> width=4
|
||||
// Bins: [20,24), [24,28), [28,32), [32,36), [36,40), [40,44), [44,48), [48,52), [52,56), [56,60]
|
||||
// Counts: 1, 0, 1, 1, 0, 1, 0, 0, 0, 1 -> Total 5 points, 10 bars
|
||||
seriesData, ok := chart.MultiSeries[0].Data.([]opts.BarData)
|
||||
require.True(t, ok, "Series data should be of type []opts.BarData")
|
||||
assert.Len(t, seriesData, 10, "Expected 10 data points (bins) in the series")
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
package stats
|
||||
|
||||
import (
|
||||
"log"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
// "gonum.org/v1/gonum/stat" // Keep commented until needed for percentiles
|
||||
)
|
||||
|
||||
// RequestResult holds the metrics for a single request.
|
||||
type RequestResult struct {
|
||||
IsSuccess bool
|
||||
Latency time.Duration
|
||||
TimeToFirstToken time.Duration // Relevant for streaming and non-streaming (time to first byte of response body)
|
||||
TotalTokens int // Tokens in the response content
|
||||
StreamLatency []time.Duration // Latency for each chunk in a stream (if streaming)
|
||||
// TODO: Add Error details? Model name?
|
||||
}
|
||||
|
||||
// FinalStats holds the aggregated results of the benchmark run.
|
||||
type FinalStats struct {
|
||||
TotalRequests int64
|
||||
SuccessfulRequests int64
|
||||
FailedRequests int64
|
||||
TotalDuration time.Duration
|
||||
AvgLatency time.Duration // Average latency of successful requests
|
||||
MinLatency time.Duration
|
||||
MaxLatency time.Duration
|
||||
P90Latency time.Duration
|
||||
P95Latency time.Duration
|
||||
P99Latency time.Duration
|
||||
AvgTimeToFirstToken time.Duration // Average TTFT of successful requests
|
||||
MinTimeToFirstToken time.Duration
|
||||
MaxTimeToFirstToken time.Duration
|
||||
P90TimeToFirstToken time.Duration
|
||||
P95TimeToFirstToken time.Duration
|
||||
P99TimeToFirstToken time.Duration
|
||||
AvgQPS float64 // Average Queries Per Second (TotalRequests / TotalDuration)
|
||||
AvgTokensPerSecond float64 // Average tokens per second (TotalTokens / Total Successful Latency)
|
||||
// Data for histograms
|
||||
LatencyData []time.Duration `json:"-"` // Raw latency data points (exclude from default JSON)
|
||||
TTFTData []time.Duration `json:"-"` // Raw TTFT data points (exclude from default JSON)
|
||||
// TODO: Add MaxQPS? Token Rate per model?
|
||||
}
|
||||
|
||||
// StatsCollector collects and aggregates statistics during the benchmark.
|
||||
// It needs to be thread-safe.
|
||||
type StatsCollector struct {
|
||||
mutex sync.Mutex
|
||||
results []RequestResult // Store individual results for percentile calculation
|
||||
startTime time.Time
|
||||
totalTokens int64
|
||||
totalReqTime time.Duration // Sum of latencies for successful requests
|
||||
successCount int64
|
||||
failCount int64
|
||||
minLatency time.Duration
|
||||
maxLatency time.Duration
|
||||
minTTFT time.Duration
|
||||
maxTTFT time.Duration
|
||||
initialized bool
|
||||
// Raw data storage
|
||||
latencies []time.Duration // Store raw latencies for histogram
|
||||
ttfts []time.Duration // Store raw TTFTs for histogram
|
||||
}
|
||||
|
||||
// NewStatsCollector creates a new thread-safe StatsCollector.
|
||||
func NewStatsCollector() *StatsCollector {
|
||||
return &StatsCollector{
|
||||
results: make([]RequestResult, 0, 1000), // Pre-allocate some capacity
|
||||
latencies: make([]time.Duration, 0, 1000), // Initialize
|
||||
ttfts: make([]time.Duration, 0, 1000), // Initialize
|
||||
startTime: time.Now(),
|
||||
minLatency: time.Duration(1<<63 - 1), // Max duration as initial min
|
||||
minTTFT: time.Duration(1<<63 - 1), // Max duration as initial min
|
||||
initialized: true,
|
||||
}
|
||||
}
|
||||
|
||||
// RecordResult records the outcome of a single request. Thread-safe.
|
||||
func (sc *StatsCollector) RecordResult(result RequestResult) {
|
||||
sc.mutex.Lock()
|
||||
defer sc.mutex.Unlock()
|
||||
|
||||
if !sc.initialized {
|
||||
// This shouldn't happen with NewStatsCollector, but defensive check
|
||||
return
|
||||
}
|
||||
|
||||
log.Printf("[StatsCollector] Recording result: Success=%t, Latency=%s, TTFT=%s, Tokens=%d", result.IsSuccess, result.Latency, result.TimeToFirstToken, result.TotalTokens)
|
||||
sc.results = append(sc.results, result) // Append for percentile calculation
|
||||
|
||||
if result.IsSuccess {
|
||||
sc.successCount++
|
||||
sc.totalTokens += int64(result.TotalTokens)
|
||||
sc.totalReqTime += result.Latency
|
||||
|
||||
if result.Latency < sc.minLatency {
|
||||
sc.minLatency = result.Latency
|
||||
}
|
||||
if result.Latency > sc.maxLatency {
|
||||
sc.maxLatency = result.Latency
|
||||
}
|
||||
sc.latencies = append(sc.latencies, result.Latency)
|
||||
|
||||
// TTFT calculations only for successful requests
|
||||
if result.TimeToFirstToken > 0 { // Ensure valid TTFT
|
||||
if result.TimeToFirstToken < sc.minTTFT {
|
||||
sc.minTTFT = result.TimeToFirstToken
|
||||
}
|
||||
if result.TimeToFirstToken > sc.maxTTFT {
|
||||
sc.maxTTFT = result.TimeToFirstToken
|
||||
}
|
||||
sc.ttfts = append(sc.ttfts, result.TimeToFirstToken)
|
||||
}
|
||||
} else {
|
||||
sc.failCount++
|
||||
}
|
||||
}
|
||||
|
||||
// CalculateStats computes the final statistics based on recorded results.
|
||||
// totalDurationOverride allows specifying the exact benchmark duration used for QPS calculation.
|
||||
func (sc *StatsCollector) CalculateStats(totalDurationOverride time.Duration) FinalStats {
|
||||
sc.mutex.Lock()
|
||||
defer sc.mutex.Unlock()
|
||||
|
||||
stats := FinalStats{
|
||||
TotalDuration: totalDurationOverride,
|
||||
}
|
||||
|
||||
if !sc.initialized || len(sc.results) == 0 {
|
||||
return stats // Return empty stats if no results
|
||||
}
|
||||
|
||||
stats.SuccessfulRequests = sc.successCount
|
||||
stats.FailedRequests = sc.failCount
|
||||
stats.TotalRequests = sc.successCount + sc.failCount
|
||||
|
||||
if totalDurationOverride.Seconds() > 0 {
|
||||
stats.AvgQPS = float64(stats.TotalRequests) / totalDurationOverride.Seconds()
|
||||
}
|
||||
|
||||
if sc.successCount > 0 {
|
||||
stats.AvgLatency = sc.totalReqTime / time.Duration(sc.successCount)
|
||||
stats.MinLatency = sc.minLatency
|
||||
stats.MaxLatency = sc.maxLatency
|
||||
|
||||
// Filter successful results for latency/TTFT percentile calculations
|
||||
successfulLatencies := make([]float64, 0, sc.successCount)
|
||||
successfulTTFTs := make([]float64, 0, sc.successCount)
|
||||
var totalTTFT time.Duration
|
||||
|
||||
for _, res := range sc.results {
|
||||
if res.IsSuccess {
|
||||
successfulLatencies = append(successfulLatencies, float64(res.Latency.Nanoseconds()))
|
||||
if res.TimeToFirstToken > 0 {
|
||||
successfulTTFTs = append(successfulTTFTs, float64(res.TimeToFirstToken.Nanoseconds()))
|
||||
totalTTFT += res.TimeToFirstToken
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Calculate Avg TTFT
|
||||
if len(successfulTTFTs) > 0 {
|
||||
stats.AvgTimeToFirstToken = totalTTFT / time.Duration(len(successfulTTFTs))
|
||||
stats.MinTimeToFirstToken = sc.minTTFT
|
||||
stats.MaxTimeToFirstToken = sc.maxTTFT
|
||||
}
|
||||
|
||||
// Calculate average tokens/sec based on successful requests' total time
|
||||
if sc.totalReqTime.Seconds() > 0 {
|
||||
stats.AvgTokensPerSecond = float64(sc.totalTokens) / sc.totalReqTime.Seconds()
|
||||
}
|
||||
|
||||
// --- Percentile Calculations (using basic sort for now) ---
|
||||
sort.Float64s(successfulLatencies)
|
||||
if len(successfulLatencies) > 0 {
|
||||
stats.P90Latency = time.Duration(percentile(successfulLatencies, 90)) * time.Nanosecond
|
||||
stats.P95Latency = time.Duration(percentile(successfulLatencies, 95)) * time.Nanosecond
|
||||
stats.P99Latency = time.Duration(percentile(successfulLatencies, 99)) * time.Nanosecond
|
||||
}
|
||||
|
||||
sort.Float64s(successfulTTFTs)
|
||||
if len(successfulTTFTs) > 0 {
|
||||
stats.P90TimeToFirstToken = time.Duration(percentile(successfulTTFTs, 90)) * time.Nanosecond
|
||||
stats.P95TimeToFirstToken = time.Duration(percentile(successfulTTFTs, 95)) * time.Nanosecond
|
||||
stats.P99TimeToFirstToken = time.Duration(percentile(successfulTTFTs, 99)) * time.Nanosecond
|
||||
}
|
||||
|
||||
} else {
|
||||
// Handle cases where there are no successful requests
|
||||
stats.MinLatency = 0
|
||||
stats.MaxLatency = 0
|
||||
stats.MinTimeToFirstToken = 0
|
||||
stats.MaxTimeToFirstToken = 0
|
||||
}
|
||||
|
||||
// Assign raw data for histograms
|
||||
stats.LatencyData = make([]time.Duration, len(sc.latencies))
|
||||
copy(stats.LatencyData, sc.latencies)
|
||||
stats.TTFTData = make([]time.Duration, len(sc.ttfts))
|
||||
copy(stats.TTFTData, sc.ttfts)
|
||||
|
||||
return stats
|
||||
}
|
||||
|
||||
// percentile calculates the p-th percentile of a sorted float64 slice.
|
||||
// Basic implementation, assumes data is already sorted.
|
||||
func percentile(data []float64, p float64) float64 {
|
||||
if len(data) == 0 {
|
||||
return 0
|
||||
}
|
||||
if p <= 0 {
|
||||
return data[0]
|
||||
}
|
||||
if p >= 100 {
|
||||
return data[len(data)-1]
|
||||
}
|
||||
index := (p / 100.0) * float64(len(data)-1)
|
||||
lower := int(index)
|
||||
upper := lower + 1
|
||||
if upper >= len(data) {
|
||||
return data[lower] // Should only happen if index is exactly len(data)-1
|
||||
}
|
||||
weight := index - float64(lower)
|
||||
return data[lower]*(1-weight) + data[upper]*weight
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
package tokenizer
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"github.com/tiktoken-go/tokenizer"
|
||||
)
|
||||
|
||||
// A sample text to generate prompts from. Using Moby Dick chapter 1 start.
|
||||
const sampleText = `Call me Ishmael. Some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen and regulating the circulation. Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin warehouses, and bringing up the rear of every funeral I meet; and especially whenever my hypos get such an upper hand of me, that it requires a strong moral principle to prevent me from deliberately stepping into the street, and methodically knocking people’s hats off—then, I account it high time to get to sea as soon as I can. This is my substitute for pistol and ball. With a philosophical flourish Cato throws himself upon his sword; I quietly take to the ship. There is nothing surprising in this. If they but knew it, almost all men in their degree, some time or other, cherish very nearly the same feelings towards the ocean with me. There now is your insular city of the Manhattoes, belted round by wharves as Indian isles by coral reefs—commerce surrounds it with her surf. Right and left, the streets take you waterward. Its extreme downtown is the battery, where that noble mole is washed by waves, and cooled by breezes, which a few hours previous were out of sight of land. Look at the crowds of water-gazers there. Circumambulate the city of a dreamy Sabbath afternoon. Go from Corlears Hook to Coenties Slip, and from thence, by Whitehall, northward. What do you see?—Posted like silent sentinels all around the town, stand thousands upon thousands of mortal men fixed in ocean reveries. Some leaning against the spiles; some seated upon the pier-heads; some looking over the bulwarks of ships from China; some high aloft in the rigging, as if striving to get a still better seaward peep. But these are all landsmen; of week days pent up in lath and plaster—tied to counters, nailed to benches, worried by ledgers. How then is this? Are the green fields gone? What do they here? But look! here come more crowds, pacing straight for the water, and seemingly bound for a dive. Strange! Nothing will content them but the extremest limit of the land; loitering under the shady lee of yonder warehouses will not suffice. No. They must get just as nigh the water as they possibly can without falling in. And there they stand—miles of them—leagues. Inlanders all, they come from lanes and alleys, streets and avenues—north, east, south, and west. Yet here they all unite. Tell me, does the magnetic virtue of the needles of the compasses of all those ships attract them thither?`
|
||||
|
||||
var (
|
||||
codec tokenizer.Codec
|
||||
encodedText []uint
|
||||
initErr error
|
||||
codecMutex sync.Mutex // Mutex to protect concurrent access to the codec
|
||||
)
|
||||
|
||||
func init() {
|
||||
// Initialize the tokenizer and encode the sample text once
|
||||
codec, initErr = tokenizer.Get(tokenizer.Cl100kBase)
|
||||
if initErr != nil {
|
||||
initErr = fmt.Errorf("failed to get tokenizer codec: %w", initErr)
|
||||
return
|
||||
}
|
||||
ids, _, err := codec.Encode(sampleText)
|
||||
if err != nil {
|
||||
initErr = fmt.Errorf("failed to encode sample text: %w", err)
|
||||
return
|
||||
}
|
||||
encodedText = ids
|
||||
}
|
||||
|
||||
// GeneratePrompt generates a text prompt with approximately the target number of tokens.
|
||||
func GeneratePrompt(targetTokenCount int) (string, error) {
|
||||
// Check initialization errors first
|
||||
if initErr != nil {
|
||||
return "", fmt.Errorf("tokenizer initialization failed: %w", initErr)
|
||||
}
|
||||
if codec == nil {
|
||||
// This case should theoretically be covered by initErr check, but belt and suspenders
|
||||
return "", errors.New("tokenizer codec is not initialized")
|
||||
}
|
||||
|
||||
if targetTokenCount <= 0 {
|
||||
return "", errors.New("target token count must be positive")
|
||||
}
|
||||
|
||||
var selectedTokens []uint
|
||||
|
||||
// Repeat or truncate the encoded text to match the target count
|
||||
if targetTokenCount > len(encodedText) {
|
||||
// Repeat tokens until target is met
|
||||
selectedTokens = make([]uint, 0, targetTokenCount)
|
||||
for len(selectedTokens) < targetTokenCount {
|
||||
remaining := targetTokenCount - len(selectedTokens)
|
||||
if remaining >= len(encodedText) {
|
||||
selectedTokens = append(selectedTokens, encodedText...)
|
||||
} else {
|
||||
selectedTokens = append(selectedTokens, encodedText[:remaining]...)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Select the first N tokens
|
||||
selectedTokens = encodedText[:targetTokenCount]
|
||||
}
|
||||
|
||||
// --- Protect the call to Decode ---
|
||||
codecMutex.Lock()
|
||||
defer codecMutex.Unlock() // Ensure unlock happens even on error
|
||||
|
||||
// Decode the selected tokens back to text
|
||||
prompt, err := codec.Decode(selectedTokens)
|
||||
if err != nil {
|
||||
// Log the problematic token slice for debugging if needed
|
||||
// fmt.Printf("Failed to decode tokens: %v\n", selectedTokens)
|
||||
return "", fmt.Errorf("failed to decode %d selected tokens: %w", len(selectedTokens), err)
|
||||
}
|
||||
// ---------------------------------
|
||||
|
||||
return prompt, nil
|
||||
}
|
||||
|
||||
// CountTokensInText counts the number of tokens in the given text using the initialized codec.
|
||||
func CountTokensInText(text string) (int, error) {
|
||||
// Check initialization errors first
|
||||
if initErr != nil {
|
||||
return 0, fmt.Errorf("tokenizer initialization failed: %w", initErr)
|
||||
}
|
||||
if codec == nil {
|
||||
return 0, errors.New("tokenizer codec is not initialized")
|
||||
}
|
||||
|
||||
// --- Protect the call to Encode --- Must protect even read-only operations if codec has internal state?
|
||||
codecMutex.Lock()
|
||||
defer codecMutex.Unlock()
|
||||
|
||||
ids, _, err := codec.Encode(text)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("failed to encode text for token counting: %w", err)
|
||||
}
|
||||
// ---------------------------------
|
||||
|
||||
return len(ids), nil
|
||||
}
|
||||
|
||||
// CheckInitStatus returns any error that occurred during package initialization.
|
||||
func CheckInitStatus() error {
|
||||
return initErr
|
||||
}
|
||||
@@ -1,327 +0,0 @@
|
||||
package prompt
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/config"
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/logger"
|
||||
"github.com/pkoukk/tiktoken-go"
|
||||
)
|
||||
|
||||
// 填充词,用于调整提示词长度
|
||||
var fillerWords = []string{
|
||||
"详细", "简要", "请", "麻烦", "帮忙", "认真", "仔细", "全面", "系统性地",
|
||||
"清晰地", "准确地", "深入地", "专业地", "客观地", "科学地", "有条理地",
|
||||
"具体地", "简洁地", "完整地", "精确地", "有逻辑地", "有说服力地",
|
||||
}
|
||||
|
||||
// 长文本填充段落,用于扩展长文档提示词
|
||||
var longFillerParagraphs = []string{
|
||||
"请确保分析全面深入,考虑各个方面的影响和变化。需要包含具体的数据和案例来支持你的观点,同时也要有理论依据。",
|
||||
"在回答中,请注意逻辑性和连贯性,确保各部分之间有合理的过渡。同时,请尽量使用专业术语,但也要确保非专业人士能够理解。",
|
||||
"希望你能够从多个角度进行分析,包括但不限于社会学、经济学、政治学、心理学等视角。同时,也请考虑不同群体、不同地区的差异性。",
|
||||
"在分析中,请注意历史背景和未来趋势,将当前情况放在更广阔的时空背景下考量。同时,也请关注国际比较,了解全球范围内的相似与差异。",
|
||||
"请在回答中提供一些可行的建议或解决方案,这些建议应该是具体的、可操作的,并且考虑到实施过程中可能遇到的困难和阻力。",
|
||||
"在分析问题时,请注意考虑各种可能的因素,包括社会、经济、政治、文化、技术等方面。同时,也请思考这些因素之间的相互作用和影响。",
|
||||
"请在回答中使用清晰的结构,可以考虑使用小标题、要点列表等方式组织内容,使读者能够更容易地理解和把握核心内容。",
|
||||
"在分析过程中,请注意区分事实和观点,确保事实准确无误,观点有理有据。同时,也请尽量避免个人偏见,保持客观中立的态度。",
|
||||
"请确保你的回答既有理论深度,又有实践指导意义。理论分析应该有学术依据,实践建议应该考虑可行性和有效性。",
|
||||
"在回答中,请注意时效性,考虑最新的发展和变化。同时,也请关注长期趋势和基本规律,避免被短期波动所干扰。",
|
||||
}
|
||||
|
||||
// 替换词,用于模板中的占位符
|
||||
var replacements = map[string][]string{
|
||||
"country": {"中国", "美国", "日本", "英国", "法国", "德国", "俄罗斯", "加拿大", "澳大利亚", "巴西", "印度"},
|
||||
"concept": {"人工智能", "机器学习", "区块链", "量子计算", "云计算", "边缘计算", "物联网", "大数据", "虚拟现实", "增强现实"},
|
||||
"topic": {"全球变暖", "可持续发展", "数字化转型", "远程工作", "网络安全", "太空探索", "生物技术", "清洁能源", "智慧城市", "数字隐私"},
|
||||
"event": {"工业革命", "互联网诞生", "冷战", "登月", "柏林墙倒塌", "9/11事件", "COVID-19疫情", "阿拉伯之春", "2008金融危机", "人类基因组计划"},
|
||||
}
|
||||
|
||||
// Generator 提示词生成器
|
||||
type Generator struct {
|
||||
config *config.Config
|
||||
tokenizer *tiktoken.Tiktoken
|
||||
rand *rand.Rand
|
||||
}
|
||||
|
||||
// NewGenerator 创建新的提示词生成器
|
||||
func NewGenerator(cfg *config.Config) (*Generator, error) {
|
||||
// 初始化分词器
|
||||
logger.Debug("初始化分词器,模型: %s", cfg.Tokenizer.Model)
|
||||
tkm, err := tiktoken.EncodingForModel(cfg.Tokenizer.Model)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("初始化分词器失败: %w", err)
|
||||
}
|
||||
logger.Debug("分词器初始化成功")
|
||||
|
||||
// 初始化随机数生成器
|
||||
source := rand.NewSource(time.Now().UnixNano())
|
||||
|
||||
logger.Debug("提示词模板配置 - 短咨询: %d个模板, 目标Token: %d",
|
||||
len(cfg.PromptTemplates.Short.Templates), cfg.PromptTemplates.Short.TargetTokens)
|
||||
logger.Debug("提示词模板配置 - 长文档: %d个模板, 目标Token: %d",
|
||||
len(cfg.PromptTemplates.Long.Templates), cfg.PromptTemplates.Long.TargetTokens)
|
||||
|
||||
return &Generator{
|
||||
config: cfg,
|
||||
tokenizer: tkm,
|
||||
rand: rand.New(source),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// GeneratePrompt 生成指定类型的提示词
|
||||
func (g *Generator) GeneratePrompt(promptType string) (string, error) {
|
||||
var templateConfig config.PromptTypeConfig
|
||||
|
||||
// 根据类型选择模板配置
|
||||
switch promptType {
|
||||
case "short":
|
||||
templateConfig = g.config.PromptTemplates.Short
|
||||
case "long":
|
||||
templateConfig = g.config.PromptTemplates.Long
|
||||
default:
|
||||
return "", fmt.Errorf("未知的提示词类型: %s", promptType)
|
||||
}
|
||||
|
||||
// 检查模板是否存在
|
||||
if len(templateConfig.Templates) == 0 {
|
||||
return "", fmt.Errorf("提示词模板为空: %s", promptType)
|
||||
}
|
||||
|
||||
// 随机选择一个模板
|
||||
templateIndex := g.rand.Intn(len(templateConfig.Templates))
|
||||
template := templateConfig.Templates[templateIndex]
|
||||
logger.Debug("为%s类型选择模板 #%d: %s", promptType, templateIndex, template)
|
||||
|
||||
// 替换模板中的占位符
|
||||
prompt := g.replacePlaceholders(template)
|
||||
logger.Debug("替换占位符后: %s", prompt)
|
||||
|
||||
// 调整提示词长度至目标Token数
|
||||
result, err := g.adjustPromptLength(prompt, templateConfig.TargetTokens)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// 计算最终Token数
|
||||
tokens := g.tokenizer.Encode(result, nil, nil)
|
||||
logger.Debug("生成%s类型提示词成功,目标Token: %d, 实际Token: %d",
|
||||
promptType, templateConfig.TargetTokens, len(tokens))
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// replacePlaceholders 替换模板中的占位符
|
||||
func (g *Generator) replacePlaceholders(template string) string {
|
||||
result := template
|
||||
|
||||
// 替换所有占位符
|
||||
for placeholder, options := range replacements {
|
||||
placeholderTag := fmt.Sprintf("{%s}", placeholder)
|
||||
if strings.Contains(result, placeholderTag) {
|
||||
// 随机选择一个替换选项
|
||||
replacementIndex := g.rand.Intn(len(options))
|
||||
replacement := options[replacementIndex]
|
||||
logger.Debug("替换占位符 %s -> %s", placeholderTag, replacement)
|
||||
result = strings.Replace(result, placeholderTag, replacement, -1)
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// adjustPromptLength 调整提示词长度至目标Token数
|
||||
func (g *Generator) adjustPromptLength(prompt string, targetTokens int) (string, error) {
|
||||
// 计算当前Token数
|
||||
tokens := g.tokenizer.Encode(prompt, nil, nil)
|
||||
currentTokens := len(tokens)
|
||||
|
||||
// 计算目标范围(±5%)
|
||||
minTokens := int(float64(targetTokens) * 0.95)
|
||||
maxTokens := int(float64(targetTokens) * 1.05)
|
||||
|
||||
logger.Debug("调整提示词长度 - 当前: %d, 目标范围: %d-%d", currentTokens, minTokens, maxTokens)
|
||||
|
||||
// 如果当前Token数在目标范围内,直接返回
|
||||
if currentTokens >= minTokens && currentTokens <= maxTokens {
|
||||
logger.Debug("提示词长度已在目标范围内,无需调整")
|
||||
return prompt, nil
|
||||
}
|
||||
|
||||
// 如果Token数过少,添加填充词
|
||||
if currentTokens < minTokens {
|
||||
logger.Debug("提示词长度过短,需要扩展")
|
||||
return g.extendPrompt(prompt, targetTokens)
|
||||
}
|
||||
|
||||
// 如果Token数过多,截断提示词
|
||||
logger.Debug("提示词长度过长,需要截断")
|
||||
return g.truncatePrompt(prompt, targetTokens)
|
||||
}
|
||||
|
||||
// extendPrompt 通过添加填充词扩展提示词
|
||||
func (g *Generator) extendPrompt(prompt string, targetTokens int) (string, error) {
|
||||
result := prompt
|
||||
|
||||
// 计算当前Token数
|
||||
tokens := g.tokenizer.Encode(result, nil, nil)
|
||||
currentTokens := len(tokens)
|
||||
|
||||
// 计算目标范围
|
||||
minTokens := int(float64(targetTokens) * 0.95)
|
||||
|
||||
// 根据目标token数选择不同的扩展策略
|
||||
if targetTokens > 500 {
|
||||
// 长文档提示词,使用段落填充
|
||||
logger.Debug("使用段落填充扩展长文档提示词")
|
||||
return g.extendPromptWithParagraphs(prompt, targetTokens)
|
||||
}
|
||||
|
||||
// 短咨询提示词,使用填充词
|
||||
logger.Debug("使用填充词扩展短咨询提示词")
|
||||
|
||||
// 添加填充词,直到达到目标Token数
|
||||
fillerCount := 0
|
||||
for currentTokens < minTokens {
|
||||
// 随机选择一个填充词
|
||||
fillerIndex := g.rand.Intn(len(fillerWords))
|
||||
filler := fillerWords[fillerIndex]
|
||||
|
||||
// 在提示词前添加填充词
|
||||
result = filler + " " + result
|
||||
fillerCount++
|
||||
|
||||
// 重新计算Token数
|
||||
tokens = g.tokenizer.Encode(result, nil, nil)
|
||||
currentTokens = len(tokens)
|
||||
|
||||
// 防止无限循环
|
||||
if currentTokens > int(float64(targetTokens)*1.05) {
|
||||
// 如果添加填充词后超过了最大Token数,回退并尝试截断
|
||||
logger.Debug("添加%d个填充词后超过最大Token数,尝试截断", fillerCount)
|
||||
return g.truncatePrompt(result, targetTokens)
|
||||
}
|
||||
|
||||
if fillerCount > 100 {
|
||||
logger.Warn("添加填充词超过100次,尝试使用段落填充")
|
||||
return g.extendPromptWithParagraphs(prompt, targetTokens)
|
||||
}
|
||||
}
|
||||
|
||||
logger.Debug("扩展提示词成功,添加了%d个填充词,当前Token数: %d", fillerCount, currentTokens)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// extendPromptWithParagraphs 通过添加段落扩展长文档提示词
|
||||
func (g *Generator) extendPromptWithParagraphs(prompt string, targetTokens int) (string, error) {
|
||||
result := prompt
|
||||
|
||||
// 计算当前Token数
|
||||
tokens := g.tokenizer.Encode(result, nil, nil)
|
||||
currentTokens := len(tokens)
|
||||
|
||||
// 计算目标范围
|
||||
minTokens := int(float64(targetTokens) * 0.95)
|
||||
|
||||
// 添加段落,直到达到目标Token数
|
||||
paragraphCount := 0
|
||||
usedParagraphs := make(map[int]bool)
|
||||
|
||||
for currentTokens < minTokens && paragraphCount < 20 {
|
||||
// 随机选择一个未使用的段落
|
||||
var paragraphIndex int
|
||||
for {
|
||||
paragraphIndex = g.rand.Intn(len(longFillerParagraphs))
|
||||
if !usedParagraphs[paragraphIndex] {
|
||||
usedParagraphs[paragraphIndex] = true
|
||||
break
|
||||
}
|
||||
// 如果所有段落都已使用,重置使用记录
|
||||
if len(usedParagraphs) >= len(longFillerParagraphs) {
|
||||
usedParagraphs = make(map[int]bool)
|
||||
}
|
||||
}
|
||||
|
||||
paragraph := longFillerParagraphs[paragraphIndex]
|
||||
|
||||
// 添加段落
|
||||
if paragraphCount == 0 {
|
||||
result = result + "\n\n" + paragraph
|
||||
} else {
|
||||
result = result + "\n\n" + paragraph
|
||||
}
|
||||
paragraphCount++
|
||||
|
||||
// 重新计算Token数
|
||||
tokens = g.tokenizer.Encode(result, nil, nil)
|
||||
currentTokens = len(tokens)
|
||||
|
||||
logger.Debug("添加第%d个段落后,当前Token数: %d", paragraphCount, currentTokens)
|
||||
|
||||
// 如果已经达到目标范围,退出循环
|
||||
if currentTokens >= minTokens {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// 如果添加了最大段落数仍未达到目标,给出警告
|
||||
if currentTokens < minTokens {
|
||||
logger.Warn("即使添加了%d个段落,仍未达到目标Token数(当前: %d, 目标: %d)",
|
||||
paragraphCount, currentTokens, targetTokens)
|
||||
} else {
|
||||
logger.Debug("扩展长文档提示词成功,添加了%d个段落,当前Token数: %d", paragraphCount, currentTokens)
|
||||
}
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// truncatePrompt 通过截断缩短提示词
|
||||
func (g *Generator) truncatePrompt(prompt string, targetTokens int) (string, error) {
|
||||
// 计算目标范围
|
||||
maxTokens := int(float64(targetTokens) * 1.05)
|
||||
|
||||
// 编码提示词
|
||||
tokens := g.tokenizer.Encode(prompt, nil, nil)
|
||||
|
||||
// 如果Token数超过最大值,截断
|
||||
if len(tokens) > maxTokens {
|
||||
// 保留目标Token数
|
||||
originalLength := len(tokens)
|
||||
tokens = tokens[:maxTokens]
|
||||
|
||||
// 解码截断后的Token
|
||||
result := g.tokenizer.Decode(tokens)
|
||||
logger.Debug("截断提示词成功,从%d个Token截断到%d个Token", originalLength, len(tokens))
|
||||
return result, nil
|
||||
}
|
||||
|
||||
return prompt, nil
|
||||
}
|
||||
|
||||
// SelectPromptType 根据权重随机选择提示词类型
|
||||
func (g *Generator) SelectPromptType() string {
|
||||
// 计算累积权重
|
||||
var cumulativeWeights []float64
|
||||
var sum float64
|
||||
|
||||
for _, req := range g.config.Requests {
|
||||
sum += req.Weight
|
||||
cumulativeWeights = append(cumulativeWeights, sum)
|
||||
}
|
||||
|
||||
// 生成随机数
|
||||
r := g.rand.Float64() * sum
|
||||
|
||||
// 根据随机数选择提示词类型
|
||||
for i, weight := range cumulativeWeights {
|
||||
if r <= weight {
|
||||
return g.config.Requests[i].Type
|
||||
}
|
||||
}
|
||||
|
||||
// 默认返回第一个类型
|
||||
return g.config.Requests[0].Type
|
||||
}
|
||||
@@ -1,635 +0,0 @@
|
||||
package report
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/benchmark"
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/config"
|
||||
"github.com/acmestudio/llm-api-benchmark-tool/logger"
|
||||
"github.com/go-echarts/go-echarts/v2/charts"
|
||||
"github.com/go-echarts/go-echarts/v2/components"
|
||||
"github.com/go-echarts/go-echarts/v2/opts"
|
||||
"github.com/montanaflynn/stats"
|
||||
)
|
||||
|
||||
// Metrics 表示性能指标
|
||||
type Metrics struct {
|
||||
TotalRequests int // 总请求数
|
||||
SuccessfulRequests int // 成功请求数
|
||||
FailedRequests int // 失败请求数
|
||||
TimeoutRatio float64 // 超时比率
|
||||
AvgResponseTime time.Duration // 平均响应时间
|
||||
P90ResponseTime time.Duration // P90响应时间
|
||||
P95ResponseTime time.Duration // P95响应时间
|
||||
P99ResponseTime time.Duration // P99响应时间
|
||||
MaxResponseTime time.Duration // 最大响应时间
|
||||
MinResponseTime time.Duration // 最小响应时间
|
||||
AvgQPS float64 // 平均QPS
|
||||
MaxQPS float64 // 最大QPS
|
||||
AvgTokenRate float64 // 平均Token生成速率
|
||||
MaxTokenRate float64 // 最大Token生成速率
|
||||
MaxConcurrency int // 最大有效并发用户数
|
||||
}
|
||||
|
||||
// GenerateReport 生成性能报告
|
||||
func GenerateReport(results *benchmark.BenchmarkResults, cfg *config.Config, outputPath string) error {
|
||||
// 创建输出目录
|
||||
logger.Info("创建报告输出目录: %s", outputPath)
|
||||
if err := os.MkdirAll(outputPath, 0755); err != nil {
|
||||
return fmt.Errorf("创建输出目录失败: %w", err)
|
||||
}
|
||||
|
||||
// 检查结果是否为空
|
||||
if len(results.Results) == 0 {
|
||||
logger.Warn("没有测试结果数据,无法生成详细报告")
|
||||
} else {
|
||||
logger.Debug("共有 %d 条测试结果记录", len(results.Results))
|
||||
}
|
||||
|
||||
// 计算性能指标
|
||||
logger.Info("计算性能指标...")
|
||||
metrics := calculateMetrics(results)
|
||||
|
||||
// 输出主要性能指标
|
||||
logger.Info("性能指标摘要:")
|
||||
logger.Info(" 总请求数: %d (成功: %d, 失败: %d)",
|
||||
metrics.TotalRequests, metrics.SuccessfulRequests, metrics.FailedRequests)
|
||||
logger.Info(" 超时比率: %.2f%%", metrics.TimeoutRatio * 100)
|
||||
logger.Info(" 平均响应时间: %v", metrics.AvgResponseTime)
|
||||
logger.Info(" P90响应时间: %v", metrics.P90ResponseTime)
|
||||
logger.Info(" P95响应时间: %v", metrics.P95ResponseTime)
|
||||
logger.Info(" P99响应时间: %v", metrics.P99ResponseTime)
|
||||
logger.Info(" 平均QPS: %.2f", metrics.AvgQPS)
|
||||
logger.Info(" 最大QPS: %.2f", metrics.MaxQPS)
|
||||
logger.Info(" 平均Token生成速率: %.2f tokens/s", metrics.AvgTokenRate)
|
||||
logger.Info(" 最大Token生成速率: %.2f tokens/s", metrics.MaxTokenRate)
|
||||
logger.Info(" 最大有效并发用户数: %d", metrics.MaxConcurrency)
|
||||
|
||||
// 生成HTML报告
|
||||
logger.Info("生成HTML报告...")
|
||||
if err := generateHTMLReport(results, metrics, cfg, outputPath); err != nil {
|
||||
return fmt.Errorf("生成HTML报告失败: %w", err)
|
||||
}
|
||||
|
||||
// 生成CSV数据
|
||||
logger.Info("生成CSV数据...")
|
||||
if err := generateCSVData(results, outputPath); err != nil {
|
||||
return fmt.Errorf("生成CSV数据失败: %w", err)
|
||||
}
|
||||
|
||||
logger.Info("报告生成完成")
|
||||
return nil
|
||||
}
|
||||
|
||||
// calculateMetrics 计算性能指标
|
||||
func calculateMetrics(results *benchmark.BenchmarkResults) *Metrics {
|
||||
metrics := &Metrics{}
|
||||
|
||||
// 总请求数
|
||||
metrics.TotalRequests = len(results.Results)
|
||||
logger.Debug("计算指标 - 总请求数: %d", metrics.TotalRequests)
|
||||
|
||||
// 统计成功和失败请求
|
||||
var successfulRequests []benchmark.Result
|
||||
for _, result := range results.Results {
|
||||
if result.Error == nil {
|
||||
successfulRequests = append(successfulRequests, result)
|
||||
}
|
||||
}
|
||||
|
||||
metrics.SuccessfulRequests = len(successfulRequests)
|
||||
metrics.FailedRequests = metrics.TotalRequests - metrics.SuccessfulRequests
|
||||
logger.Debug("计算指标 - 成功请求: %d, 失败请求: %d",
|
||||
metrics.SuccessfulRequests, metrics.FailedRequests)
|
||||
|
||||
// 计算超时比率
|
||||
metrics.TimeoutRatio = float64(metrics.FailedRequests) / float64(metrics.TotalRequests)
|
||||
logger.Debug("计算指标 - 超时比率: %.2f%%", metrics.TimeoutRatio * 100)
|
||||
|
||||
// 如果没有成功的请求,返回
|
||||
if len(successfulRequests) == 0 {
|
||||
logger.Warn("没有成功的请求,无法计算详细指标")
|
||||
return metrics
|
||||
}
|
||||
|
||||
// 计算响应时间指标
|
||||
var responseTimes []float64
|
||||
for _, result := range successfulRequests {
|
||||
responseTimes = append(responseTimes, float64(result.ResponseTime.Milliseconds()))
|
||||
}
|
||||
|
||||
// 排序响应时间
|
||||
sort.Float64s(responseTimes)
|
||||
|
||||
// 计算响应时间百分位数
|
||||
p90, _ := stats.Percentile(responseTimes, 90)
|
||||
p95, _ := stats.Percentile(responseTimes, 95)
|
||||
p99, _ := stats.Percentile(responseTimes, 99)
|
||||
avg, _ := stats.Mean(responseTimes)
|
||||
max := responseTimes[len(responseTimes)-1]
|
||||
min := responseTimes[0]
|
||||
|
||||
metrics.AvgResponseTime = time.Duration(avg) * time.Millisecond
|
||||
metrics.P90ResponseTime = time.Duration(p90) * time.Millisecond
|
||||
metrics.P95ResponseTime = time.Duration(p95) * time.Millisecond
|
||||
metrics.P99ResponseTime = time.Duration(p99) * time.Millisecond
|
||||
metrics.MaxResponseTime = time.Duration(max) * time.Millisecond
|
||||
metrics.MinResponseTime = time.Duration(min) * time.Millisecond
|
||||
|
||||
logger.Debug("计算指标 - 响应时间(ms) - 平均: %.2f, P90: %.2f, P95: %.2f, P99: %.2f, 最小: %.2f, 最大: %.2f",
|
||||
avg, p90, p95, p99, min, max)
|
||||
|
||||
// 计算QPS
|
||||
totalDuration := results.EndTime.Sub(results.StartTime).Seconds()
|
||||
metrics.AvgQPS = float64(metrics.SuccessfulRequests) / totalDuration
|
||||
logger.Debug("计算指标 - 平均QPS: %.2f (总持续时间: %.2f秒)", metrics.AvgQPS, totalDuration)
|
||||
|
||||
// 计算每个并发步骤的QPS
|
||||
logger.Debug("计算各并发步骤的QPS...")
|
||||
for concurrency, stepResults := range results.ConcurrencyData {
|
||||
// 找出该步骤的第一个和最后一个请求
|
||||
if len(stepResults) == 0 {
|
||||
logger.Debug(" 并发步骤 %d: 没有请求数据", concurrency)
|
||||
continue
|
||||
}
|
||||
|
||||
// 排序结果,按时间戳
|
||||
sort.Slice(stepResults, func(i, j int) bool {
|
||||
return stepResults[i].Timestamp.Before(stepResults[j].Timestamp)
|
||||
})
|
||||
|
||||
firstRequest := stepResults[0]
|
||||
lastRequest := stepResults[len(stepResults)-1]
|
||||
|
||||
// 计算持续时间
|
||||
stepDuration := lastRequest.Timestamp.Sub(firstRequest.Timestamp).Seconds()
|
||||
if stepDuration <= 0 {
|
||||
logger.Debug(" 并发步骤 %d: 持续时间过短", concurrency)
|
||||
continue
|
||||
}
|
||||
|
||||
// 计算成功请求数
|
||||
var successCount int
|
||||
for _, result := range stepResults {
|
||||
if result.Error == nil {
|
||||
successCount++
|
||||
}
|
||||
}
|
||||
|
||||
// 计算QPS
|
||||
stepQPS := float64(successCount) / stepDuration
|
||||
logger.Debug(" 并发步骤 %d: QPS = %.2f (成功请求: %d, 持续时间: %.2f秒)",
|
||||
concurrency, stepQPS, successCount, stepDuration)
|
||||
|
||||
if stepQPS > metrics.MaxQPS {
|
||||
metrics.MaxQPS = stepQPS
|
||||
logger.Debug(" 更新最大QPS: %.2f (并发步骤: %d)", metrics.MaxQPS, concurrency)
|
||||
}
|
||||
|
||||
// 更新最大并发用户数
|
||||
if concurrency > metrics.MaxConcurrency && stepQPS > 0 {
|
||||
metrics.MaxConcurrency = concurrency
|
||||
logger.Debug(" 更新最大有效并发用户数: %d", metrics.MaxConcurrency)
|
||||
}
|
||||
}
|
||||
|
||||
// 计算Token生成速率
|
||||
var totalTokens int
|
||||
for _, result := range successfulRequests {
|
||||
totalTokens += result.CompletionTokens
|
||||
}
|
||||
|
||||
metrics.AvgTokenRate = float64(totalTokens) / totalDuration
|
||||
logger.Debug("计算指标 - 平均Token生成速率: %.2f tokens/s (总Token: %d)",
|
||||
metrics.AvgTokenRate, totalTokens)
|
||||
|
||||
// 计算每个并发步骤的Token生成速率
|
||||
logger.Debug("计算各并发步骤的Token生成速率...")
|
||||
for concurrency, stepResults := range results.ConcurrencyData {
|
||||
// 找出该步骤的第一个和最后一个请求
|
||||
if len(stepResults) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// 排序结果,按时间戳
|
||||
sort.Slice(stepResults, func(i, j int) bool {
|
||||
return stepResults[i].Timestamp.Before(stepResults[j].Timestamp)
|
||||
})
|
||||
|
||||
firstRequest := stepResults[0]
|
||||
lastRequest := stepResults[len(stepResults)-1]
|
||||
|
||||
// 计算持续时间
|
||||
stepDuration := lastRequest.Timestamp.Sub(firstRequest.Timestamp).Seconds()
|
||||
if stepDuration <= 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// 计算总Token数
|
||||
var stepTokens int
|
||||
for _, result := range stepResults {
|
||||
if result.Error == nil {
|
||||
stepTokens += result.CompletionTokens
|
||||
}
|
||||
}
|
||||
|
||||
// 计算Token生成速率
|
||||
stepTokenRate := float64(stepTokens) / stepDuration
|
||||
logger.Debug(" 并发步骤 %d: Token生成速率 = %.2f tokens/s (总Token: %d, 持续时间: %.2f秒)",
|
||||
concurrency, stepTokenRate, stepTokens, stepDuration)
|
||||
|
||||
if stepTokenRate > metrics.MaxTokenRate {
|
||||
metrics.MaxTokenRate = stepTokenRate
|
||||
logger.Debug(" 更新最大Token生成速率: %.2f tokens/s (并发步骤: %d)",
|
||||
metrics.MaxTokenRate, concurrency)
|
||||
}
|
||||
}
|
||||
|
||||
return metrics
|
||||
}
|
||||
|
||||
// generateHTMLReport 生成HTML报告
|
||||
func generateHTMLReport(results *benchmark.BenchmarkResults, metrics *Metrics, cfg *config.Config, outputPath string) error {
|
||||
// 创建页面
|
||||
logger.Debug("创建HTML报告页面...")
|
||||
page := components.NewPage()
|
||||
page.PageTitle = "LLM API 基准测试报告"
|
||||
page.Theme = "white"
|
||||
|
||||
// 添加概述
|
||||
logger.Debug("添加概述图表...")
|
||||
page.AddCharts(createOverviewChart(results, metrics, cfg))
|
||||
|
||||
// 添加响应时间分布图
|
||||
logger.Debug("添加响应时间分布图...")
|
||||
page.AddCharts(createResponseTimeDistributionChart(results))
|
||||
|
||||
// 添加QPS随时间变化图
|
||||
logger.Debug("添加QPS随时间变化图...")
|
||||
page.AddCharts(createQPSOverTimeChart(results))
|
||||
|
||||
// 添加Token生成速率随时间变化图
|
||||
logger.Debug("添加Token生成速率随时间变化图...")
|
||||
page.AddCharts(createTokenRateOverTimeChart(results))
|
||||
|
||||
// 添加并发与响应时间关系图
|
||||
logger.Debug("添加并发与响应时间关系图...")
|
||||
page.AddCharts(createConcurrencyVsResponseTimeChart(results))
|
||||
|
||||
// 保存HTML报告
|
||||
htmlPath := filepath.Join(outputPath, "report.html")
|
||||
logger.Debug("保存HTML报告: %s", htmlPath)
|
||||
f, err := os.Create(htmlPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
return page.Render(f)
|
||||
}
|
||||
|
||||
// createOverviewChart 创建概述图表
|
||||
func createOverviewChart(results *benchmark.BenchmarkResults, metrics *Metrics, cfg *config.Config) *charts.Bar {
|
||||
// 创建图表
|
||||
chart := charts.NewBar()
|
||||
chart.SetGlobalOptions(
|
||||
charts.WithTitleOpts(opts.Title{
|
||||
Title: "性能指标概述",
|
||||
}),
|
||||
)
|
||||
|
||||
// 添加数据
|
||||
chart.SetXAxis([]string{"平均响应时间 (ms)", "P90响应时间 (ms)", "P95响应时间 (ms)", "P99响应时间 (ms)"})
|
||||
chart.AddSeries("响应时间", []opts.BarData{
|
||||
{Value: metrics.AvgResponseTime.Milliseconds()},
|
||||
{Value: metrics.P90ResponseTime.Milliseconds()},
|
||||
{Value: metrics.P95ResponseTime.Milliseconds()},
|
||||
{Value: metrics.P99ResponseTime.Milliseconds()},
|
||||
})
|
||||
|
||||
return chart
|
||||
}
|
||||
|
||||
// createResponseTimeDistributionChart 创建响应时间分布图
|
||||
func createResponseTimeDistributionChart(results *benchmark.BenchmarkResults) *charts.Bar {
|
||||
// 创建图表
|
||||
chart := charts.NewBar()
|
||||
chart.SetGlobalOptions(
|
||||
charts.WithTitleOpts(opts.Title{
|
||||
Title: "响应时间分布",
|
||||
}),
|
||||
)
|
||||
|
||||
// 统计响应时间分布
|
||||
timeRanges := []int{100, 200, 500, 1000, 2000, 5000, 10000}
|
||||
counts := make([]int, len(timeRanges)+1)
|
||||
|
||||
for _, result := range results.Results {
|
||||
if result.Error != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
responseTime := result.ResponseTime.Milliseconds()
|
||||
found := false
|
||||
|
||||
for i, timeRange := range timeRanges {
|
||||
if responseTime <= int64(timeRange) {
|
||||
counts[i]++
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if !found {
|
||||
counts[len(timeRanges)]++
|
||||
}
|
||||
}
|
||||
|
||||
// 构建X轴标签
|
||||
xAxisLabels := make([]string, len(timeRanges)+1)
|
||||
for i, timeRange := range timeRanges {
|
||||
if i == 0 {
|
||||
xAxisLabels[i] = fmt.Sprintf("≤%dms", timeRange)
|
||||
} else {
|
||||
xAxisLabels[i] = fmt.Sprintf("%d-%dms", timeRanges[i-1], timeRange)
|
||||
}
|
||||
}
|
||||
xAxisLabels[len(timeRanges)] = fmt.Sprintf(">%dms", timeRanges[len(timeRanges)-1])
|
||||
|
||||
// 添加数据
|
||||
chart.SetXAxis(xAxisLabels)
|
||||
|
||||
barData := make([]opts.BarData, len(counts))
|
||||
for i, count := range counts {
|
||||
barData[i] = opts.BarData{Value: count}
|
||||
}
|
||||
|
||||
chart.AddSeries("请求数", barData)
|
||||
|
||||
return chart
|
||||
}
|
||||
|
||||
// createQPSOverTimeChart 创建QPS随时间变化图
|
||||
func createQPSOverTimeChart(results *benchmark.BenchmarkResults) *charts.Line {
|
||||
// 创建图表
|
||||
chart := charts.NewLine()
|
||||
chart.SetGlobalOptions(
|
||||
charts.WithTitleOpts(opts.Title{
|
||||
Title: "QPS随时间变化",
|
||||
}),
|
||||
)
|
||||
|
||||
// 按并发步骤分组
|
||||
for concurrency, stepResults := range results.ConcurrencyData {
|
||||
// 按时间窗口统计QPS
|
||||
windowSize := 5 * time.Second
|
||||
|
||||
// 找出该步骤的第一个和最后一个请求
|
||||
if len(stepResults) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// 排序结果,按时间戳
|
||||
sort.Slice(stepResults, func(i, j int) bool {
|
||||
return stepResults[i].Timestamp.Before(stepResults[j].Timestamp)
|
||||
})
|
||||
|
||||
firstRequest := stepResults[0]
|
||||
lastRequest := stepResults[len(stepResults)-1]
|
||||
|
||||
// 计算时间窗口数
|
||||
duration := lastRequest.Timestamp.Sub(firstRequest.Timestamp)
|
||||
windowCount := int(duration / windowSize) + 1
|
||||
|
||||
// 初始化时间窗口
|
||||
windows := make([]time.Time, windowCount)
|
||||
counts := make([]int, windowCount)
|
||||
|
||||
for i := 0; i < windowCount; i++ {
|
||||
windows[i] = firstRequest.Timestamp.Add(time.Duration(i) * windowSize)
|
||||
}
|
||||
|
||||
// 统计每个时间窗口的请求数
|
||||
for _, result := range stepResults {
|
||||
if result.Error != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
windowIndex := int(result.Timestamp.Sub(firstRequest.Timestamp) / windowSize)
|
||||
if windowIndex >= 0 && windowIndex < windowCount {
|
||||
counts[windowIndex]++
|
||||
}
|
||||
}
|
||||
|
||||
// 计算QPS
|
||||
qpsData := make([]opts.LineData, windowCount)
|
||||
xAxisData := make([]string, windowCount)
|
||||
|
||||
for i := 0; i < windowCount; i++ {
|
||||
qps := float64(counts[i]) / windowSize.Seconds()
|
||||
qpsData[i] = opts.LineData{Value: qps}
|
||||
xAxisData[i] = windows[i].Format("15:04:05")
|
||||
}
|
||||
|
||||
// 添加数据
|
||||
chart.SetXAxis(xAxisData)
|
||||
chart.AddSeries(fmt.Sprintf("并发 %d", concurrency), qpsData)
|
||||
}
|
||||
|
||||
return chart
|
||||
}
|
||||
|
||||
// createTokenRateOverTimeChart 创建Token生成速率随时间变化图
|
||||
func createTokenRateOverTimeChart(results *benchmark.BenchmarkResults) *charts.Line {
|
||||
// 创建图表
|
||||
chart := charts.NewLine()
|
||||
chart.SetGlobalOptions(
|
||||
charts.WithTitleOpts(opts.Title{
|
||||
Title: "Token生成速率随时间变化",
|
||||
}),
|
||||
)
|
||||
|
||||
// 按并发步骤分组
|
||||
for concurrency, stepResults := range results.ConcurrencyData {
|
||||
// 按时间窗口统计Token生成速率
|
||||
windowSize := 5 * time.Second
|
||||
|
||||
// 找出该步骤的第一个和最后一个请求
|
||||
if len(stepResults) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// 排序结果,按时间戳
|
||||
sort.Slice(stepResults, func(i, j int) bool {
|
||||
return stepResults[i].Timestamp.Before(stepResults[j].Timestamp)
|
||||
})
|
||||
|
||||
firstRequest := stepResults[0]
|
||||
lastRequest := stepResults[len(stepResults)-1]
|
||||
|
||||
// 计算时间窗口数
|
||||
duration := lastRequest.Timestamp.Sub(firstRequest.Timestamp)
|
||||
windowCount := int(duration / windowSize) + 1
|
||||
|
||||
// 初始化时间窗口
|
||||
windows := make([]time.Time, windowCount)
|
||||
tokens := make([]int, windowCount)
|
||||
|
||||
for i := 0; i < windowCount; i++ {
|
||||
windows[i] = firstRequest.Timestamp.Add(time.Duration(i) * windowSize)
|
||||
}
|
||||
|
||||
// 统计每个时间窗口的Token数
|
||||
for _, result := range stepResults {
|
||||
if result.Error != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
windowIndex := int(result.Timestamp.Sub(firstRequest.Timestamp) / windowSize)
|
||||
if windowIndex >= 0 && windowIndex < windowCount {
|
||||
tokens[windowIndex] += result.CompletionTokens
|
||||
}
|
||||
}
|
||||
|
||||
// 计算Token生成速率
|
||||
tokenRateData := make([]opts.LineData, windowCount)
|
||||
xAxisData := make([]string, windowCount)
|
||||
|
||||
for i := 0; i < windowCount; i++ {
|
||||
tokenRate := float64(tokens[i]) / windowSize.Seconds()
|
||||
tokenRateData[i] = opts.LineData{Value: tokenRate}
|
||||
xAxisData[i] = windows[i].Format("15:04:05")
|
||||
}
|
||||
|
||||
// 添加数据
|
||||
chart.SetXAxis(xAxisData)
|
||||
chart.AddSeries(fmt.Sprintf("并发 %d", concurrency), tokenRateData)
|
||||
}
|
||||
|
||||
return chart
|
||||
}
|
||||
|
||||
// createConcurrencyVsResponseTimeChart 创建并发与响应时间关系图
|
||||
func createConcurrencyVsResponseTimeChart(results *benchmark.BenchmarkResults) *charts.Line {
|
||||
// 创建图表
|
||||
chart := charts.NewLine()
|
||||
chart.SetGlobalOptions(
|
||||
charts.WithTitleOpts(opts.Title{
|
||||
Title: "并发与响应时间关系",
|
||||
}),
|
||||
)
|
||||
|
||||
// 准备数据
|
||||
var concurrencies []int
|
||||
var avgResponseTimes []opts.LineData
|
||||
var p90ResponseTimes []opts.LineData
|
||||
var p95ResponseTimes []opts.LineData
|
||||
|
||||
// 按并发步骤排序
|
||||
for concurrency := range results.ConcurrencyData {
|
||||
concurrencies = append(concurrencies, concurrency)
|
||||
}
|
||||
sort.Ints(concurrencies)
|
||||
|
||||
// 计算每个并发步骤的响应时间
|
||||
for _, concurrency := range concurrencies {
|
||||
stepResults := results.ConcurrencyData[concurrency]
|
||||
|
||||
// 收集成功请求的响应时间
|
||||
var responseTimes []float64
|
||||
for _, result := range stepResults {
|
||||
if result.Error == nil {
|
||||
responseTimes = append(responseTimes, float64(result.ResponseTime.Milliseconds()))
|
||||
}
|
||||
}
|
||||
|
||||
// 如果没有成功的请求,跳过
|
||||
if len(responseTimes) == 0 {
|
||||
avgResponseTimes = append(avgResponseTimes, opts.LineData{Value: 0})
|
||||
p90ResponseTimes = append(p90ResponseTimes, opts.LineData{Value: 0})
|
||||
p95ResponseTimes = append(p95ResponseTimes, opts.LineData{Value: 0})
|
||||
continue
|
||||
}
|
||||
|
||||
// 计算响应时间指标
|
||||
avg, _ := stats.Mean(responseTimes)
|
||||
p90, _ := stats.Percentile(responseTimes, 90)
|
||||
p95, _ := stats.Percentile(responseTimes, 95)
|
||||
|
||||
avgResponseTimes = append(avgResponseTimes, opts.LineData{Value: avg})
|
||||
p90ResponseTimes = append(p90ResponseTimes, opts.LineData{Value: p90})
|
||||
p95ResponseTimes = append(p95ResponseTimes, opts.LineData{Value: p95})
|
||||
}
|
||||
|
||||
// 将并发数转换为字符串
|
||||
xAxisData := make([]string, len(concurrencies))
|
||||
for i, concurrency := range concurrencies {
|
||||
xAxisData[i] = fmt.Sprintf("%d", concurrency)
|
||||
}
|
||||
|
||||
// 添加数据
|
||||
chart.SetXAxis(xAxisData)
|
||||
chart.AddSeries("平均响应时间 (ms)", avgResponseTimes)
|
||||
chart.AddSeries("P90响应时间 (ms)", p90ResponseTimes)
|
||||
chart.AddSeries("P95响应时间 (ms)", p95ResponseTimes)
|
||||
|
||||
return chart
|
||||
}
|
||||
|
||||
// generateCSVData 生成CSV数据
|
||||
func generateCSVData(results *benchmark.BenchmarkResults, outputPath string) error {
|
||||
// 创建CSV文件
|
||||
csvPath := filepath.Join(outputPath, "results.csv")
|
||||
logger.Debug("创建CSV文件: %s", csvPath)
|
||||
f, err := os.Create(csvPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
// 写入CSV头
|
||||
logger.Debug("写入CSV头...")
|
||||
_, err = f.WriteString("Timestamp,PromptType,PromptTokens,CompletionTokens,ResponseTime,Concurrency,Error\n")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 检查结果是否为空
|
||||
if len(results.Results) == 0 {
|
||||
logger.Warn("没有测试结果数据,CSV文件将只包含表头")
|
||||
return nil
|
||||
}
|
||||
|
||||
// 写入CSV数据
|
||||
logger.Debug("写入CSV数据,共%d条记录...", len(results.Results))
|
||||
for _, result := range results.Results {
|
||||
// 格式化错误信息
|
||||
errorMsg := ""
|
||||
if result.Error != nil {
|
||||
errorMsg = result.Error.Error()
|
||||
}
|
||||
|
||||
// 写入一行数据
|
||||
line := fmt.Sprintf("%s,%s,%d,%d,%d,%d,%s\n",
|
||||
result.Timestamp.Format(time.RFC3339),
|
||||
result.PromptType,
|
||||
result.PromptTokens,
|
||||
result.CompletionTokens,
|
||||
result.ResponseTime.Milliseconds(),
|
||||
result.Concurrency,
|
||||
errorMsg,
|
||||
)
|
||||
|
||||
_, err = f.WriteString(line)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
logger.Debug("CSV数据写入完成")
|
||||
return nil
|
||||
}
|
||||
@@ -1,104 +0,0 @@
|
||||
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>LLM API 基准测试报告</title>
|
||||
<script src="https://go-echarts.github.io/go-echarts-assets/assets/echarts.min.js"></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<style> .container {display: flex;justify-content: center;align-items: center;} .item {margin: auto;} </style>
|
||||
<div class="container">
|
||||
<div class="item" id="LNbBAPYnepJk" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_LNbBAPYnepJk = echarts.init(document.getElementById('LNbBAPYnepJk'), "white");
|
||||
let option_LNbBAPYnepJk = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"响应时间","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":40867},{"value":55877},{"value":57093},{"value":59096}]}],"title":{"text":"性能指标概述"},"tooltip":{"show":false},"xAxis":[{"data":["平均响应时间 (ms)","P90响应时间 (ms)","P95响应时间 (ms)","P99响应时间 (ms)"]}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_LNbBAPYnepJk = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_LNbBAPYnepJk.setOption(option_LNbBAPYnepJk);
|
||||
goecharts_LNbBAPYnepJk.dispatchAction(action_LNbBAPYnepJk);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="GOPBARBCnPTU" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_GOPBARBCnPTU = echarts.init(document.getElementById('GOPBARBCnPTU'), "white");
|
||||
let option_GOPBARBCnPTU = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"请求数","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":1},{"value":108}]}],"title":{"text":"响应时间分布"},"tooltip":{"show":false},"xAxis":[{"data":["≤100ms","100-200ms","200-500ms","500-1000ms","1000-2000ms","2000-5000ms","5000-10000ms",">10000ms"]}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_GOPBARBCnPTU = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_GOPBARBCnPTU.setOption(option_GOPBARBCnPTU);
|
||||
goecharts_GOPBARBCnPTU.dispatchAction(action_GOPBARBCnPTU);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="XkXZxRAArLPb" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_XkXZxRAArLPb = echarts.init(document.getElementById('XkXZxRAArLPb'), "white");
|
||||
let option_XkXZxRAArLPb = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"并发 1","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0}]},{"name":"并发 5","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":0.8,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.4,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0.2Line truncated
|
||||
;
|
||||
|
||||
let action_XkXZxRAArLPb = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_XkXZxRAArLPb.setOption(option_XkXZxRAArLPb);
|
||||
goecharts_XkXZxRAArLPb.dispatchAction(action_XkXZxRAArLPb);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="OFgvWOHKVKpD" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_OFgvWOHKVKpD = echarts.init(document.getElementById('OFgvWOHKVKpD'), "white");
|
||||
let option_OFgvWOHKVKpD = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"并发 1","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":353.4,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":248,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":316,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":347.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":237,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":223.6,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":215,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":338.6,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":289.8,"XAxisIndex":0,"YAxisIndex":0}]},{"name":"并发 5","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":1158.8,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":295.8,"XAxisIndex":0,"YAxisIndex":0},{"value":235.4,"XAxisIndex":0,"YAxisIndex":0},{"value":355.6,"XAxisIndex":0,"YAxisIndex":0},{"value":237.8,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":278,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":132.4,"XAxisIndex":0,"YAxisIndex":0},{"value":273.2,"XAxisIndex":0,"YAxisIndex":0},{"value":348.6,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":710.4,"XAxisIndex":0,"YAxisIndex":0},{"value":268.4,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":353.8,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":125,"XAxisIndex":0,"YAxisIndex":0},{"value":298.8,"XAxisIndex":0,"YAxisIndex":0},{"value":306,"XAxisIndex":0,"YAxisIndex":0},{"value":41.2,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":292.6,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxisIndex":0,"YAxisIndex":0},{"value":0,"XAxiLine truncated
|
||||
;
|
||||
|
||||
let action_OFgvWOHKVKpD = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_OFgvWOHKVKpD.setOption(option_OFgvWOHKVKpD);
|
||||
goecharts_OFgvWOHKVKpD.dispatchAction(action_OFgvWOHKVKpD);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="PSQeXGkwaeXv" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_PSQeXGkwaeXv = echarts.init(document.getElementById('PSQeXGkwaeXv'), "white");
|
||||
let option_PSQeXGkwaeXv = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"平均响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":41457,"XAxisIndex":0,"YAxisIndex":0},{"value":40918.05263157895,"XAxisIndex":0,"YAxisIndex":0},{"value":40750.27419354839,"XAxisIndex":0,"YAxisIndex":0}]},{"name":"P90响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":50759,"XAxisIndex":0,"YAxisIndex":0},{"value":55646.5,"XAxisIndex":0,"YAxisIndex":0},{"value":55877.5,"XAxisIndex":0,"YAxisIndex":0}]},{"name":"P95响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":[{"value":50759,"XAxisIndex":0,"YAxisIndex":0},{"value":56847.5,"XAxisIndex":0,"YAxisIndex":0},{"value":57093,"XAxisIndex":0,"YAxisIndex":0}]}],"title":{"text":"并发与响应时间关系"},"tooltip":{"show":false},"xAxis":[{"data":["1","5","10"]}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_PSQeXGkwaeXv = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_PSQeXGkwaeXv.setOption(option_PSQeXGkwaeXv);
|
||||
goecharts_PSQeXGkwaeXv.dispatchAction(action_PSQeXGkwaeXv);
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,125 +0,0 @@
|
||||
Timestamp,PromptType,PromptTokens,ResponseTokens,ResponseTime,ConcurrencyStep,Error
|
||||
2025-04-21T15:29:08+08:00,long,290,1767,50692,1,
|
||||
2025-04-21T15:29:59+08:00,short,32,1240,35591,1,
|
||||
2025-04-21T15:30:35+08:00,long,299,1580,46800,1,
|
||||
2025-04-21T15:31:22+08:00,long,285,1736,50826,1,
|
||||
2025-04-21T15:32:13+08:00,long,294,1185,35257,1,
|
||||
2025-04-21T15:32:48+08:00,long,278,1118,33428,1,
|
||||
2025-04-21T15:33:22+08:00,short,30,1075,31594,1,
|
||||
2025-04-21T15:33:54+08:00,short,35,1693,48817,1,
|
||||
2025-04-21T15:34:43+08:00,short,30,1449,40108,1,
|
||||
2025-04-21T15:35:23+08:00,short,31,1267,39528,5,
|
||||
2025-04-21T15:35:23+08:00,short,34,1331,43050,5,
|
||||
2025-04-21T15:35:23+08:00,long,281,1455,47254,5,
|
||||
2025-04-21T15:35:23+08:00,short,34,1741,54033,5,
|
||||
2025-04-21T15:35:23+08:00,short,0,0,60000,5,发送请求失败: timeout
|
||||
2025-04-21T15:36:07+08:00,short,34,1177,37663,5,
|
||||
2025-04-21T15:36:03+08:00,long,298,1479,47924,5,
|
||||
2025-04-21T15:36:18+08:00,short,36,1189,39376,5,
|
||||
2025-04-21T15:36:11+08:00,long,300,1778,56227,5,
|
||||
2025-04-21T15:36:45+08:00,short,30,662,22887,5,
|
||||
2025-04-21T15:36:23+08:00,long,295,1390,44672,5,
|
||||
2025-04-21T15:36:51+08:00,long,306,1366,44047,5,
|
||||
2025-04-21T15:37:08+08:00,long,298,1342,43062,5,
|
||||
2025-04-21T15:36:57+08:00,long,291,1743,56131,5,
|
||||
2025-04-21T15:37:08+08:00,short,31,1675,52352,5,
|
||||
2025-04-21T15:37:07+08:00,long,294,1877,58539,5,
|
||||
2025-04-21T15:37:52+08:00,short,31,625,19803,5,
|
||||
2025-04-21T15:38:06+08:00,short,33,206,7176,5,
|
||||
2025-04-21T15:37:35+08:00,long,288,1769,55162,5,
|
||||
2025-04-21T15:37:54+08:00,long,296,1494,47829,5,
|
||||
2025-04-21T15:38:01+08:00,long,295,1530,50380,5,
|
||||
2025-04-21T15:38:13+08:00,short,38,1463,45724,5,
|
||||
2025-04-21T15:38:31+08:00,short,34,906,28607,5,
|
||||
2025-04-21T15:38:42+08:00,short,33,691,21916,5,
|
||||
2025-04-21T15:38:12+08:00,long,0,0,60000,5,发送请求失败: timeout
|
||||
2025-04-21T15:38:59+08:00,short,34,676,22665,5,
|
||||
2025-04-21T15:38:51+08:00,short,34,1006,30879,5,
|
||||
2025-04-21T15:38:59+08:00,short,28,1185,36817,5,
|
||||
2025-04-21T15:39:04+08:00,long,281,1688,53184,5,
|
||||
2025-04-21T15:39:22+08:00,short,34,1192,35771,5,
|
||||
2025-04-21T15:39:37+08:00,short,35,922,29691,5,
|
||||
2025-04-21T15:39:12+08:00,long,299,1814,55118,5,
|
||||
2025-04-21T15:39:22+08:00,short,32,1555,46992,5,
|
||||
2025-04-21T15:40:07+08:00,short,33,777,25157,5,
|
||||
2025-04-21T15:39:57+08:00,short,29,1375,41815,5,
|
||||
2025-04-21T15:40:09+08:00,long,292,1278,38409,5,
|
||||
2025-04-21T15:40:06+08:00,long,290,1586,45621,5,
|
||||
2025-04-21T15:39:59+08:00,long,301,1874,57468,5,
|
||||
2025-04-21T15:40:52+08:00,long,287,1565,45984,5,
|
||||
2025-04-21T15:41:39+08:00,short,34,859,25973,5,
|
||||
2025-04-21T15:42:05+08:00,short,33,1031,36665,10,
|
||||
2025-04-21T15:42:05+08:00,short,31,1032,38132,10,
|
||||
2025-04-21T15:42:05+08:00,short,33,1124,41458,10,
|
||||
2025-04-21T15:42:05+08:00,short,31,1209,42973,10,
|
||||
2025-04-21T15:42:05+08:00,short,36,1260,45000,10,
|
||||
2025-04-21T15:42:05+08:00,long,301,1254,45741,10,
|
||||
2025-04-21T15:42:05+08:00,short,0,0,60025,10,发送请求失败: timeout
|
||||
2025-04-21T15:42:05+08:00,long,0,0,60024,10,发送请求失败: timeout
|
||||
2025-04-21T15:42:05+08:00,long,0,0,60024,10,发送请求失败: timeout
|
||||
2025-04-21T15:42:05+08:00,long,0,0,60027,10,发送请求失败: timeout
|
||||
2025-04-21T15:42:43+08:00,short,30,627,24785,10,
|
||||
2025-04-21T15:42:41+08:00,short,33,880,33993,10,
|
||||
2025-04-21T15:42:46+08:00,short,33,763,29495,10,
|
||||
2025-04-21T15:42:48+08:00,short,32,867,34001,10,
|
||||
2025-04-21T15:42:51+08:00,short,32,878,33547,10,
|
||||
2025-04-21T15:43:08+08:00,short,34,959,36283,10,
|
||||
2025-04-21T15:43:05+08:00,short,32,1059,40263,10,
|
||||
2025-04-21T15:42:50+08:00,long,0,0,60001,10,发送请求失败: timeout
|
||||
2025-04-21T15:43:05+08:00,long,302,1515,55759,10,
|
||||
2025-04-21T15:43:05+08:00,short,32,1480,55895,10,
|
||||
2025-04-21T15:43:05+08:00,long,0,0,60025,10,发送请求失败: timeout
|
||||
2025-04-21T15:43:44+08:00,short,34,663,25933,10,
|
||||
2025-04-21T15:43:16+08:00,long,302,1503,55131,10,
|
||||
2025-04-21T15:43:24+08:00,short,34,1385,50030,10,
|
||||
2025-04-21T15:43:16+08:00,long,290,1646,59167,10,
|
||||
2025-04-21T15:43:22+08:00,short,31,1512,56955,10,
|
||||
2025-04-21T15:43:45+08:00,short,28,1248,47396,10,
|
||||
2025-04-21T15:44:05+08:00,short,34,945,35028,10,
|
||||
2025-04-21T15:44:01+08:00,short,35,1134,43283,10,
|
||||
2025-04-21T15:44:15+08:00,short,35,921,32485,10,
|
||||
2025-04-21T15:43:50+08:00,short,34,1628,59025,10,
|
||||
2025-04-21T15:44:11+08:00,short,31,1160,42541,10,
|
||||
2025-04-21T15:44:15+08:00,short,34,1036,41375,10,
|
||||
2025-04-21T15:44:01+08:00,long,286,1507,56831,10,
|
||||
2025-04-21T15:44:48+08:00,short,28,345,13944,10,
|
||||
2025-04-21T15:44:33+08:00,short,28,965,35746,10,
|
||||
2025-04-21T15:44:10+08:00,short,34,1640,59733,10,
|
||||
2025-04-21T15:44:19+08:00,short,31,1563,55769,10,
|
||||
2025-04-21T15:44:54+08:00,short,37,721,27026,10,
|
||||
2025-04-21T15:44:44+08:00,long,291,1268,45194,10,
|
||||
2025-04-21T15:44:58+08:00,short,30,800,31400,10,
|
||||
2025-04-21T15:44:40+08:00,long,287,1422,54009,10,
|
||||
2025-04-21T15:44:49+08:00,short,32,1479,52350,10,
|
||||
2025-04-21T15:45:15+08:00,short,31,728,27863,10,
|
||||
2025-04-21T15:45:09+08:00,short,35,1127,42373,10,
|
||||
2025-04-21T15:44:56+08:00,short,0,0,60001,10,发送请求失败: timeout
|
||||
2025-04-21T15:45:02+08:00,short,34,1491,55425,10,
|
||||
2025-04-21T15:45:10+08:00,short,0,0,60001,10,发送请求失败: timeout
|
||||
2025-04-21T15:45:43+08:00,short,31,814,32256,10,
|
||||
2025-04-21T15:45:21+08:00,long,0,0,60001,10,发送请求失败: timeout
|
||||
2025-04-21T15:45:51+08:00,short,33,808,31357,10,
|
||||
2025-04-21T15:45:30+08:00,long,302,1447,54647,10,
|
||||
2025-04-21T15:45:56+08:00,short,36,761,29637,10,
|
||||
2025-04-21T15:45:42+08:00,short,34,1214,45591,10,
|
||||
2025-04-21T15:45:35+08:00,short,29,1465,54649,10,
|
||||
2025-04-21T15:45:29+08:00,long,0,0,60000,10,发送请求失败: timeout
|
||||
2025-04-21T15:45:58+08:00,short,35,874,34326,10,
|
||||
2025-04-21T15:46:10+08:00,short,27,1111,41640,10,
|
||||
2025-04-21T15:46:21+08:00,short,32,816,32250,10,
|
||||
2025-04-21T15:46:29+08:00,short,32,709,25910,10,
|
||||
2025-04-21T15:46:25+08:00,short,37,863,31820,10,
|
||||
2025-04-21T15:46:32+08:00,short,31,652,24537,10,
|
||||
2025-04-21T15:46:28+08:00,short,30,1034,37449,10,
|
||||
2025-04-21T15:46:15+08:00,long,0,0,60001,10,发送请求失败: timeout
|
||||
2025-04-21T15:46:54+08:00,short,28,607,23222,10,
|
||||
2025-04-21T15:46:26+08:00,long,294,1390,51678,10,
|
||||
2025-04-21T15:46:56+08:00,short,28,705,26915,10,
|
||||
2025-04-21T15:46:23+08:00,long,0,0,60001,10,发送请求失败: timeout
|
||||
2025-04-21T15:46:57+08:00,short,33,818,30689,10,
|
||||
2025-04-21T15:46:30+08:00,short,0,0,60030,10,发送请求失败: timeout
|
||||
2025-04-21T15:46:57+08:00,short,37,1139,40595,10,
|
||||
2025-04-21T15:46:52+08:00,long,293,1659,57231,10,
|
||||
2025-04-21T15:47:30+08:00,short,32,781,24085,10,
|
||||
2025-04-21T15:47:06+08:00,long,300,1785,55860,10,
|
||||
2025-04-21T15:47:49+08:00,long,297,1450,40171,10,
|
||||
|
@@ -1,104 +0,0 @@
|
||||
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>LLM API 基准测试报告</title>
|
||||
<script src="https://go-echarts.github.io/go-echarts-assets/assets/echarts.min.js"></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<style> .container {display: flex;justify-content: center;align-items: center;} .item {margin: auto;} </style>
|
||||
<div class="container">
|
||||
<div class="item" id="EHMZswDJUWqH" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_EHMZswDJUWqH = echarts.init(document.getElementById('EHMZswDJUWqH'), "white");
|
||||
let option_EHMZswDJUWqH = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"响应时间","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":0},{"value":0},{"value":0},{"value":0}]}],"title":{"text":"性能指标概述"},"tooltip":{"show":false},"xAxis":[{"data":["平均响应时间 (ms)","P90响应时间 (ms)","P95响应时间 (ms)","P99响应时间 (ms)"]}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_EHMZswDJUWqH = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_EHMZswDJUWqH.setOption(option_EHMZswDJUWqH);
|
||||
goecharts_EHMZswDJUWqH.dispatchAction(action_EHMZswDJUWqH);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="FLqyKjIZgeEc" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_FLqyKjIZgeEc = echarts.init(document.getElementById('FLqyKjIZgeEc'), "white");
|
||||
let option_FLqyKjIZgeEc = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"请求数","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0}]}],"title":{"text":"响应时间分布"},"tooltip":{"show":false},"xAxis":[{"data":["≤100ms","100-200ms","200-500ms","500-1000ms","1000-2000ms","2000-5000ms","5000-10000ms",">10000ms"]}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_FLqyKjIZgeEc = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_FLqyKjIZgeEc.setOption(option_FLqyKjIZgeEc);
|
||||
goecharts_FLqyKjIZgeEc.dispatchAction(action_FLqyKjIZgeEc);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="RXbTCdDVHsPX" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_RXbTCdDVHsPX = echarts.init(document.getElementById('RXbTCdDVHsPX'), "white");
|
||||
let option_RXbTCdDVHsPX = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":null,"title":{"text":"QPS随时间变化"},"tooltip":{"show":false},"xAxis":[{}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_RXbTCdDVHsPX = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_RXbTCdDVHsPX.setOption(option_RXbTCdDVHsPX);
|
||||
goecharts_RXbTCdDVHsPX.dispatchAction(action_RXbTCdDVHsPX);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="XNYfcbGGZVDe" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_XNYfcbGGZVDe = echarts.init(document.getElementById('XNYfcbGGZVDe'), "white");
|
||||
let option_XNYfcbGGZVDe = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":null,"title":{"text":"Token生成速率随时间变化"},"tooltip":{"show":false},"xAxis":[{}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_XNYfcbGGZVDe = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_XNYfcbGGZVDe.setOption(option_XNYfcbGGZVDe);
|
||||
goecharts_XNYfcbGGZVDe.dispatchAction(action_XNYfcbGGZVDe);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="odZNEMJQVnFD" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_odZNEMJQVnFD = echarts.init(document.getElementById('odZNEMJQVnFD'), "white");
|
||||
let option_odZNEMJQVnFD = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"平均响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null},{"name":"P90响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null},{"name":"P95响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null}],"title":{"text":"并发与响应时间关系"},"tooltip":{"show":false},"xAxis":[{"data":[]}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_odZNEMJQVnFD = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_odZNEMJQVnFD.setOption(option_odZNEMJQVnFD);
|
||||
goecharts_odZNEMJQVnFD.dispatchAction(action_odZNEMJQVnFD);
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
@@ -1 +0,0 @@
|
||||
Timestamp,PromptType,PromptTokens,CompletionTokens,ResponseTime,Concurrency,Error
|
||||
|
@@ -1,104 +0,0 @@
|
||||
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>LLM API 基准测试报告</title>
|
||||
<script src="https://go-echarts.github.io/go-echarts-assets/assets/echarts.min.js"></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<style> .container {display: flex;justify-content: center;align-items: center;} .item {margin: auto;} </style>
|
||||
<div class="container">
|
||||
<div class="item" id="VtXivSkVGWsI" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_VtXivSkVGWsI = echarts.init(document.getElementById('VtXivSkVGWsI'), "white");
|
||||
let option_VtXivSkVGWsI = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"响应时间","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":0},{"value":0},{"value":0},{"value":0}]}],"title":{"text":"性能指标概述"},"tooltip":{"show":false},"xAxis":[{"data":["平均响应时间 (ms)","P90响应时间 (ms)","P95响应时间 (ms)","P99响应时间 (ms)"]}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_VtXivSkVGWsI = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_VtXivSkVGWsI.setOption(option_VtXivSkVGWsI);
|
||||
goecharts_VtXivSkVGWsI.dispatchAction(action_VtXivSkVGWsI);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="vwmlpHKQyHVV" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_vwmlpHKQyHVV = echarts.init(document.getElementById('vwmlpHKQyHVV'), "white");
|
||||
let option_vwmlpHKQyHVV = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"请求数","type":"bar","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":false,"data":[{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0}]}],"title":{"text":"响应时间分布"},"tooltip":{"show":false},"xAxis":[{"data":["≤100ms","100-200ms","200-500ms","500-1000ms","1000-2000ms","2000-5000ms","5000-10000ms",">10000ms"]}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_vwmlpHKQyHVV = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_vwmlpHKQyHVV.setOption(option_vwmlpHKQyHVV);
|
||||
goecharts_vwmlpHKQyHVV.dispatchAction(action_vwmlpHKQyHVV);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="ONXViPrQSJxX" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_ONXViPrQSJxX = echarts.init(document.getElementById('ONXViPrQSJxX'), "white");
|
||||
let option_ONXViPrQSJxX = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":null,"title":{"text":"QPS随时间变化"},"tooltip":{"show":false},"xAxis":[{}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_ONXViPrQSJxX = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_ONXViPrQSJxX.setOption(option_ONXViPrQSJxX);
|
||||
goecharts_ONXViPrQSJxX.dispatchAction(action_ONXViPrQSJxX);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="KFGiaxLKrHzO" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_KFGiaxLKrHzO = echarts.init(document.getElementById('KFGiaxLKrHzO'), "white");
|
||||
let option_KFGiaxLKrHzO = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":null,"title":{"text":"Token生成速率随时间变化"},"tooltip":{"show":false},"xAxis":[{}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_KFGiaxLKrHzO = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_KFGiaxLKrHzO.setOption(option_KFGiaxLKrHzO);
|
||||
goecharts_KFGiaxLKrHzO.dispatchAction(action_KFGiaxLKrHzO);
|
||||
</script>
|
||||
|
||||
<div class="container">
|
||||
<div class="item" id="eUNIZaAWZYZJ" style="width:900px;height:500px;"></div>
|
||||
</div>
|
||||
|
||||
<script type="text/javascript">
|
||||
"use strict";
|
||||
let goecharts_eUNIZaAWZYZJ = echarts.init(document.getElementById('eUNIZaAWZYZJ'), "white");
|
||||
let option_eUNIZaAWZYZJ = {"animation":true,"color":["#5470c6","#91cc75","#fac858","#ee6666","#73c0de","#3ba272","#fc8452","#9a60b4","#ea7ccc"],"legend":{"show":true,"type":""},"series":[{"name":"平均响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null},{"name":"P90响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null},{"name":"P95响应时间 (ms)","type":"line","smooth":false,"connectNulls":false,"showSymbol":false,"waveAnimation":false,"renderLabelForZeroData":false,"selectedMode":false,"animation":true,"data":null}],"title":{"text":"并发与响应时间关系"},"tooltip":{"show":false},"xAxis":[{"data":[]}],"yAxis":[{}]}
|
||||
;
|
||||
|
||||
let action_eUNIZaAWZYZJ = {"areas":{},"type":""}
|
||||
;
|
||||
|
||||
goecharts_eUNIZaAWZYZJ.setOption(option_eUNIZaAWZYZJ);
|
||||
goecharts_eUNIZaAWZYZJ.dispatchAction(action_eUNIZaAWZYZJ);
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
@@ -1 +0,0 @@
|
||||
Timestamp,PromptType,PromptTokens,CompletionTokens,ResponseTime,Concurrency,Error
|
||||
|
@@ -0,0 +1,170 @@
|
||||
package test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"llm-api-benchmark-tool/pkg/client"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"io"
|
||||
"fmt"
|
||||
"time"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// TestHTTPClientDo tests the non-streaming Do method of the HTTPClient interface.
|
||||
func TestHTTPClientDo(t *testing.T) {
|
||||
// 1. Mock server setup
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
// Basic validation of the incoming request
|
||||
assert.Equal(t, "POST", r.Method)
|
||||
assert.Equal(t, "/test-endpoint", r.URL.Path)
|
||||
assert.Equal(t, "application/json", r.Header.Get("Content-Type"))
|
||||
assert.Equal(t, "Bearer test-key", r.Header.Get("Authorization"))
|
||||
|
||||
if r.ContentLength <= 0 {
|
||||
t.Errorf("Server received zero or negative Content-Length: %d", r.ContentLength)
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
_, _ = w.Write([]byte(`{"error":"missing or invalid content length"}`))
|
||||
return
|
||||
}
|
||||
|
||||
bodyBytes := make([]byte, r.ContentLength)
|
||||
_, err := io.ReadFull(r.Body, bodyBytes)
|
||||
assert.NoError(t, err, "Server failed to read expected number of body bytes")
|
||||
assert.Equal(t, `{"message":"ping"}`, string(bodyBytes))
|
||||
|
||||
// Send mock response
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Header().Set("X-Test-Header", "test-value")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write([]byte(`{"reply":"pong"}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
// 2. Instantiate client implementation
|
||||
httpClient := client.NewFastHTTPClient() // Using fasthttp client
|
||||
|
||||
// 3. Create mock Request
|
||||
req := &client.Request{
|
||||
Method: "POST",
|
||||
URL: server.URL + "/test-endpoint",
|
||||
Body: []byte(`{"message":"ping"}`),
|
||||
Headers: map[string]string{
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": "Bearer test-key",
|
||||
},
|
||||
}
|
||||
|
||||
// 4. Call Do method
|
||||
resp, err := httpClient.Do(req)
|
||||
|
||||
// 5. Assert Response correctness
|
||||
assert.NoError(t, err)
|
||||
assert.NotNil(t, resp)
|
||||
assert.Equal(t, http.StatusOK, resp.StatusCode)
|
||||
assert.Equal(t, `{"reply":"pong"}`, string(resp.Body))
|
||||
assert.Equal(t, "application/json", resp.Headers["Content-Type"])
|
||||
assert.Equal(t, "test-value", resp.Headers["X-Test-Header"])
|
||||
|
||||
// TODO: Add test cases for error handling (e.g., server error, network error)
|
||||
}
|
||||
|
||||
// TestHTTPClientStream tests the streaming Stream method of the HTTPClient interface.
|
||||
// It should verify SSE parsing, callback invocation, TTFT measurement, and handling of [DONE] message.
|
||||
func TestHTTPClientStream(t *testing.T) {
|
||||
// 1. Mock SSE server setup
|
||||
mockSSEData := []string{
|
||||
`data: {"id":"1","choices":[{"delta":{"role":"assistant"}}]}`, // Initial metadata
|
||||
`data: {"id":"1","choices":[{"delta":{"content":"Hello"}}]}`, // First content
|
||||
`data: {"id":"1","choices":[{"delta":{"reasoning_content":" Thinking..."}}]}`, // Reasoning content
|
||||
`data: {"id":"1","choices":[{"delta":{"content":" world"}}]}`, // More content
|
||||
`data: [DONE]`,
|
||||
}
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
assert.Equal(t, "POST", r.Method)
|
||||
assert.Equal(t, "/test-stream", r.URL.Path)
|
||||
assert.Equal(t, "Bearer test-key-stream", r.Header.Get("Authorization"))
|
||||
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
w.Header().Set("Cache-Control", "no-cache")
|
||||
w.Header().Set("Connection", "keep-alive")
|
||||
|
||||
flusher, ok := w.(http.Flusher)
|
||||
if !ok {
|
||||
t.Fatal("Server does not support flushing")
|
||||
}
|
||||
|
||||
for _, line := range mockSSEData {
|
||||
fmt.Fprintf(w, "%s\n\n", line)
|
||||
flusher.Flush() // Ensure data is sent immediately
|
||||
time.Sleep(10 * time.Millisecond) // Small delay between chunks
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
// 2. Instantiate client implementation
|
||||
httpClient := client.NewFastHTTPClient()
|
||||
|
||||
// 3. Create mock Request
|
||||
req := &client.Request{
|
||||
Method: "POST",
|
||||
URL: server.URL + "/test-stream",
|
||||
Body: []byte(`{"prompt":"test prompt","stream":true}`),
|
||||
Headers: map[string]string{
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": "Bearer test-key-stream",
|
||||
},
|
||||
}
|
||||
|
||||
// 4. Define callback function
|
||||
var receivedChunks []client.SSEChunk
|
||||
var firstChunkTime int64
|
||||
var mu sync.Mutex // Protect access to shared variables
|
||||
|
||||
callback := func(chunk client.SSEChunk) error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if firstChunkTime == 0 && len(chunk.Data) > 0 && !chunk.IsDone {
|
||||
firstChunkTime = chunk.Timestamp // Record time of first data chunk
|
||||
}
|
||||
receivedChunks = append(receivedChunks, chunk)
|
||||
// fmt.Printf("Callback received: IsDone=%v, Data=%s\n", chunk.IsDone, string(chunk.Data))
|
||||
return nil // No error from callback in this test
|
||||
}
|
||||
|
||||
// 5. Call Stream method (EXPECTED TO FAIL INITIALLY)
|
||||
startTime := time.Now().UnixNano()
|
||||
err := httpClient.Stream(req, callback)
|
||||
|
||||
// 6. Assert results (WILL FAIL INITIALLY)
|
||||
// assert.NoError(t, err) // Initially, this will fail
|
||||
if err != nil {
|
||||
assert.ErrorContains(t, err, "FastHTTPClient.Stream not implemented yet")
|
||||
// Since it's not implemented, we don't assert chunk content or TTFT yet
|
||||
} else {
|
||||
// --- Assertions for when Stream IS implemented ---
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
assert.NotEmpty(t, receivedChunks, "Callback should have been called")
|
||||
assert.Len(t, receivedChunks, len(mockSSEData), "Should receive the same number of chunks as sent")
|
||||
|
||||
// Verify last chunk is [DONE]
|
||||
lastChunk := receivedChunks[len(receivedChunks)-1]
|
||||
assert.True(t, lastChunk.IsDone, "Last chunk should be marked as IsDone")
|
||||
assert.Equal(t, "[DONE]", string(lastChunk.Data), "Last chunk data should be [DONE]")
|
||||
|
||||
// Verify TTFT calculation (placeholder)
|
||||
assert.Greater(t, firstChunkTime, startTime, "First chunk time should be after start time")
|
||||
ttftMs := float64(firstChunkTime-startTime) / 1e6
|
||||
t.Logf("Measured TTFT: %.2f ms", ttftMs)
|
||||
assert.Less(t, ttftMs, 100.0, "TTFT should be reasonably small in mock test") // Adjust threshold as needed
|
||||
|
||||
// TODO: Add more detailed assertions on specific chunk content if needed
|
||||
// TODO: Add assertions for handling 'content' vs 'reasoning_content'
|
||||
// TODO: Add test cases for error handling (server error, invalid SSE, callback error)
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// TODO: Add helper functions for mock server/client setup if needed
|
||||
@@ -0,0 +1,124 @@
|
||||
package test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt" // Added for createTestRequest
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"llm-api-benchmark-tool/pkg/client" // Needs client types
|
||||
"llm-api-benchmark-tool/pkg/concurrency" // Target package (to be created)
|
||||
"llm-api-benchmark-tool/pkg/stats" // Added for stats collector
|
||||
)
|
||||
|
||||
// --- Mock HTTP Client ---
|
||||
|
||||
type MockHTTPClient struct {
|
||||
DoFunc func(req *client.Request) (*client.Response, error)
|
||||
StreamFunc func(req *client.Request, callback func(chunk client.SSEChunk) error) error
|
||||
ReqCount atomic.Int64 // Track calls
|
||||
}
|
||||
|
||||
func (m *MockHTTPClient) Do(req *client.Request) (*client.Response, error) {
|
||||
m.ReqCount.Add(1)
|
||||
if m.DoFunc != nil {
|
||||
return m.DoFunc(req)
|
||||
}
|
||||
// Default mock response for Do
|
||||
return &client.Response{
|
||||
StatusCode: 200,
|
||||
Body: []byte(`{"mock": "response"}`),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (m *MockHTTPClient) Stream(req *client.Request, callback func(chunk client.SSEChunk) error) error {
|
||||
m.ReqCount.Add(1)
|
||||
if m.StreamFunc != nil {
|
||||
return m.StreamFunc(req, callback)
|
||||
}
|
||||
// Default mock stream behavior
|
||||
go func() { // Simulate receiving chunks
|
||||
time.Sleep(5 * time.Millisecond) // Simulate TTFT
|
||||
// Ensure callback is not nil before calling
|
||||
if callback != nil {
|
||||
callback(client.SSEChunk{Data: []byte(`{"mock": "chunk1"}`), Timestamp: time.Now().UnixNano()})
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
callback(client.SSEChunk{Data: []byte(`[DONE]`), Timestamp: time.Now().UnixNano(), IsDone: true})
|
||||
}
|
||||
}()
|
||||
return nil
|
||||
}
|
||||
|
||||
// --- Test Concurrency Manager ---
|
||||
|
||||
func TestConcurrencyManager_RunBenchmark(t *testing.T) {
|
||||
mockClient := &MockHTTPClient{}
|
||||
|
||||
params := concurrency.BenchmarkParams{
|
||||
HTTPClient: mockClient,
|
||||
TargetURL: "http://mock.test/api",
|
||||
TotalDuration: 100 * time.Millisecond, // Short duration for testing
|
||||
Concurrency: 10,
|
||||
RequestInterval: 5 * time.Millisecond, // Interval between starting new requests
|
||||
IsStreaming: false, // Test non-streaming first
|
||||
PromptTokens: 50, // Example prompt size
|
||||
// Add other necessary params like API Key, Model later
|
||||
}
|
||||
|
||||
// Create the manager (package and struct to be defined)
|
||||
statsCollector := stats.NewStatsCollector() // Create mock stats collector
|
||||
manager := concurrency.NewManager(params, statsCollector) // Assuming a constructor
|
||||
|
||||
// Run the benchmark
|
||||
// TODO: Define the result struct if RunBenchmark returns results
|
||||
err := manager.RunBenchmark(context.Background()) // Use background context for now
|
||||
|
||||
require.NoError(t, err, "RunBenchmark should complete without error")
|
||||
|
||||
// Assertions
|
||||
// 1. Check if roughly the correct number of requests were made.
|
||||
// Expected rate = Concurrency / RequestInterval (adjust if interval is per worker)
|
||||
// Expected requests ~ TotalDuration / RequestInterval (simplified, needs refinement)
|
||||
// Let's just check if *some* requests were made for now.
|
||||
madeRequests := mockClient.ReqCount.Load()
|
||||
assert.Greater(t, madeRequests, int64(0), "Should have made at least one request")
|
||||
t.Logf("Total requests made by mock client: %d", madeRequests)
|
||||
|
||||
// 2. Test with Streaming
|
||||
mockClientStreaming := &MockHTTPClient{}
|
||||
paramsStreaming := params
|
||||
paramsStreaming.HTTPClient = mockClientStreaming
|
||||
paramsStreaming.IsStreaming = true
|
||||
paramsStreaming.TotalDuration = 150 * time.Millisecond // Slightly longer for streaming
|
||||
|
||||
statsCollectorStreaming := stats.NewStatsCollector() // Create mock stats collector
|
||||
managerStreaming := concurrency.NewManager(paramsStreaming, statsCollectorStreaming)
|
||||
errStreaming := managerStreaming.RunBenchmark(context.Background())
|
||||
|
||||
require.NoError(t, errStreaming, "RunBenchmark (streaming) should complete without error")
|
||||
madeRequestsStreaming := mockClientStreaming.ReqCount.Load()
|
||||
assert.Greater(t, madeRequestsStreaming, int64(0), "Should have made at least one streaming request")
|
||||
t.Logf("Total streaming requests made by mock client: %d", madeRequestsStreaming)
|
||||
|
||||
|
||||
// TODO: Add test case for context cancellation/timeout
|
||||
// TODO: Add test case for handling client errors
|
||||
// TODO: Add more precise assertions on request count and timing if possible
|
||||
}
|
||||
|
||||
// Helper function to create a basic request (might move to a shared test util)
|
||||
func createTestRequest(url string, isStreaming bool, promptTokens int) *client.Request {
|
||||
// In a real scenario, we'd use the tokenizer here
|
||||
mockPrompt := fmt.Sprintf("mock prompt %d tokens", promptTokens)
|
||||
body := []byte(fmt.Sprintf(`{"prompt": "%s", "stream": %v}`, mockPrompt, isStreaming))
|
||||
return &client.Request{
|
||||
URL: url,
|
||||
Method: "POST",
|
||||
Headers: map[string]string{"Content-Type": "application/json"},
|
||||
Body: body,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,210 @@
|
||||
package test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"html"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"llm-api-benchmark-tool/pkg/client"
|
||||
"llm-api-benchmark-tool/pkg/concurrency"
|
||||
"llm-api-benchmark-tool/pkg/config"
|
||||
"llm-api-benchmark-tool/pkg/report"
|
||||
"llm-api-benchmark-tool/pkg/stats"
|
||||
)
|
||||
|
||||
// Helper to create durations
|
||||
func dms(ms int) time.Duration {
|
||||
return time.Duration(ms) * time.Millisecond
|
||||
}
|
||||
|
||||
// TestBenchmarkIntegration performs an end-to-end test of the benchmark tool.
|
||||
func TestBenchmarkIntegration(t *testing.T) {
|
||||
// --- 1. Set up Mock HTTP Server ---
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/v1/chat/completions", func(w http.ResponseWriter, r *http.Request) {
|
||||
// Simulate some processing time
|
||||
time.Sleep(dms(50 + int(time.Now().UnixMilli()%50))) // 50-100ms latency
|
||||
|
||||
// Check if it's a streaming request based on header or request body field
|
||||
// For simplicity, let's assume non-streaming for now
|
||||
isStreaming := false // TODO: Detect streaming requests properly
|
||||
|
||||
if isStreaming {
|
||||
// Simulate SSE streaming
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
w.Header().Set("Cache-Control", "no-cache")
|
||||
w.Header().Set("Connection", "keep-alive")
|
||||
flusher, ok := w.(http.Flusher)
|
||||
if !ok {
|
||||
http.Error(w, "Streaming unsupported!", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Send TTFT event quickly
|
||||
time.Sleep(dms(10 + int(time.Now().UnixMilli()%10))) // 10-20ms TTFT
|
||||
fmt.Fprintf(w, "id: msg1\nevent: content_block_delta\ndata: {\"type\": \"content_block_delta\", \"index\": 0, \"delta\": {\"type\": \"text_delta\", \"text\": \"Hello\"}}\n\n")
|
||||
flusher.Flush()
|
||||
|
||||
// Send subsequent events
|
||||
time.Sleep(dms(30))
|
||||
fmt.Fprintf(w, "id: msg2\nevent: content_block_delta\ndata: {\"type\": \"content_block_delta\", \"index\": 0, \"delta\": {\"type\": \"text_delta\", \"text\": \" world!\"}}\n\n")
|
||||
flusher.Flush()
|
||||
|
||||
// Send stop event
|
||||
time.Sleep(dms(10))
|
||||
fmt.Fprintf(w, "id: msg3\nevent: message_stop\ndata: {\"type\": \"message_stop\"}\n\n")
|
||||
flusher.Flush()
|
||||
} else {
|
||||
// Simulate non-streaming response
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
// A simple, valid-looking JSON response
|
||||
responseBody := `{
|
||||
"id": "chatcmpl-mock123",
|
||||
"object": "chat.completion",
|
||||
"created": 1677652288,
|
||||
"model": "gpt-mock-3.5",
|
||||
"choices": [{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "This is a mock response."
|
||||
},
|
||||
"finish_reason": "stop"
|
||||
}],
|
||||
"usage": {
|
||||
"prompt_tokens": 9,
|
||||
"completion_tokens": 12,
|
||||
"total_tokens": 21
|
||||
}
|
||||
}`
|
||||
fmt.Fprintln(w, responseBody)
|
||||
}
|
||||
})
|
||||
|
||||
server := httptest.NewServer(mux)
|
||||
defer server.Close()
|
||||
|
||||
// --- 2. Set up Configuration ---
|
||||
testDuration := 100 * time.Millisecond // Short duration for testing
|
||||
cfg := config.Config{
|
||||
APIEndpoint: server.URL + "/v1/chat/completions", // Use mock server URL
|
||||
Model: "gpt-mock-3.5",
|
||||
Client: "fasthttp", // Use fasthttp for testing
|
||||
Timeout: 5 * time.Second,
|
||||
Concurrency: 2, // Low concurrency for test
|
||||
Duration: testDuration,
|
||||
Requests: 10, // Limit requests for test (Note: Duration takes precedence)
|
||||
RateLimit: 0, // No rate limit for test
|
||||
Streaming: false, // Test non-streaming first
|
||||
PromptTokens: 10,
|
||||
MaxTokens: 20,
|
||||
OutputReport: filepath.Join(t.TempDir(), "integration_report.html"),
|
||||
Headers: make(map[string]string),
|
||||
Payload: make(map[string]interface{}),
|
||||
}
|
||||
|
||||
// --- 3. Initialize Components ---
|
||||
httpClient := client.NewFastHTTPClient()
|
||||
require.NotNil(t, httpClient, "Failed to create HTTP client")
|
||||
|
||||
statsCollector := stats.NewStatsCollector()
|
||||
|
||||
params := concurrency.BenchmarkParams{
|
||||
TargetURL: cfg.APIEndpoint, // Use correct field name
|
||||
HTTPClient: httpClient,
|
||||
TotalDuration: cfg.Duration,
|
||||
Concurrency: cfg.Concurrency,
|
||||
RequestInterval: 0, // No artificial delay between requests
|
||||
IsStreaming: cfg.Streaming,
|
||||
PromptTokens: cfg.PromptTokens,
|
||||
}
|
||||
|
||||
manager := concurrency.NewManager(params, statsCollector)
|
||||
require.NotNil(t, manager, "Manager should not be nil")
|
||||
|
||||
// --- 4. Run Benchmark ---
|
||||
t.Logf("Running benchmark against mock server: %s", cfg.APIEndpoint)
|
||||
startTime := time.Now()
|
||||
err := manager.RunBenchmark(context.Background()) // Pass context
|
||||
actualDuration := time.Since(startTime)
|
||||
require.NoError(t, err, "Benchmark run failed")
|
||||
|
||||
// --- 5. Get Final Stats ---
|
||||
finalStats := statsCollector.CalculateStats(actualDuration)
|
||||
t.Logf("Benchmark finished. Final stats: %+v", finalStats)
|
||||
|
||||
// Explicit check before assertion
|
||||
if finalStats.TotalRequests > 0 {
|
||||
t.Logf("Check PASSED: finalStats.TotalRequests (%d) > 0", finalStats.TotalRequests)
|
||||
} else {
|
||||
t.Errorf("Check FAILED: finalStats.TotalRequests (%d) is not > 0", finalStats.TotalRequests)
|
||||
}
|
||||
|
||||
// Replace assert.Greater with standard check due to potential testify issue
|
||||
if !(finalStats.TotalRequests > 0) { // Explicitly check the condition
|
||||
t.Errorf("Test FAILED: Expected TotalRequests (%d) to be greater than 0", finalStats.TotalRequests)
|
||||
}
|
||||
|
||||
// Replace assert.LessOrEqual with standard check due to potential testify issue
|
||||
if !(finalStats.FailedRequests <= 1) { // Explicitly check the condition
|
||||
t.Errorf("Test FAILED: Expected FailedRequests (%d) to be less than or equal to 1", finalStats.FailedRequests)
|
||||
}
|
||||
|
||||
// --- 6. Generate Report ---
|
||||
err = report.GenerateHTMLReport(finalStats, cfg.OutputReport) // Pass report path string
|
||||
require.NoError(t, err, "Failed to generate HTML report")
|
||||
t.Logf("Report generated at: %s", cfg.OutputReport)
|
||||
|
||||
// --- 7. Verify Report Content ---
|
||||
reportBytes, err := os.ReadFile(cfg.OutputReport)
|
||||
require.NoError(t, err, "Failed to read generated report file")
|
||||
reportContent := string(reportBytes)
|
||||
|
||||
// Check for basic summary stats in the report (adjust format as needed)
|
||||
assert.Contains(t, reportContent, "Total Requests:", "Report missing actual summary data (e.g., 'Total Requests:')")
|
||||
assert.Contains(t, reportContent, fmt.Sprintf("Total Requests:</td><td>%d", finalStats.TotalRequests), "Report missing or incorrect total requests")
|
||||
assert.Contains(t, reportContent, fmt.Sprintf("Successful Requests:</td><td>%d", finalStats.SuccessfulRequests), "Report missing or incorrect successful requests")
|
||||
assert.Contains(t, reportContent, "Avg Latency:", "Report missing avg latency")
|
||||
|
||||
// Check for chart divs
|
||||
assert.Contains(t, reportContent, "<div id=\"latencyHistogram\"", "Report missing latency histogram div")
|
||||
assert.Contains(t, reportContent, "<div id=\"ttftHistogram\"", "Report missing TTFT histogram div")
|
||||
|
||||
// Check for chart initialization JS (presence, not exact content)
|
||||
assert.Contains(t, reportContent, "echarts.init(document.getElementById('latencyHistogram')", "Report missing latency chart JS init")
|
||||
assert.Contains(t, reportContent, "echarts.init(document.getElementById('ttftHistogram')", "Report missing TTFT chart JS init")
|
||||
|
||||
// Check that histogram data is not just the default empty/"No Data" case
|
||||
// We need to find the data part of the JS options
|
||||
latencyOptionStart := strings.Index(reportContent, "let option_latencyHistogram = {")
|
||||
require.True(t, latencyOptionStart > 0, "Latency chart options not found in JS")
|
||||
|
||||
latencyOptionEnd := strings.Index(reportContent[latencyOptionStart:], "};")
|
||||
require.True(t, latencyOptionEnd > 0, "Latency chart options end not found")
|
||||
|
||||
latencyOptionJS := reportContent[latencyOptionStart : latencyOptionStart+latencyOptionEnd]
|
||||
|
||||
// Check that the default "No Data" category isn't present if we have data
|
||||
if len(finalStats.LatencyData) > 0 {
|
||||
assert.NotContains(t, latencyOptionJS, `"name":"Latency","type":"bar","data":[{"value":0}]`, "Latency chart shows default empty data when actual data exists")
|
||||
assert.NotContains(t, latencyOptionJS, `xaxis":[{"data":["No Data"]}]`, "Latency chart shows 'No Data' category when actual data exists") // Check category
|
||||
}
|
||||
|
||||
// Check HTML escaping for summary values (find an example value)
|
||||
avgLatencyStr := html.EscapeString(finalStats.AvgLatency.String())
|
||||
assert.Contains(t, reportContent, avgLatencyStr, "AvgLatency string not found or not escaped in HTML")
|
||||
|
||||
t.Log("Integration test completed successfully.")
|
||||
}
|
||||
|
||||
// TODO: Add another integration test case for streaming requests
|
||||
@@ -0,0 +1,86 @@
|
||||
package test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llm-api-benchmark-tool/pkg/report"
|
||||
"llm-api-benchmark-tool/pkg/stats"
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// createSampleStats creates a sample FinalStats struct for testing.
|
||||
func createSampleStats() stats.FinalStats {
|
||||
return stats.FinalStats{
|
||||
TotalRequests: 1000,
|
||||
SuccessfulRequests: 950,
|
||||
FailedRequests: 50,
|
||||
TotalDuration: 30 * time.Second,
|
||||
AvgLatency: 150 * time.Millisecond,
|
||||
MinLatency: 50 * time.Millisecond,
|
||||
MaxLatency: 500 * time.Millisecond,
|
||||
P90Latency: 250 * time.Millisecond,
|
||||
P95Latency: 300 * time.Millisecond,
|
||||
P99Latency: 400 * time.Millisecond,
|
||||
AvgTimeToFirstToken: 80 * time.Millisecond,
|
||||
MinTimeToFirstToken: 30 * time.Millisecond,
|
||||
MaxTimeToFirstToken: 200 * time.Millisecond,
|
||||
P90TimeToFirstToken: 120 * time.Millisecond,
|
||||
P95TimeToFirstToken: 150 * time.Millisecond,
|
||||
P99TimeToFirstToken: 180 * time.Millisecond,
|
||||
AvgQPS: 33.33,
|
||||
AvgTokensPerSecond: 500.5,
|
||||
// Add more fields if necessary for report testing
|
||||
}
|
||||
}
|
||||
|
||||
// TestGenerateHTMLReport tests the basic HTML report generation.
|
||||
func TestGenerateHTMLReport(t *testing.T) {
|
||||
sampleStats := createSampleStats()
|
||||
outputDir := t.TempDir() // Create a temporary directory for the report
|
||||
outputFilePath := outputDir + "/report.html"
|
||||
|
||||
// Define expected chart IDs (must match IDs set in generator.go)
|
||||
latencyChartID := "latencyHistogram"
|
||||
ttftChartID := "ttftHistogram"
|
||||
|
||||
// Generate the report
|
||||
err := report.GenerateHTMLReport(sampleStats, outputFilePath)
|
||||
assert.NoError(t, err, "GenerateHTMLReport should not return an error")
|
||||
|
||||
// Check if the file was created
|
||||
_, err = os.Stat(outputFilePath)
|
||||
assert.NoError(t, err, "HTML report file should be created")
|
||||
|
||||
// Read the generated file content
|
||||
contentBytes, err := os.ReadFile(outputFilePath)
|
||||
assert.NoError(t, err, "Should be able to read the generated report file")
|
||||
content := string(contentBytes)
|
||||
|
||||
// Basic checks on the HTML content
|
||||
assert.True(t, strings.HasPrefix(content, "<!DOCTYPE html>"), "Output should start with DOCTYPE html")
|
||||
assert.Contains(t, content, "<title>LLM API Benchmark Report</title>", "HTML should contain the correct title")
|
||||
assert.Contains(t, content, "<h1>LLM API Benchmark Report</h1>", "HTML should contain the main heading")
|
||||
assert.Contains(t, content, "<h2>Summary</h2>", "HTML should contain Summary section")
|
||||
assert.Contains(t, content, "<h2>Charts</h2>", "HTML should contain Charts section")
|
||||
|
||||
// Check if some key stats are present (as simple text for now)
|
||||
assert.Contains(t, content, "Total Requests:</td><td>1000</td>", "HTML should contain Total Requests stat")
|
||||
assert.Contains(t, content, "Avg Latency:</td><td>150ms</td>", "HTML should contain Avg Latency stat")
|
||||
assert.Contains(t, content, "Avg TTFT:</td><td>80ms</td>", "HTML should contain Avg TTFT stat")
|
||||
assert.Contains(t, content, "Avg QPS:</td><td>33.33</td>", "HTML should contain Avg QPS stat")
|
||||
|
||||
// Check if the ECharts initialization script snippet is present for each chart
|
||||
// latencyInitScript := fmt.Sprintf("echarts.init(document.getElementById('%s'))", latencyChartID)
|
||||
|
||||
// assert.Contains(t, content, latencyInitScript, "HTML should contain JS init for latency chart")
|
||||
|
||||
// Check if the chart divs are present (the ones generated by go-echarts render)
|
||||
assert.Contains(t, content, fmt.Sprintf(`<div id="%s"`, latencyChartID), "HTML should contain div for latency chart")
|
||||
assert.Contains(t, content, fmt.Sprintf(`<div id="%s"`, ttftChartID), "HTML should contain div for TTFT chart")
|
||||
|
||||
// TODO: Add more sophisticated checks? (e.g., parse HTML, check chart script data)
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
package test
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llm-api-benchmark-tool/pkg/stats"
|
||||
"github.com/stretchr/testify/assert"
|
||||
// "gonum.org/v1/gonum/stat" // Might need later for percentile verification
|
||||
)
|
||||
|
||||
// TestStatsCollector_BasicRecording tests basic recording and aggregation.
|
||||
func TestStatsCollector_BasicRecording(t *testing.T) {
|
||||
collector := stats.NewStatsCollector()
|
||||
|
||||
// Record some sample results
|
||||
collector.RecordResult(stats.RequestResult{
|
||||
IsSuccess: true,
|
||||
Latency: 100 * time.Millisecond,
|
||||
TimeToFirstToken: 50 * time.Millisecond, // Example TTFT
|
||||
TotalTokens: 150, // Example token count
|
||||
StreamLatency: nil, // Not a streaming result for this example
|
||||
})
|
||||
collector.RecordResult(stats.RequestResult{
|
||||
IsSuccess: true,
|
||||
Latency: 120 * time.Millisecond,
|
||||
TimeToFirstToken: 60 * time.Millisecond,
|
||||
TotalTokens: 160,
|
||||
StreamLatency: nil,
|
||||
})
|
||||
collector.RecordResult(stats.RequestResult{
|
||||
IsSuccess: false, // Failed request
|
||||
Latency: 50 * time.Millisecond, // Failed quickly
|
||||
TimeToFirstToken: 0, // No TTFT for failed requests
|
||||
TotalTokens: 0, // No tokens for failed requests
|
||||
StreamLatency: nil,
|
||||
})
|
||||
|
||||
// Calculate final stats
|
||||
finalStats := collector.CalculateStats(2 * time.Second) // Example total duration
|
||||
|
||||
assert.Equal(t, int64(3), finalStats.TotalRequests, "Total requests should be 3")
|
||||
assert.Equal(t, int64(2), finalStats.SuccessfulRequests, "Successful requests should be 2")
|
||||
assert.Equal(t, int64(1), finalStats.FailedRequests, "Failed requests should be 1")
|
||||
|
||||
// Basic average latency check (only successful)
|
||||
// Expected avg = (100 + 120) / 2 = 110 ms
|
||||
// Compare float64 seconds
|
||||
assert.InDelta(t, 110*time.Millisecond.Seconds(), finalStats.AvgLatency.Seconds(), 0.01, "Average latency mismatch")
|
||||
|
||||
// Basic average TTFT check (only successful)
|
||||
// Expected avg = (50 + 60) / 2 = 55 ms
|
||||
// Compare float64 seconds
|
||||
assert.InDelta(t, 55*time.Millisecond.Seconds(), finalStats.AvgTimeToFirstToken.Seconds(), 0.01, "Average TTFT mismatch")
|
||||
|
||||
// Check Min/Max Latency (successful only)
|
||||
assert.Equal(t, 100*time.Millisecond, finalStats.MinLatency, "Min latency mismatch")
|
||||
assert.Equal(t, 120*time.Millisecond, finalStats.MaxLatency, "Max latency mismatch")
|
||||
|
||||
// Check Min/Max TTFT (successful only)
|
||||
assert.Equal(t, 50*time.Millisecond, finalStats.MinTimeToFirstToken, "Min TTFT mismatch")
|
||||
assert.Equal(t, 60*time.Millisecond, finalStats.MaxTimeToFirstToken, "Max TTFT mismatch")
|
||||
|
||||
// Basic QPS check
|
||||
// Expected QPS = 3 requests / 2 seconds = 1.5
|
||||
assert.InDelta(t, 1.5, finalStats.AvgQPS, 0.01, "Average QPS mismatch")
|
||||
|
||||
// Basic Token Rate check (only successful)
|
||||
// Total tokens = 150 + 160 = 310
|
||||
// Total successful time = 100ms + 120ms = 220ms = 0.22s
|
||||
// Expected rate = 310 / 0.22 = ~1409 tokens/sec
|
||||
assert.InDelta(t, 310/0.22, finalStats.AvgTokensPerSecond, 1, "Average tokens/sec mismatch") // Larger delta due to potential floating point issues
|
||||
|
||||
// Check Percentiles (calculated manually for [100, 120] ms latency)
|
||||
// P90 = 100*(1-0.9) + 120*0.9 = 10 + 108 = 118
|
||||
// P95 = 100*(1-0.95) + 120*0.95 = 5 + 114 = 119
|
||||
// P99 = 100*(1-0.99) + 120*0.99 = 1 + 118.8 = 119.8
|
||||
assert.Equal(t, 118*time.Millisecond, finalStats.P90Latency, "P90 Latency mismatch")
|
||||
assert.Equal(t, 119*time.Millisecond, finalStats.P95Latency, "P95 Latency mismatch")
|
||||
assert.InDelta(t, 119.8*float64(time.Millisecond), float64(finalStats.P99Latency.Nanoseconds()), 0.1*float64(time.Millisecond), "P99 Latency mismatch")
|
||||
|
||||
// Check Percentiles (calculated manually for [50, 60] ms TTFT)
|
||||
// P90 = 50*0.1 + 60*0.9 = 5 + 54 = 59
|
||||
// P95 = 50*0.05 + 60*0.95 = 2.5 + 57 = 59.5
|
||||
// P99 = 50*0.01 + 60*0.99 = 0.5 + 59.4 = 59.9
|
||||
assert.Equal(t, 59*time.Millisecond, finalStats.P90TimeToFirstToken, "P90 TTFT mismatch")
|
||||
assert.InDelta(t, 59.5*float64(time.Millisecond), float64(finalStats.P95TimeToFirstToken.Nanoseconds()), 0.1*float64(time.Millisecond), "P95 TTFT mismatch")
|
||||
assert.InDelta(t, 59.9*float64(time.Millisecond), float64(finalStats.P99TimeToFirstToken.Nanoseconds()), 0.1*float64(time.Millisecond), "P99 TTFT mismatch")
|
||||
}
|
||||
|
||||
// TestStatsCollector_ConcurrentRecording tests thread-safety.
|
||||
func TestStatsCollector_ConcurrentRecording(t *testing.T) {
|
||||
collector := stats.NewStatsCollector()
|
||||
numGoroutines := 50
|
||||
numRequestsPerGoroutine := 100
|
||||
totalRequests := numGoroutines * numRequestsPerGoroutine
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(numGoroutines)
|
||||
|
||||
for i := 0; i < numGoroutines; i++ {
|
||||
go func(routineID int) {
|
||||
defer wg.Done()
|
||||
for j := 0; j < numRequestsPerGoroutine; j++ {
|
||||
latency := time.Duration(100+routineID+j) * time.Millisecond
|
||||
ttft := time.Duration(50+routineID+j) * time.Millisecond
|
||||
collector.RecordResult(stats.RequestResult{
|
||||
IsSuccess: true,
|
||||
Latency: latency,
|
||||
TimeToFirstToken: ttft,
|
||||
TotalTokens: 100,
|
||||
StreamLatency: nil,
|
||||
})
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
|
||||
finalStats := collector.CalculateStats(10 * time.Second) // Arbitrary duration
|
||||
|
||||
assert.Equal(t, int64(totalRequests), finalStats.TotalRequests, "Total requests should match after concurrent writes")
|
||||
assert.Equal(t, int64(totalRequests), finalStats.SuccessfulRequests, "All requests should be successful")
|
||||
assert.Equal(t, int64(0), finalStats.FailedRequests, "No requests should fail")
|
||||
// Add more assertions for concurrent results if necessary
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/tiktoken-go/tokenizer" // Assuming this is the correct import path
|
||||
|
||||
token "llm-api-benchmark-tool/pkg/tokenizer" // Target package (to be created)
|
||||
)
|
||||
|
||||
const (
|
||||
tolerance = 0.05 // 5%
|
||||
modelName = "gpt-4" // Or another common model for tiktoken
|
||||
)
|
||||
|
||||
// Helper function to count tokens
|
||||
func countTokens(t *testing.T, text string, model string) int {
|
||||
codec, err := tokenizer.Get(tokenizer.Cl100kBase) // Or appropriate codec based on model
|
||||
require.NoError(t, err, "Failed to get tokenizer codec")
|
||||
ids, _, err := codec.Encode(text)
|
||||
require.NoError(t, err, "Failed to encode text")
|
||||
return len(ids)
|
||||
}
|
||||
|
||||
func TestGeneratePrompt50Tokens(t *testing.T) {
|
||||
targetTokens := 50
|
||||
minTokens := int(float64(targetTokens) * (1 - tolerance))
|
||||
maxTokens := int(float64(targetTokens) * (1 + tolerance))
|
||||
|
||||
// Assuming GeneratePrompt takes the target count and maybe a model/codec hint
|
||||
prompt, err := token.GeneratePrompt(targetTokens)
|
||||
require.NoError(t, err, "GeneratePrompt should not return an error for 50 tokens")
|
||||
require.NotEmpty(t, prompt, "Generated prompt should not be empty")
|
||||
|
||||
actualTokens := countTokens(t, prompt, modelName)
|
||||
|
||||
assert.GreaterOrEqual(t, actualTokens, minTokens, fmt.Sprintf("Token count %d should be >= %d (target %d)", actualTokens, minTokens, targetTokens))
|
||||
assert.LessOrEqual(t, actualTokens, maxTokens, fmt.Sprintf("Token count %d should be <= %d (target %d)", actualTokens, maxTokens, targetTokens))
|
||||
t.Logf("Generated 50-token prompt (target: %d±%.0f%%): %d tokens", targetTokens, tolerance*100, actualTokens)
|
||||
}
|
||||
|
||||
func TestGeneratePrompt1000Tokens(t *testing.T) {
|
||||
targetTokens := 1000
|
||||
minTokens := int(float64(targetTokens) * (1 - tolerance))
|
||||
maxTokens := int(float64(targetTokens) * (1 + tolerance))
|
||||
|
||||
// Assuming GeneratePrompt takes the target count
|
||||
prompt, err := token.GeneratePrompt(targetTokens)
|
||||
require.NoError(t, err, "GeneratePrompt should not return an error for 1000 tokens")
|
||||
require.NotEmpty(t, prompt, "Generated prompt should not be empty")
|
||||
|
||||
actualTokens := countTokens(t, prompt, modelName)
|
||||
|
||||
assert.GreaterOrEqual(t, actualTokens, minTokens, fmt.Sprintf("Token count %d should be >= %d (target %d)", actualTokens, minTokens, targetTokens))
|
||||
assert.LessOrEqual(t, actualTokens, maxTokens, fmt.Sprintf("Token count %d should be <= %d (target %d)", actualTokens, maxTokens, targetTokens))
|
||||
t.Logf("Generated 1000-token prompt (target: %d±%.0f%%): %d tokens", targetTokens, tolerance*100, actualTokens)
|
||||
}
|
||||
|
||||
// TODO: Add test for invalid target token count (e.g., 0 or negative) if applicable
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
Copyright (c) 2009, 2010, 2013-2016 by the Brotli Authors.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in
|
||||
all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
This package is a brotli compressor and decompressor implemented in Go.
|
||||
It was translated from the reference implementation (https://github.com/google/brotli)
|
||||
with the `c2go` tool at https://github.com/andybalholm/c2go.
|
||||
|
||||
I have been working on new compression algorithms (not translated from C)
|
||||
in the matchfinder package.
|
||||
You can use them with the NewWriterV2 function.
|
||||
Currently they give better results than the old implementation
|
||||
(at least for compressing my test file, Newton’s *Opticks*)
|
||||
on levels 2 to 6.
|
||||
|
||||
I am using it in production with https://github.com/andybalholm/redwood.
|
||||
|
||||
API documentation is found at https://pkg.go.dev/github.com/andybalholm/brotli?tab=doc.
|
||||
+185
@@ -0,0 +1,185 @@
|
||||
package brotli
|
||||
|
||||
import (
|
||||
"sync"
|
||||
)
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Function to find backward reference copies. */
|
||||
|
||||
func computeDistanceCode(distance uint, max_distance uint, dist_cache []int) uint {
|
||||
if distance <= max_distance {
|
||||
var distance_plus_3 uint = distance + 3
|
||||
var offset0 uint = distance_plus_3 - uint(dist_cache[0])
|
||||
var offset1 uint = distance_plus_3 - uint(dist_cache[1])
|
||||
if distance == uint(dist_cache[0]) {
|
||||
return 0
|
||||
} else if distance == uint(dist_cache[1]) {
|
||||
return 1
|
||||
} else if offset0 < 7 {
|
||||
return (0x9750468 >> (4 * offset0)) & 0xF
|
||||
} else if offset1 < 7 {
|
||||
return (0xFDB1ACE >> (4 * offset1)) & 0xF
|
||||
} else if distance == uint(dist_cache[2]) {
|
||||
return 2
|
||||
} else if distance == uint(dist_cache[3]) {
|
||||
return 3
|
||||
}
|
||||
}
|
||||
|
||||
return distance + numDistanceShortCodes - 1
|
||||
}
|
||||
|
||||
var hasherSearchResultPool sync.Pool
|
||||
|
||||
func createBackwardReferences(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, hasher hasherHandle, dist_cache []int, last_insert_len *uint, commands *[]command, num_literals *uint) {
|
||||
var max_backward_limit uint = maxBackwardLimit(params.lgwin)
|
||||
var insert_length uint = *last_insert_len
|
||||
var pos_end uint = position + num_bytes
|
||||
var store_end uint
|
||||
if num_bytes >= hasher.StoreLookahead() {
|
||||
store_end = position + num_bytes - hasher.StoreLookahead() + 1
|
||||
} else {
|
||||
store_end = position
|
||||
}
|
||||
var random_heuristics_window_size uint = literalSpreeLengthForSparseSearch(params)
|
||||
var apply_random_heuristics uint = position + random_heuristics_window_size
|
||||
var gap uint = 0
|
||||
/* Set maximum distance, see section 9.1. of the spec. */
|
||||
|
||||
const kMinScore uint = scoreBase + 100
|
||||
|
||||
/* For speed up heuristics for random data. */
|
||||
|
||||
/* Minimum score to accept a backward reference. */
|
||||
hasher.PrepareDistanceCache(dist_cache)
|
||||
sr2, _ := hasherSearchResultPool.Get().(*hasherSearchResult)
|
||||
if sr2 == nil {
|
||||
sr2 = &hasherSearchResult{}
|
||||
}
|
||||
sr, _ := hasherSearchResultPool.Get().(*hasherSearchResult)
|
||||
if sr == nil {
|
||||
sr = &hasherSearchResult{}
|
||||
}
|
||||
|
||||
for position+hasher.HashTypeLength() < pos_end {
|
||||
var max_length uint = pos_end - position
|
||||
var max_distance uint = brotli_min_size_t(position, max_backward_limit)
|
||||
sr.len = 0
|
||||
sr.len_code_delta = 0
|
||||
sr.distance = 0
|
||||
sr.score = kMinScore
|
||||
hasher.FindLongestMatch(¶ms.dictionary, ringbuffer, ringbuffer_mask, dist_cache, position, max_length, max_distance, gap, params.dist.max_distance, sr)
|
||||
if sr.score > kMinScore {
|
||||
/* Found a match. Let's look for something even better ahead. */
|
||||
var delayed_backward_references_in_row int = 0
|
||||
max_length--
|
||||
for ; ; max_length-- {
|
||||
var cost_diff_lazy uint = 175
|
||||
if params.quality < minQualityForExtensiveReferenceSearch {
|
||||
sr2.len = brotli_min_size_t(sr.len-1, max_length)
|
||||
} else {
|
||||
sr2.len = 0
|
||||
}
|
||||
sr2.len_code_delta = 0
|
||||
sr2.distance = 0
|
||||
sr2.score = kMinScore
|
||||
max_distance = brotli_min_size_t(position+1, max_backward_limit)
|
||||
hasher.FindLongestMatch(¶ms.dictionary, ringbuffer, ringbuffer_mask, dist_cache, position+1, max_length, max_distance, gap, params.dist.max_distance, sr2)
|
||||
if sr2.score >= sr.score+cost_diff_lazy {
|
||||
/* Ok, let's just write one byte for now and start a match from the
|
||||
next byte. */
|
||||
position++
|
||||
|
||||
insert_length++
|
||||
*sr = *sr2
|
||||
delayed_backward_references_in_row++
|
||||
if delayed_backward_references_in_row < 4 && position+hasher.HashTypeLength() < pos_end {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
break
|
||||
}
|
||||
|
||||
apply_random_heuristics = position + 2*sr.len + random_heuristics_window_size
|
||||
max_distance = brotli_min_size_t(position, max_backward_limit)
|
||||
{
|
||||
/* The first 16 codes are special short-codes,
|
||||
and the minimum offset is 1. */
|
||||
var distance_code uint = computeDistanceCode(sr.distance, max_distance+gap, dist_cache)
|
||||
if (sr.distance <= (max_distance + gap)) && distance_code > 0 {
|
||||
dist_cache[3] = dist_cache[2]
|
||||
dist_cache[2] = dist_cache[1]
|
||||
dist_cache[1] = dist_cache[0]
|
||||
dist_cache[0] = int(sr.distance)
|
||||
hasher.PrepareDistanceCache(dist_cache)
|
||||
}
|
||||
|
||||
*commands = append(*commands, makeCommand(¶ms.dist, insert_length, sr.len, sr.len_code_delta, distance_code))
|
||||
}
|
||||
|
||||
*num_literals += insert_length
|
||||
insert_length = 0
|
||||
/* Put the hash keys into the table, if there are enough bytes left.
|
||||
Depending on the hasher implementation, it can push all positions
|
||||
in the given range or only a subset of them.
|
||||
Avoid hash poisoning with RLE data. */
|
||||
{
|
||||
var range_start uint = position + 2
|
||||
var range_end uint = brotli_min_size_t(position+sr.len, store_end)
|
||||
if sr.distance < sr.len>>2 {
|
||||
range_start = brotli_min_size_t(range_end, brotli_max_size_t(range_start, position+sr.len-(sr.distance<<2)))
|
||||
}
|
||||
|
||||
hasher.StoreRange(ringbuffer, ringbuffer_mask, range_start, range_end)
|
||||
}
|
||||
|
||||
position += sr.len
|
||||
} else {
|
||||
insert_length++
|
||||
position++
|
||||
|
||||
/* If we have not seen matches for a long time, we can skip some
|
||||
match lookups. Unsuccessful match lookups are very very expensive
|
||||
and this kind of a heuristic speeds up compression quite
|
||||
a lot. */
|
||||
if position > apply_random_heuristics {
|
||||
/* Going through uncompressible data, jump. */
|
||||
if position > apply_random_heuristics+4*random_heuristics_window_size {
|
||||
var kMargin uint = brotli_max_size_t(hasher.StoreLookahead()-1, 4)
|
||||
/* It is quite a long time since we saw a copy, so we assume
|
||||
that this data is not compressible, and store hashes less
|
||||
often. Hashes of non compressible data are less likely to
|
||||
turn out to be useful in the future, too, so we store less of
|
||||
them to not to flood out the hash table of good compressible
|
||||
data. */
|
||||
|
||||
var pos_jump uint = brotli_min_size_t(position+16, pos_end-kMargin)
|
||||
for ; position < pos_jump; position += 4 {
|
||||
hasher.Store(ringbuffer, ringbuffer_mask, position)
|
||||
insert_length += 4
|
||||
}
|
||||
} else {
|
||||
var kMargin uint = brotli_max_size_t(hasher.StoreLookahead()-1, 2)
|
||||
var pos_jump uint = brotli_min_size_t(position+8, pos_end-kMargin)
|
||||
for ; position < pos_jump; position += 2 {
|
||||
hasher.Store(ringbuffer, ringbuffer_mask, position)
|
||||
insert_length += 2
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
insert_length += pos_end - position
|
||||
*last_insert_len = insert_length
|
||||
|
||||
hasherSearchResultPool.Put(sr)
|
||||
hasherSearchResultPool.Put(sr2)
|
||||
}
|
||||
+796
@@ -0,0 +1,796 @@
|
||||
package brotli
|
||||
|
||||
import "math"
|
||||
|
||||
type zopfliNode struct {
|
||||
length uint32
|
||||
distance uint32
|
||||
dcode_insert_length uint32
|
||||
u struct {
|
||||
cost float32
|
||||
next uint32
|
||||
shortcut uint32
|
||||
}
|
||||
}
|
||||
|
||||
const maxEffectiveDistanceAlphabetSize = 544
|
||||
|
||||
const kInfinity float32 = 1.7e38 /* ~= 2 ^ 127 */
|
||||
|
||||
var kDistanceCacheIndex = []uint32{0, 1, 2, 3, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1}
|
||||
|
||||
var kDistanceCacheOffset = []int{0, 0, 0, 0, -1, 1, -2, 2, -3, 3, -1, 1, -2, 2, -3, 3}
|
||||
|
||||
func initZopfliNodes(array []zopfliNode, length uint) {
|
||||
var stub zopfliNode
|
||||
var i uint
|
||||
stub.length = 1
|
||||
stub.distance = 0
|
||||
stub.dcode_insert_length = 0
|
||||
stub.u.cost = kInfinity
|
||||
for i = 0; i < length; i++ {
|
||||
array[i] = stub
|
||||
}
|
||||
}
|
||||
|
||||
func zopfliNodeCopyLength(self *zopfliNode) uint32 {
|
||||
return self.length & 0x1FFFFFF
|
||||
}
|
||||
|
||||
func zopfliNodeLengthCode(self *zopfliNode) uint32 {
|
||||
var modifier uint32 = self.length >> 25
|
||||
return zopfliNodeCopyLength(self) + 9 - modifier
|
||||
}
|
||||
|
||||
func zopfliNodeCopyDistance(self *zopfliNode) uint32 {
|
||||
return self.distance
|
||||
}
|
||||
|
||||
func zopfliNodeDistanceCode(self *zopfliNode) uint32 {
|
||||
var short_code uint32 = self.dcode_insert_length >> 27
|
||||
if short_code == 0 {
|
||||
return zopfliNodeCopyDistance(self) + numDistanceShortCodes - 1
|
||||
} else {
|
||||
return short_code - 1
|
||||
}
|
||||
}
|
||||
|
||||
func zopfliNodeCommandLength(self *zopfliNode) uint32 {
|
||||
return zopfliNodeCopyLength(self) + (self.dcode_insert_length & 0x7FFFFFF)
|
||||
}
|
||||
|
||||
/* Histogram based cost model for zopflification. */
|
||||
type zopfliCostModel struct {
|
||||
cost_cmd_ [numCommandSymbols]float32
|
||||
cost_dist_ []float32
|
||||
distance_histogram_size uint32
|
||||
literal_costs_ []float32
|
||||
min_cost_cmd_ float32
|
||||
num_bytes_ uint
|
||||
}
|
||||
|
||||
func initZopfliCostModel(self *zopfliCostModel, dist *distanceParams, num_bytes uint) {
|
||||
var distance_histogram_size uint32 = dist.alphabet_size
|
||||
if distance_histogram_size > maxEffectiveDistanceAlphabetSize {
|
||||
distance_histogram_size = maxEffectiveDistanceAlphabetSize
|
||||
}
|
||||
|
||||
self.num_bytes_ = num_bytes
|
||||
self.literal_costs_ = make([]float32, (num_bytes + 2))
|
||||
self.cost_dist_ = make([]float32, (dist.alphabet_size))
|
||||
self.distance_histogram_size = distance_histogram_size
|
||||
}
|
||||
|
||||
func cleanupZopfliCostModel(self *zopfliCostModel) {
|
||||
self.literal_costs_ = nil
|
||||
self.cost_dist_ = nil
|
||||
}
|
||||
|
||||
func setCost(histogram []uint32, histogram_size uint, literal_histogram bool, cost []float32) {
|
||||
var sum uint = 0
|
||||
var missing_symbol_sum uint
|
||||
var log2sum float32
|
||||
var missing_symbol_cost float32
|
||||
var i uint
|
||||
for i = 0; i < histogram_size; i++ {
|
||||
sum += uint(histogram[i])
|
||||
}
|
||||
|
||||
log2sum = float32(fastLog2(sum))
|
||||
missing_symbol_sum = sum
|
||||
if !literal_histogram {
|
||||
for i = 0; i < histogram_size; i++ {
|
||||
if histogram[i] == 0 {
|
||||
missing_symbol_sum++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
missing_symbol_cost = float32(fastLog2(missing_symbol_sum)) + 2
|
||||
for i = 0; i < histogram_size; i++ {
|
||||
if histogram[i] == 0 {
|
||||
cost[i] = missing_symbol_cost
|
||||
continue
|
||||
}
|
||||
|
||||
/* Shannon bits for this symbol. */
|
||||
cost[i] = log2sum - float32(fastLog2(uint(histogram[i])))
|
||||
|
||||
/* Cannot be coded with less than 1 bit */
|
||||
if cost[i] < 1 {
|
||||
cost[i] = 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func zopfliCostModelSetFromCommands(self *zopfliCostModel, position uint, ringbuffer []byte, ringbuffer_mask uint, commands []command, last_insert_len uint) {
|
||||
var histogram_literal [numLiteralSymbols]uint32
|
||||
var histogram_cmd [numCommandSymbols]uint32
|
||||
var histogram_dist [maxEffectiveDistanceAlphabetSize]uint32
|
||||
var cost_literal [numLiteralSymbols]float32
|
||||
var pos uint = position - last_insert_len
|
||||
var min_cost_cmd float32 = kInfinity
|
||||
var cost_cmd []float32 = self.cost_cmd_[:]
|
||||
var literal_costs []float32
|
||||
|
||||
histogram_literal = [numLiteralSymbols]uint32{}
|
||||
histogram_cmd = [numCommandSymbols]uint32{}
|
||||
histogram_dist = [maxEffectiveDistanceAlphabetSize]uint32{}
|
||||
|
||||
for i := range commands {
|
||||
var inslength uint = uint(commands[i].insert_len_)
|
||||
var copylength uint = uint(commandCopyLen(&commands[i]))
|
||||
var distcode uint = uint(commands[i].dist_prefix_) & 0x3FF
|
||||
var cmdcode uint = uint(commands[i].cmd_prefix_)
|
||||
var j uint
|
||||
|
||||
histogram_cmd[cmdcode]++
|
||||
if cmdcode >= 128 {
|
||||
histogram_dist[distcode]++
|
||||
}
|
||||
|
||||
for j = 0; j < inslength; j++ {
|
||||
histogram_literal[ringbuffer[(pos+j)&ringbuffer_mask]]++
|
||||
}
|
||||
|
||||
pos += inslength + copylength
|
||||
}
|
||||
|
||||
setCost(histogram_literal[:], numLiteralSymbols, true, cost_literal[:])
|
||||
setCost(histogram_cmd[:], numCommandSymbols, false, cost_cmd)
|
||||
setCost(histogram_dist[:], uint(self.distance_histogram_size), false, self.cost_dist_)
|
||||
|
||||
for i := 0; i < numCommandSymbols; i++ {
|
||||
min_cost_cmd = brotli_min_float(min_cost_cmd, cost_cmd[i])
|
||||
}
|
||||
|
||||
self.min_cost_cmd_ = min_cost_cmd
|
||||
{
|
||||
literal_costs = self.literal_costs_
|
||||
var literal_carry float32 = 0.0
|
||||
num_bytes := int(self.num_bytes_)
|
||||
literal_costs[0] = 0.0
|
||||
for i := 0; i < num_bytes; i++ {
|
||||
literal_carry += cost_literal[ringbuffer[(position+uint(i))&ringbuffer_mask]]
|
||||
literal_costs[i+1] = literal_costs[i] + literal_carry
|
||||
literal_carry -= literal_costs[i+1] - literal_costs[i]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func zopfliCostModelSetFromLiteralCosts(self *zopfliCostModel, position uint, ringbuffer []byte, ringbuffer_mask uint) {
|
||||
var literal_costs []float32 = self.literal_costs_
|
||||
var literal_carry float32 = 0.0
|
||||
var cost_dist []float32 = self.cost_dist_
|
||||
var cost_cmd []float32 = self.cost_cmd_[:]
|
||||
var num_bytes uint = self.num_bytes_
|
||||
var i uint
|
||||
estimateBitCostsForLiterals(position, num_bytes, ringbuffer_mask, ringbuffer, literal_costs[1:])
|
||||
literal_costs[0] = 0.0
|
||||
for i = 0; i < num_bytes; i++ {
|
||||
literal_carry += literal_costs[i+1]
|
||||
literal_costs[i+1] = literal_costs[i] + literal_carry
|
||||
literal_carry -= literal_costs[i+1] - literal_costs[i]
|
||||
}
|
||||
|
||||
for i = 0; i < numCommandSymbols; i++ {
|
||||
cost_cmd[i] = float32(fastLog2(uint(11 + uint32(i))))
|
||||
}
|
||||
|
||||
for i = 0; uint32(i) < self.distance_histogram_size; i++ {
|
||||
cost_dist[i] = float32(fastLog2(uint(20 + uint32(i))))
|
||||
}
|
||||
|
||||
self.min_cost_cmd_ = float32(fastLog2(11))
|
||||
}
|
||||
|
||||
func zopfliCostModelGetCommandCost(self *zopfliCostModel, cmdcode uint16) float32 {
|
||||
return self.cost_cmd_[cmdcode]
|
||||
}
|
||||
|
||||
func zopfliCostModelGetDistanceCost(self *zopfliCostModel, distcode uint) float32 {
|
||||
return self.cost_dist_[distcode]
|
||||
}
|
||||
|
||||
func zopfliCostModelGetLiteralCosts(self *zopfliCostModel, from uint, to uint) float32 {
|
||||
return self.literal_costs_[to] - self.literal_costs_[from]
|
||||
}
|
||||
|
||||
func zopfliCostModelGetMinCostCmd(self *zopfliCostModel) float32 {
|
||||
return self.min_cost_cmd_
|
||||
}
|
||||
|
||||
/* REQUIRES: len >= 2, start_pos <= pos */
|
||||
/* REQUIRES: cost < kInfinity, nodes[start_pos].cost < kInfinity */
|
||||
/* Maintains the "ZopfliNode array invariant". */
|
||||
func updateZopfliNode(nodes []zopfliNode, pos uint, start_pos uint, len uint, len_code uint, dist uint, short_code uint, cost float32) {
|
||||
var next *zopfliNode = &nodes[pos+len]
|
||||
next.length = uint32(len | (len+9-len_code)<<25)
|
||||
next.distance = uint32(dist)
|
||||
next.dcode_insert_length = uint32(short_code<<27 | (pos - start_pos))
|
||||
next.u.cost = cost
|
||||
}
|
||||
|
||||
type posData struct {
|
||||
pos uint
|
||||
distance_cache [4]int
|
||||
costdiff float32
|
||||
cost float32
|
||||
}
|
||||
|
||||
/* Maintains the smallest 8 cost difference together with their positions */
|
||||
type startPosQueue struct {
|
||||
q_ [8]posData
|
||||
idx_ uint
|
||||
}
|
||||
|
||||
func initStartPosQueue(self *startPosQueue) {
|
||||
self.idx_ = 0
|
||||
}
|
||||
|
||||
func startPosQueueSize(self *startPosQueue) uint {
|
||||
return brotli_min_size_t(self.idx_, 8)
|
||||
}
|
||||
|
||||
func startPosQueuePush(self *startPosQueue, posdata *posData) {
|
||||
var offset uint = ^(self.idx_) & 7
|
||||
self.idx_++
|
||||
var len uint = startPosQueueSize(self)
|
||||
var i uint
|
||||
var q []posData = self.q_[:]
|
||||
q[offset] = *posdata
|
||||
|
||||
/* Restore the sorted order. In the list of |len| items at most |len - 1|
|
||||
adjacent element comparisons / swaps are required. */
|
||||
for i = 1; i < len; i++ {
|
||||
if q[offset&7].costdiff > q[(offset+1)&7].costdiff {
|
||||
var tmp posData = q[offset&7]
|
||||
q[offset&7] = q[(offset+1)&7]
|
||||
q[(offset+1)&7] = tmp
|
||||
}
|
||||
|
||||
offset++
|
||||
}
|
||||
}
|
||||
|
||||
func startPosQueueAt(self *startPosQueue, k uint) *posData {
|
||||
return &self.q_[(k-self.idx_)&7]
|
||||
}
|
||||
|
||||
/* Returns the minimum possible copy length that can improve the cost of any */
|
||||
/* future position. */
|
||||
func computeMinimumCopyLength(start_cost float32, nodes []zopfliNode, num_bytes uint, pos uint) uint {
|
||||
var min_cost float32 = start_cost
|
||||
var len uint = 2
|
||||
var next_len_bucket uint = 4
|
||||
/* Compute the minimum possible cost of reaching any future position. */
|
||||
|
||||
var next_len_offset uint = 10
|
||||
for pos+len <= num_bytes && nodes[pos+len].u.cost <= min_cost {
|
||||
/* We already reached (pos + len) with no more cost than the minimum
|
||||
possible cost of reaching anything from this pos, so there is no point in
|
||||
looking for lengths <= len. */
|
||||
len++
|
||||
|
||||
if len == next_len_offset {
|
||||
/* We reached the next copy length code bucket, so we add one more
|
||||
extra bit to the minimum cost. */
|
||||
min_cost += 1.0
|
||||
|
||||
next_len_offset += next_len_bucket
|
||||
next_len_bucket *= 2
|
||||
}
|
||||
}
|
||||
|
||||
return uint(len)
|
||||
}
|
||||
|
||||
/* REQUIRES: nodes[pos].cost < kInfinity
|
||||
REQUIRES: nodes[0..pos] satisfies that "ZopfliNode array invariant". */
|
||||
func computeDistanceShortcut(block_start uint, pos uint, max_backward_limit uint, gap uint, nodes []zopfliNode) uint32 {
|
||||
var clen uint = uint(zopfliNodeCopyLength(&nodes[pos]))
|
||||
var ilen uint = uint(nodes[pos].dcode_insert_length & 0x7FFFFFF)
|
||||
var dist uint = uint(zopfliNodeCopyDistance(&nodes[pos]))
|
||||
|
||||
/* Since |block_start + pos| is the end position of the command, the copy part
|
||||
starts from |block_start + pos - clen|. Distances that are greater than
|
||||
this or greater than |max_backward_limit| + |gap| are static dictionary
|
||||
references, and do not update the last distances.
|
||||
Also distance code 0 (last distance) does not update the last distances. */
|
||||
if pos == 0 {
|
||||
return 0
|
||||
} else if dist+clen <= block_start+pos+gap && dist <= max_backward_limit+gap && zopfliNodeDistanceCode(&nodes[pos]) > 0 {
|
||||
return uint32(pos)
|
||||
} else {
|
||||
return nodes[pos-clen-ilen].u.shortcut
|
||||
}
|
||||
}
|
||||
|
||||
/* Fills in dist_cache[0..3] with the last four distances (as defined by
|
||||
Section 4. of the Spec) that would be used at (block_start + pos) if we
|
||||
used the shortest path of commands from block_start, computed from
|
||||
nodes[0..pos]. The last four distances at block_start are in
|
||||
starting_dist_cache[0..3].
|
||||
REQUIRES: nodes[pos].cost < kInfinity
|
||||
REQUIRES: nodes[0..pos] satisfies that "ZopfliNode array invariant". */
|
||||
func computeDistanceCache(pos uint, starting_dist_cache []int, nodes []zopfliNode, dist_cache []int) {
|
||||
var idx int = 0
|
||||
var p uint = uint(nodes[pos].u.shortcut)
|
||||
for idx < 4 && p > 0 {
|
||||
var ilen uint = uint(nodes[p].dcode_insert_length & 0x7FFFFFF)
|
||||
var clen uint = uint(zopfliNodeCopyLength(&nodes[p]))
|
||||
var dist uint = uint(zopfliNodeCopyDistance(&nodes[p]))
|
||||
dist_cache[idx] = int(dist)
|
||||
idx++
|
||||
|
||||
/* Because of prerequisite, p >= clen + ilen >= 2. */
|
||||
p = uint(nodes[p-clen-ilen].u.shortcut)
|
||||
}
|
||||
|
||||
for ; idx < 4; idx++ {
|
||||
dist_cache[idx] = starting_dist_cache[0]
|
||||
starting_dist_cache = starting_dist_cache[1:]
|
||||
}
|
||||
}
|
||||
|
||||
/* Maintains "ZopfliNode array invariant" and pushes node to the queue, if it
|
||||
is eligible. */
|
||||
func evaluateNode(block_start uint, pos uint, max_backward_limit uint, gap uint, starting_dist_cache []int, model *zopfliCostModel, queue *startPosQueue, nodes []zopfliNode) {
|
||||
/* Save cost, because ComputeDistanceCache invalidates it. */
|
||||
var node_cost float32 = nodes[pos].u.cost
|
||||
nodes[pos].u.shortcut = computeDistanceShortcut(block_start, pos, max_backward_limit, gap, nodes)
|
||||
if node_cost <= zopfliCostModelGetLiteralCosts(model, 0, pos) {
|
||||
var posdata posData
|
||||
posdata.pos = pos
|
||||
posdata.cost = node_cost
|
||||
posdata.costdiff = node_cost - zopfliCostModelGetLiteralCosts(model, 0, pos)
|
||||
computeDistanceCache(pos, starting_dist_cache, nodes, posdata.distance_cache[:])
|
||||
startPosQueuePush(queue, &posdata)
|
||||
}
|
||||
}
|
||||
|
||||
/* Returns longest copy length. */
|
||||
func updateNodes(num_bytes uint, block_start uint, pos uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, max_backward_limit uint, starting_dist_cache []int, num_matches uint, matches []backwardMatch, model *zopfliCostModel, queue *startPosQueue, nodes []zopfliNode) uint {
|
||||
var cur_ix uint = block_start + pos
|
||||
var cur_ix_masked uint = cur_ix & ringbuffer_mask
|
||||
var max_distance uint = brotli_min_size_t(cur_ix, max_backward_limit)
|
||||
var max_len uint = num_bytes - pos
|
||||
var max_zopfli_len uint = maxZopfliLen(params)
|
||||
var max_iters uint = maxZopfliCandidates(params)
|
||||
var min_len uint
|
||||
var result uint = 0
|
||||
var k uint
|
||||
var gap uint = 0
|
||||
|
||||
evaluateNode(block_start, pos, max_backward_limit, gap, starting_dist_cache, model, queue, nodes)
|
||||
{
|
||||
var posdata *posData = startPosQueueAt(queue, 0)
|
||||
var min_cost float32 = (posdata.cost + zopfliCostModelGetMinCostCmd(model) + zopfliCostModelGetLiteralCosts(model, posdata.pos, pos))
|
||||
min_len = computeMinimumCopyLength(min_cost, nodes, num_bytes, pos)
|
||||
}
|
||||
|
||||
/* Go over the command starting positions in order of increasing cost
|
||||
difference. */
|
||||
for k = 0; k < max_iters && k < startPosQueueSize(queue); k++ {
|
||||
var posdata *posData = startPosQueueAt(queue, k)
|
||||
var start uint = posdata.pos
|
||||
var inscode uint16 = getInsertLengthCode(pos - start)
|
||||
var start_costdiff float32 = posdata.costdiff
|
||||
var base_cost float32 = start_costdiff + float32(getInsertExtra(inscode)) + zopfliCostModelGetLiteralCosts(model, 0, pos)
|
||||
var best_len uint = min_len - 1
|
||||
var j uint = 0
|
||||
/* Look for last distance matches using the distance cache from this
|
||||
starting position. */
|
||||
for ; j < numDistanceShortCodes && best_len < max_len; j++ {
|
||||
var idx uint = uint(kDistanceCacheIndex[j])
|
||||
var backward uint = uint(posdata.distance_cache[idx] + kDistanceCacheOffset[j])
|
||||
var prev_ix uint = cur_ix - backward
|
||||
var len uint = 0
|
||||
var continuation byte = ringbuffer[cur_ix_masked+best_len]
|
||||
if cur_ix_masked+best_len > ringbuffer_mask {
|
||||
break
|
||||
}
|
||||
|
||||
if backward > max_distance+gap {
|
||||
/* Word dictionary -> ignore. */
|
||||
continue
|
||||
}
|
||||
|
||||
if backward <= max_distance {
|
||||
/* Regular backward reference. */
|
||||
if prev_ix >= cur_ix {
|
||||
continue
|
||||
}
|
||||
|
||||
prev_ix &= ringbuffer_mask
|
||||
if prev_ix+best_len > ringbuffer_mask || continuation != ringbuffer[prev_ix+best_len] {
|
||||
continue
|
||||
}
|
||||
|
||||
len = findMatchLengthWithLimit(ringbuffer[prev_ix:], ringbuffer[cur_ix_masked:], max_len)
|
||||
} else {
|
||||
continue
|
||||
}
|
||||
{
|
||||
var dist_cost float32 = base_cost + zopfliCostModelGetDistanceCost(model, j)
|
||||
var l uint
|
||||
for l = best_len + 1; l <= len; l++ {
|
||||
var copycode uint16 = getCopyLengthCode(l)
|
||||
var cmdcode uint16 = combineLengthCodes(inscode, copycode, j == 0)
|
||||
var tmp float32
|
||||
if cmdcode < 128 {
|
||||
tmp = base_cost
|
||||
} else {
|
||||
tmp = dist_cost
|
||||
}
|
||||
var cost float32 = tmp + float32(getCopyExtra(copycode)) + zopfliCostModelGetCommandCost(model, cmdcode)
|
||||
if cost < nodes[pos+l].u.cost {
|
||||
updateZopfliNode(nodes, pos, start, l, l, backward, j+1, cost)
|
||||
result = brotli_max_size_t(result, l)
|
||||
}
|
||||
|
||||
best_len = l
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* At higher iterations look only for new last distance matches, since
|
||||
looking only for new command start positions with the same distances
|
||||
does not help much. */
|
||||
if k >= 2 {
|
||||
continue
|
||||
}
|
||||
{
|
||||
/* Loop through all possible copy lengths at this position. */
|
||||
var len uint = min_len
|
||||
for j = 0; j < num_matches; j++ {
|
||||
var match backwardMatch = matches[j]
|
||||
var dist uint = uint(match.distance)
|
||||
var is_dictionary_match bool = (dist > max_distance+gap)
|
||||
var dist_code uint = dist + numDistanceShortCodes - 1
|
||||
var dist_symbol uint16
|
||||
var distextra uint32
|
||||
var distnumextra uint32
|
||||
var dist_cost float32
|
||||
var max_match_len uint
|
||||
/* We already tried all possible last distance matches, so we can use
|
||||
normal distance code here. */
|
||||
prefixEncodeCopyDistance(dist_code, uint(params.dist.num_direct_distance_codes), uint(params.dist.distance_postfix_bits), &dist_symbol, &distextra)
|
||||
|
||||
distnumextra = uint32(dist_symbol) >> 10
|
||||
dist_cost = base_cost + float32(distnumextra) + zopfliCostModelGetDistanceCost(model, uint(dist_symbol)&0x3FF)
|
||||
|
||||
/* Try all copy lengths up until the maximum copy length corresponding
|
||||
to this distance. If the distance refers to the static dictionary, or
|
||||
the maximum length is long enough, try only one maximum length. */
|
||||
max_match_len = backwardMatchLength(&match)
|
||||
|
||||
if len < max_match_len && (is_dictionary_match || max_match_len > max_zopfli_len) {
|
||||
len = max_match_len
|
||||
}
|
||||
|
||||
for ; len <= max_match_len; len++ {
|
||||
var len_code uint
|
||||
if is_dictionary_match {
|
||||
len_code = backwardMatchLengthCode(&match)
|
||||
} else {
|
||||
len_code = len
|
||||
}
|
||||
var copycode uint16 = getCopyLengthCode(len_code)
|
||||
var cmdcode uint16 = combineLengthCodes(inscode, copycode, false)
|
||||
var cost float32 = dist_cost + float32(getCopyExtra(copycode)) + zopfliCostModelGetCommandCost(model, cmdcode)
|
||||
if cost < nodes[pos+len].u.cost {
|
||||
updateZopfliNode(nodes, pos, start, uint(len), len_code, dist, 0, cost)
|
||||
if len > result {
|
||||
result = len
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
func computeShortestPathFromNodes(num_bytes uint, nodes []zopfliNode) uint {
|
||||
var index uint = num_bytes
|
||||
var num_commands uint = 0
|
||||
for nodes[index].dcode_insert_length&0x7FFFFFF == 0 && nodes[index].length == 1 {
|
||||
index--
|
||||
}
|
||||
nodes[index].u.next = math.MaxUint32
|
||||
for index != 0 {
|
||||
var len uint = uint(zopfliNodeCommandLength(&nodes[index]))
|
||||
index -= uint(len)
|
||||
nodes[index].u.next = uint32(len)
|
||||
num_commands++
|
||||
}
|
||||
|
||||
return num_commands
|
||||
}
|
||||
|
||||
/* REQUIRES: nodes != NULL and len(nodes) >= num_bytes + 1 */
|
||||
func zopfliCreateCommands(num_bytes uint, block_start uint, nodes []zopfliNode, dist_cache []int, last_insert_len *uint, params *encoderParams, commands *[]command, num_literals *uint) {
|
||||
var max_backward_limit uint = maxBackwardLimit(params.lgwin)
|
||||
var pos uint = 0
|
||||
var offset uint32 = nodes[0].u.next
|
||||
var i uint
|
||||
var gap uint = 0
|
||||
for i = 0; offset != math.MaxUint32; i++ {
|
||||
var next *zopfliNode = &nodes[uint32(pos)+offset]
|
||||
var copy_length uint = uint(zopfliNodeCopyLength(next))
|
||||
var insert_length uint = uint(next.dcode_insert_length & 0x7FFFFFF)
|
||||
pos += insert_length
|
||||
offset = next.u.next
|
||||
if i == 0 {
|
||||
insert_length += *last_insert_len
|
||||
*last_insert_len = 0
|
||||
}
|
||||
{
|
||||
var distance uint = uint(zopfliNodeCopyDistance(next))
|
||||
var len_code uint = uint(zopfliNodeLengthCode(next))
|
||||
var max_distance uint = brotli_min_size_t(block_start+pos, max_backward_limit)
|
||||
var is_dictionary bool = (distance > max_distance+gap)
|
||||
var dist_code uint = uint(zopfliNodeDistanceCode(next))
|
||||
*commands = append(*commands, makeCommand(¶ms.dist, insert_length, copy_length, int(len_code)-int(copy_length), dist_code))
|
||||
|
||||
if !is_dictionary && dist_code > 0 {
|
||||
dist_cache[3] = dist_cache[2]
|
||||
dist_cache[2] = dist_cache[1]
|
||||
dist_cache[1] = dist_cache[0]
|
||||
dist_cache[0] = int(distance)
|
||||
}
|
||||
}
|
||||
|
||||
*num_literals += insert_length
|
||||
pos += copy_length
|
||||
}
|
||||
|
||||
*last_insert_len += num_bytes - pos
|
||||
}
|
||||
|
||||
func zopfliIterate(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, gap uint, dist_cache []int, model *zopfliCostModel, num_matches []uint32, matches []backwardMatch, nodes []zopfliNode) uint {
|
||||
var max_backward_limit uint = maxBackwardLimit(params.lgwin)
|
||||
var max_zopfli_len uint = maxZopfliLen(params)
|
||||
var queue startPosQueue
|
||||
var cur_match_pos uint = 0
|
||||
var i uint
|
||||
nodes[0].length = 0
|
||||
nodes[0].u.cost = 0
|
||||
initStartPosQueue(&queue)
|
||||
for i = 0; i+3 < num_bytes; i++ {
|
||||
var skip uint = updateNodes(num_bytes, position, i, ringbuffer, ringbuffer_mask, params, max_backward_limit, dist_cache, uint(num_matches[i]), matches[cur_match_pos:], model, &queue, nodes)
|
||||
if skip < longCopyQuickStep {
|
||||
skip = 0
|
||||
}
|
||||
cur_match_pos += uint(num_matches[i])
|
||||
if num_matches[i] == 1 && backwardMatchLength(&matches[cur_match_pos-1]) > max_zopfli_len {
|
||||
skip = brotli_max_size_t(backwardMatchLength(&matches[cur_match_pos-1]), skip)
|
||||
}
|
||||
|
||||
if skip > 1 {
|
||||
skip--
|
||||
for skip != 0 {
|
||||
i++
|
||||
if i+3 >= num_bytes {
|
||||
break
|
||||
}
|
||||
evaluateNode(position, i, max_backward_limit, gap, dist_cache, model, &queue, nodes)
|
||||
cur_match_pos += uint(num_matches[i])
|
||||
skip--
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return computeShortestPathFromNodes(num_bytes, nodes)
|
||||
}
|
||||
|
||||
/* Computes the shortest path of commands from position to at most
|
||||
position + num_bytes.
|
||||
|
||||
On return, path->size() is the number of commands found and path[i] is the
|
||||
length of the i-th command (copy length plus insert length).
|
||||
Note that the sum of the lengths of all commands can be less than num_bytes.
|
||||
|
||||
On return, the nodes[0..num_bytes] array will have the following
|
||||
"ZopfliNode array invariant":
|
||||
For each i in [1..num_bytes], if nodes[i].cost < kInfinity, then
|
||||
(1) nodes[i].copy_length() >= 2
|
||||
(2) nodes[i].command_length() <= i and
|
||||
(3) nodes[i - nodes[i].command_length()].cost < kInfinity
|
||||
|
||||
REQUIRES: nodes != nil and len(nodes) >= num_bytes + 1 */
|
||||
func zopfliComputeShortestPath(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, dist_cache []int, hasher *h10, nodes []zopfliNode) uint {
|
||||
var max_backward_limit uint = maxBackwardLimit(params.lgwin)
|
||||
var max_zopfli_len uint = maxZopfliLen(params)
|
||||
var model zopfliCostModel
|
||||
var queue startPosQueue
|
||||
var matches [2 * (maxNumMatchesH10 + 64)]backwardMatch
|
||||
var store_end uint
|
||||
if num_bytes >= hasher.StoreLookahead() {
|
||||
store_end = position + num_bytes - hasher.StoreLookahead() + 1
|
||||
} else {
|
||||
store_end = position
|
||||
}
|
||||
var i uint
|
||||
var gap uint = 0
|
||||
var lz_matches_offset uint = 0
|
||||
nodes[0].length = 0
|
||||
nodes[0].u.cost = 0
|
||||
initZopfliCostModel(&model, ¶ms.dist, num_bytes)
|
||||
zopfliCostModelSetFromLiteralCosts(&model, position, ringbuffer, ringbuffer_mask)
|
||||
initStartPosQueue(&queue)
|
||||
for i = 0; i+hasher.HashTypeLength()-1 < num_bytes; i++ {
|
||||
var pos uint = position + i
|
||||
var max_distance uint = brotli_min_size_t(pos, max_backward_limit)
|
||||
var skip uint
|
||||
var num_matches uint
|
||||
num_matches = findAllMatchesH10(hasher, ¶ms.dictionary, ringbuffer, ringbuffer_mask, pos, num_bytes-i, max_distance, gap, params, matches[lz_matches_offset:])
|
||||
if num_matches > 0 && backwardMatchLength(&matches[num_matches-1]) > max_zopfli_len {
|
||||
matches[0] = matches[num_matches-1]
|
||||
num_matches = 1
|
||||
}
|
||||
|
||||
skip = updateNodes(num_bytes, position, i, ringbuffer, ringbuffer_mask, params, max_backward_limit, dist_cache, num_matches, matches[:], &model, &queue, nodes)
|
||||
if skip < longCopyQuickStep {
|
||||
skip = 0
|
||||
}
|
||||
if num_matches == 1 && backwardMatchLength(&matches[0]) > max_zopfli_len {
|
||||
skip = brotli_max_size_t(backwardMatchLength(&matches[0]), skip)
|
||||
}
|
||||
|
||||
if skip > 1 {
|
||||
/* Add the tail of the copy to the hasher. */
|
||||
hasher.StoreRange(ringbuffer, ringbuffer_mask, pos+1, brotli_min_size_t(pos+skip, store_end))
|
||||
|
||||
skip--
|
||||
for skip != 0 {
|
||||
i++
|
||||
if i+hasher.HashTypeLength()-1 >= num_bytes {
|
||||
break
|
||||
}
|
||||
evaluateNode(position, i, max_backward_limit, gap, dist_cache, &model, &queue, nodes)
|
||||
skip--
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
cleanupZopfliCostModel(&model)
|
||||
return computeShortestPathFromNodes(num_bytes, nodes)
|
||||
}
|
||||
|
||||
func createZopfliBackwardReferences(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, hasher *h10, dist_cache []int, last_insert_len *uint, commands *[]command, num_literals *uint) {
|
||||
var nodes []zopfliNode
|
||||
nodes = make([]zopfliNode, (num_bytes + 1))
|
||||
initZopfliNodes(nodes, num_bytes+1)
|
||||
zopfliComputeShortestPath(num_bytes, position, ringbuffer, ringbuffer_mask, params, dist_cache, hasher, nodes)
|
||||
zopfliCreateCommands(num_bytes, position, nodes, dist_cache, last_insert_len, params, commands, num_literals)
|
||||
nodes = nil
|
||||
}
|
||||
|
||||
func createHqZopfliBackwardReferences(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint, params *encoderParams, hasher hasherHandle, dist_cache []int, last_insert_len *uint, commands *[]command, num_literals *uint) {
|
||||
var max_backward_limit uint = maxBackwardLimit(params.lgwin)
|
||||
var num_matches []uint32 = make([]uint32, num_bytes)
|
||||
var matches_size uint = 4 * num_bytes
|
||||
var store_end uint
|
||||
if num_bytes >= hasher.StoreLookahead() {
|
||||
store_end = position + num_bytes - hasher.StoreLookahead() + 1
|
||||
} else {
|
||||
store_end = position
|
||||
}
|
||||
var cur_match_pos uint = 0
|
||||
var i uint
|
||||
var orig_num_literals uint
|
||||
var orig_last_insert_len uint
|
||||
var orig_dist_cache [4]int
|
||||
var orig_num_commands int
|
||||
var model zopfliCostModel
|
||||
var nodes []zopfliNode
|
||||
var matches []backwardMatch = make([]backwardMatch, matches_size)
|
||||
var gap uint = 0
|
||||
var shadow_matches uint = 0
|
||||
var new_array []backwardMatch
|
||||
for i = 0; i+hasher.HashTypeLength()-1 < num_bytes; i++ {
|
||||
var pos uint = position + i
|
||||
var max_distance uint = brotli_min_size_t(pos, max_backward_limit)
|
||||
var max_length uint = num_bytes - i
|
||||
var num_found_matches uint
|
||||
var cur_match_end uint
|
||||
var j uint
|
||||
|
||||
/* Ensure that we have enough free slots. */
|
||||
if matches_size < cur_match_pos+maxNumMatchesH10+shadow_matches {
|
||||
var new_size uint = matches_size
|
||||
if new_size == 0 {
|
||||
new_size = cur_match_pos + maxNumMatchesH10 + shadow_matches
|
||||
}
|
||||
|
||||
for new_size < cur_match_pos+maxNumMatchesH10+shadow_matches {
|
||||
new_size *= 2
|
||||
}
|
||||
|
||||
new_array = make([]backwardMatch, new_size)
|
||||
if matches_size != 0 {
|
||||
copy(new_array, matches[:matches_size])
|
||||
}
|
||||
|
||||
matches = new_array
|
||||
matches_size = new_size
|
||||
}
|
||||
|
||||
num_found_matches = findAllMatchesH10(hasher.(*h10), ¶ms.dictionary, ringbuffer, ringbuffer_mask, pos, max_length, max_distance, gap, params, matches[cur_match_pos+shadow_matches:])
|
||||
cur_match_end = cur_match_pos + num_found_matches
|
||||
for j = cur_match_pos; j+1 < cur_match_end; j++ {
|
||||
assert(backwardMatchLength(&matches[j]) <= backwardMatchLength(&matches[j+1]))
|
||||
}
|
||||
|
||||
num_matches[i] = uint32(num_found_matches)
|
||||
if num_found_matches > 0 {
|
||||
var match_len uint = backwardMatchLength(&matches[cur_match_end-1])
|
||||
if match_len > maxZopfliLenQuality11 {
|
||||
var skip uint = match_len - 1
|
||||
matches[cur_match_pos] = matches[cur_match_end-1]
|
||||
cur_match_pos++
|
||||
num_matches[i] = 1
|
||||
|
||||
/* Add the tail of the copy to the hasher. */
|
||||
hasher.StoreRange(ringbuffer, ringbuffer_mask, pos+1, brotli_min_size_t(pos+match_len, store_end))
|
||||
var pos uint = i
|
||||
for i := 0; i < int(skip); i++ {
|
||||
num_matches[pos+1:][i] = 0
|
||||
}
|
||||
i += skip
|
||||
} else {
|
||||
cur_match_pos = cur_match_end
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
orig_num_literals = *num_literals
|
||||
orig_last_insert_len = *last_insert_len
|
||||
copy(orig_dist_cache[:], dist_cache[:4])
|
||||
orig_num_commands = len(*commands)
|
||||
nodes = make([]zopfliNode, (num_bytes + 1))
|
||||
initZopfliCostModel(&model, ¶ms.dist, num_bytes)
|
||||
for i = 0; i < 2; i++ {
|
||||
initZopfliNodes(nodes, num_bytes+1)
|
||||
if i == 0 {
|
||||
zopfliCostModelSetFromLiteralCosts(&model, position, ringbuffer, ringbuffer_mask)
|
||||
} else {
|
||||
zopfliCostModelSetFromCommands(&model, position, ringbuffer, ringbuffer_mask, (*commands)[orig_num_commands:], orig_last_insert_len)
|
||||
}
|
||||
|
||||
*commands = (*commands)[:orig_num_commands]
|
||||
*num_literals = orig_num_literals
|
||||
*last_insert_len = orig_last_insert_len
|
||||
copy(dist_cache, orig_dist_cache[:4])
|
||||
zopfliIterate(num_bytes, position, ringbuffer, ringbuffer_mask, params, gap, dist_cache, &model, num_matches, matches, nodes)
|
||||
zopfliCreateCommands(num_bytes, position, nodes, dist_cache, last_insert_len, params, commands, num_literals)
|
||||
}
|
||||
|
||||
cleanupZopfliCostModel(&model)
|
||||
nodes = nil
|
||||
matches = nil
|
||||
num_matches = nil
|
||||
}
|
||||
+436
@@ -0,0 +1,436 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Functions to estimate the bit cost of Huffman trees. */
|
||||
func shannonEntropy(population []uint32, size uint, total *uint) float64 {
|
||||
var sum uint = 0
|
||||
var retval float64 = 0
|
||||
var population_end []uint32 = population[size:]
|
||||
var p uint
|
||||
for -cap(population) < -cap(population_end) {
|
||||
p = uint(population[0])
|
||||
population = population[1:]
|
||||
sum += p
|
||||
retval -= float64(p) * fastLog2(p)
|
||||
}
|
||||
|
||||
if sum != 0 {
|
||||
retval += float64(sum) * fastLog2(sum)
|
||||
}
|
||||
*total = sum
|
||||
return retval
|
||||
}
|
||||
|
||||
func bitsEntropy(population []uint32, size uint) float64 {
|
||||
var sum uint
|
||||
var retval float64 = shannonEntropy(population, size, &sum)
|
||||
if retval < float64(sum) {
|
||||
/* At least one bit per literal is needed. */
|
||||
retval = float64(sum)
|
||||
}
|
||||
|
||||
return retval
|
||||
}
|
||||
|
||||
const kOneSymbolHistogramCost float64 = 12
|
||||
const kTwoSymbolHistogramCost float64 = 20
|
||||
const kThreeSymbolHistogramCost float64 = 28
|
||||
const kFourSymbolHistogramCost float64 = 37
|
||||
|
||||
func populationCostLiteral(histogram *histogramLiteral) float64 {
|
||||
var data_size uint = histogramDataSizeLiteral()
|
||||
var count int = 0
|
||||
var s [5]uint
|
||||
var bits float64 = 0.0
|
||||
var i uint
|
||||
if histogram.total_count_ == 0 {
|
||||
return kOneSymbolHistogramCost
|
||||
}
|
||||
|
||||
for i = 0; i < data_size; i++ {
|
||||
if histogram.data_[i] > 0 {
|
||||
s[count] = i
|
||||
count++
|
||||
if count > 4 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if count == 1 {
|
||||
return kOneSymbolHistogramCost
|
||||
}
|
||||
|
||||
if count == 2 {
|
||||
return kTwoSymbolHistogramCost + float64(histogram.total_count_)
|
||||
}
|
||||
|
||||
if count == 3 {
|
||||
var histo0 uint32 = histogram.data_[s[0]]
|
||||
var histo1 uint32 = histogram.data_[s[1]]
|
||||
var histo2 uint32 = histogram.data_[s[2]]
|
||||
var histomax uint32 = brotli_max_uint32_t(histo0, brotli_max_uint32_t(histo1, histo2))
|
||||
return kThreeSymbolHistogramCost + 2*(float64(histo0)+float64(histo1)+float64(histo2)) - float64(histomax)
|
||||
}
|
||||
|
||||
if count == 4 {
|
||||
var histo [4]uint32
|
||||
var h23 uint32
|
||||
var histomax uint32
|
||||
for i = 0; i < 4; i++ {
|
||||
histo[i] = histogram.data_[s[i]]
|
||||
}
|
||||
|
||||
/* Sort */
|
||||
for i = 0; i < 4; i++ {
|
||||
var j uint
|
||||
for j = i + 1; j < 4; j++ {
|
||||
if histo[j] > histo[i] {
|
||||
var tmp uint32 = histo[j]
|
||||
histo[j] = histo[i]
|
||||
histo[i] = tmp
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
h23 = histo[2] + histo[3]
|
||||
histomax = brotli_max_uint32_t(h23, histo[0])
|
||||
return kFourSymbolHistogramCost + 3*float64(h23) + 2*(float64(histo[0])+float64(histo[1])) - float64(histomax)
|
||||
}
|
||||
{
|
||||
var max_depth uint = 1
|
||||
var depth_histo = [codeLengthCodes]uint32{0}
|
||||
/* In this loop we compute the entropy of the histogram and simultaneously
|
||||
build a simplified histogram of the code length codes where we use the
|
||||
zero repeat code 17, but we don't use the non-zero repeat code 16. */
|
||||
|
||||
var log2total float64 = fastLog2(histogram.total_count_)
|
||||
for i = 0; i < data_size; {
|
||||
if histogram.data_[i] > 0 {
|
||||
var log2p float64 = log2total - fastLog2(uint(histogram.data_[i]))
|
||||
/* Compute -log2(P(symbol)) = -log2(count(symbol)/total_count) =
|
||||
= log2(total_count) - log2(count(symbol)) */
|
||||
|
||||
var depth uint = uint(log2p + 0.5)
|
||||
/* Approximate the bit depth by round(-log2(P(symbol))) */
|
||||
bits += float64(histogram.data_[i]) * log2p
|
||||
|
||||
if depth > 15 {
|
||||
depth = 15
|
||||
}
|
||||
|
||||
if depth > max_depth {
|
||||
max_depth = depth
|
||||
}
|
||||
|
||||
depth_histo[depth]++
|
||||
i++
|
||||
} else {
|
||||
var reps uint32 = 1
|
||||
/* Compute the run length of zeros and add the appropriate number of 0
|
||||
and 17 code length codes to the code length code histogram. */
|
||||
|
||||
var k uint
|
||||
for k = i + 1; k < data_size && histogram.data_[k] == 0; k++ {
|
||||
reps++
|
||||
}
|
||||
|
||||
i += uint(reps)
|
||||
if i == data_size {
|
||||
/* Don't add any cost for the last zero run, since these are encoded
|
||||
only implicitly. */
|
||||
break
|
||||
}
|
||||
|
||||
if reps < 3 {
|
||||
depth_histo[0] += reps
|
||||
} else {
|
||||
reps -= 2
|
||||
for reps > 0 {
|
||||
depth_histo[repeatZeroCodeLength]++
|
||||
|
||||
/* Add the 3 extra bits for the 17 code length code. */
|
||||
bits += 3
|
||||
|
||||
reps >>= 3
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Add the estimated encoding cost of the code length code histogram. */
|
||||
bits += float64(18 + 2*max_depth)
|
||||
|
||||
/* Add the entropy of the code length code histogram. */
|
||||
bits += bitsEntropy(depth_histo[:], codeLengthCodes)
|
||||
}
|
||||
|
||||
return bits
|
||||
}
|
||||
|
||||
func populationCostCommand(histogram *histogramCommand) float64 {
|
||||
var data_size uint = histogramDataSizeCommand()
|
||||
var count int = 0
|
||||
var s [5]uint
|
||||
var bits float64 = 0.0
|
||||
var i uint
|
||||
if histogram.total_count_ == 0 {
|
||||
return kOneSymbolHistogramCost
|
||||
}
|
||||
|
||||
for i = 0; i < data_size; i++ {
|
||||
if histogram.data_[i] > 0 {
|
||||
s[count] = i
|
||||
count++
|
||||
if count > 4 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if count == 1 {
|
||||
return kOneSymbolHistogramCost
|
||||
}
|
||||
|
||||
if count == 2 {
|
||||
return kTwoSymbolHistogramCost + float64(histogram.total_count_)
|
||||
}
|
||||
|
||||
if count == 3 {
|
||||
var histo0 uint32 = histogram.data_[s[0]]
|
||||
var histo1 uint32 = histogram.data_[s[1]]
|
||||
var histo2 uint32 = histogram.data_[s[2]]
|
||||
var histomax uint32 = brotli_max_uint32_t(histo0, brotli_max_uint32_t(histo1, histo2))
|
||||
return kThreeSymbolHistogramCost + 2*(float64(histo0)+float64(histo1)+float64(histo2)) - float64(histomax)
|
||||
}
|
||||
|
||||
if count == 4 {
|
||||
var histo [4]uint32
|
||||
var h23 uint32
|
||||
var histomax uint32
|
||||
for i = 0; i < 4; i++ {
|
||||
histo[i] = histogram.data_[s[i]]
|
||||
}
|
||||
|
||||
/* Sort */
|
||||
for i = 0; i < 4; i++ {
|
||||
var j uint
|
||||
for j = i + 1; j < 4; j++ {
|
||||
if histo[j] > histo[i] {
|
||||
var tmp uint32 = histo[j]
|
||||
histo[j] = histo[i]
|
||||
histo[i] = tmp
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
h23 = histo[2] + histo[3]
|
||||
histomax = brotli_max_uint32_t(h23, histo[0])
|
||||
return kFourSymbolHistogramCost + 3*float64(h23) + 2*(float64(histo[0])+float64(histo[1])) - float64(histomax)
|
||||
}
|
||||
{
|
||||
var max_depth uint = 1
|
||||
var depth_histo = [codeLengthCodes]uint32{0}
|
||||
/* In this loop we compute the entropy of the histogram and simultaneously
|
||||
build a simplified histogram of the code length codes where we use the
|
||||
zero repeat code 17, but we don't use the non-zero repeat code 16. */
|
||||
|
||||
var log2total float64 = fastLog2(histogram.total_count_)
|
||||
for i = 0; i < data_size; {
|
||||
if histogram.data_[i] > 0 {
|
||||
var log2p float64 = log2total - fastLog2(uint(histogram.data_[i]))
|
||||
/* Compute -log2(P(symbol)) = -log2(count(symbol)/total_count) =
|
||||
= log2(total_count) - log2(count(symbol)) */
|
||||
|
||||
var depth uint = uint(log2p + 0.5)
|
||||
/* Approximate the bit depth by round(-log2(P(symbol))) */
|
||||
bits += float64(histogram.data_[i]) * log2p
|
||||
|
||||
if depth > 15 {
|
||||
depth = 15
|
||||
}
|
||||
|
||||
if depth > max_depth {
|
||||
max_depth = depth
|
||||
}
|
||||
|
||||
depth_histo[depth]++
|
||||
i++
|
||||
} else {
|
||||
var reps uint32 = 1
|
||||
/* Compute the run length of zeros and add the appropriate number of 0
|
||||
and 17 code length codes to the code length code histogram. */
|
||||
|
||||
var k uint
|
||||
for k = i + 1; k < data_size && histogram.data_[k] == 0; k++ {
|
||||
reps++
|
||||
}
|
||||
|
||||
i += uint(reps)
|
||||
if i == data_size {
|
||||
/* Don't add any cost for the last zero run, since these are encoded
|
||||
only implicitly. */
|
||||
break
|
||||
}
|
||||
|
||||
if reps < 3 {
|
||||
depth_histo[0] += reps
|
||||
} else {
|
||||
reps -= 2
|
||||
for reps > 0 {
|
||||
depth_histo[repeatZeroCodeLength]++
|
||||
|
||||
/* Add the 3 extra bits for the 17 code length code. */
|
||||
bits += 3
|
||||
|
||||
reps >>= 3
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Add the estimated encoding cost of the code length code histogram. */
|
||||
bits += float64(18 + 2*max_depth)
|
||||
|
||||
/* Add the entropy of the code length code histogram. */
|
||||
bits += bitsEntropy(depth_histo[:], codeLengthCodes)
|
||||
}
|
||||
|
||||
return bits
|
||||
}
|
||||
|
||||
func populationCostDistance(histogram *histogramDistance) float64 {
|
||||
var data_size uint = histogramDataSizeDistance()
|
||||
var count int = 0
|
||||
var s [5]uint
|
||||
var bits float64 = 0.0
|
||||
var i uint
|
||||
if histogram.total_count_ == 0 {
|
||||
return kOneSymbolHistogramCost
|
||||
}
|
||||
|
||||
for i = 0; i < data_size; i++ {
|
||||
if histogram.data_[i] > 0 {
|
||||
s[count] = i
|
||||
count++
|
||||
if count > 4 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if count == 1 {
|
||||
return kOneSymbolHistogramCost
|
||||
}
|
||||
|
||||
if count == 2 {
|
||||
return kTwoSymbolHistogramCost + float64(histogram.total_count_)
|
||||
}
|
||||
|
||||
if count == 3 {
|
||||
var histo0 uint32 = histogram.data_[s[0]]
|
||||
var histo1 uint32 = histogram.data_[s[1]]
|
||||
var histo2 uint32 = histogram.data_[s[2]]
|
||||
var histomax uint32 = brotli_max_uint32_t(histo0, brotli_max_uint32_t(histo1, histo2))
|
||||
return kThreeSymbolHistogramCost + 2*(float64(histo0)+float64(histo1)+float64(histo2)) - float64(histomax)
|
||||
}
|
||||
|
||||
if count == 4 {
|
||||
var histo [4]uint32
|
||||
var h23 uint32
|
||||
var histomax uint32
|
||||
for i = 0; i < 4; i++ {
|
||||
histo[i] = histogram.data_[s[i]]
|
||||
}
|
||||
|
||||
/* Sort */
|
||||
for i = 0; i < 4; i++ {
|
||||
var j uint
|
||||
for j = i + 1; j < 4; j++ {
|
||||
if histo[j] > histo[i] {
|
||||
var tmp uint32 = histo[j]
|
||||
histo[j] = histo[i]
|
||||
histo[i] = tmp
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
h23 = histo[2] + histo[3]
|
||||
histomax = brotli_max_uint32_t(h23, histo[0])
|
||||
return kFourSymbolHistogramCost + 3*float64(h23) + 2*(float64(histo[0])+float64(histo[1])) - float64(histomax)
|
||||
}
|
||||
{
|
||||
var max_depth uint = 1
|
||||
var depth_histo = [codeLengthCodes]uint32{0}
|
||||
/* In this loop we compute the entropy of the histogram and simultaneously
|
||||
build a simplified histogram of the code length codes where we use the
|
||||
zero repeat code 17, but we don't use the non-zero repeat code 16. */
|
||||
|
||||
var log2total float64 = fastLog2(histogram.total_count_)
|
||||
for i = 0; i < data_size; {
|
||||
if histogram.data_[i] > 0 {
|
||||
var log2p float64 = log2total - fastLog2(uint(histogram.data_[i]))
|
||||
/* Compute -log2(P(symbol)) = -log2(count(symbol)/total_count) =
|
||||
= log2(total_count) - log2(count(symbol)) */
|
||||
|
||||
var depth uint = uint(log2p + 0.5)
|
||||
/* Approximate the bit depth by round(-log2(P(symbol))) */
|
||||
bits += float64(histogram.data_[i]) * log2p
|
||||
|
||||
if depth > 15 {
|
||||
depth = 15
|
||||
}
|
||||
|
||||
if depth > max_depth {
|
||||
max_depth = depth
|
||||
}
|
||||
|
||||
depth_histo[depth]++
|
||||
i++
|
||||
} else {
|
||||
var reps uint32 = 1
|
||||
/* Compute the run length of zeros and add the appropriate number of 0
|
||||
and 17 code length codes to the code length code histogram. */
|
||||
|
||||
var k uint
|
||||
for k = i + 1; k < data_size && histogram.data_[k] == 0; k++ {
|
||||
reps++
|
||||
}
|
||||
|
||||
i += uint(reps)
|
||||
if i == data_size {
|
||||
/* Don't add any cost for the last zero run, since these are encoded
|
||||
only implicitly. */
|
||||
break
|
||||
}
|
||||
|
||||
if reps < 3 {
|
||||
depth_histo[0] += reps
|
||||
} else {
|
||||
reps -= 2
|
||||
for reps > 0 {
|
||||
depth_histo[repeatZeroCodeLength]++
|
||||
|
||||
/* Add the 3 extra bits for the 17 code length code. */
|
||||
bits += 3
|
||||
|
||||
reps >>= 3
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Add the estimated encoding cost of the code length code histogram. */
|
||||
bits += float64(18 + 2*max_depth)
|
||||
|
||||
/* Add the entropy of the code length code histogram. */
|
||||
bits += bitsEntropy(depth_histo[:], codeLengthCodes)
|
||||
}
|
||||
|
||||
return bits
|
||||
}
|
||||
+266
@@ -0,0 +1,266 @@
|
||||
package brotli
|
||||
|
||||
import "encoding/binary"
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Bit reading helpers */
|
||||
|
||||
const shortFillBitWindowRead = (8 >> 1)
|
||||
|
||||
var kBitMask = [33]uint32{
|
||||
0x00000000,
|
||||
0x00000001,
|
||||
0x00000003,
|
||||
0x00000007,
|
||||
0x0000000F,
|
||||
0x0000001F,
|
||||
0x0000003F,
|
||||
0x0000007F,
|
||||
0x000000FF,
|
||||
0x000001FF,
|
||||
0x000003FF,
|
||||
0x000007FF,
|
||||
0x00000FFF,
|
||||
0x00001FFF,
|
||||
0x00003FFF,
|
||||
0x00007FFF,
|
||||
0x0000FFFF,
|
||||
0x0001FFFF,
|
||||
0x0003FFFF,
|
||||
0x0007FFFF,
|
||||
0x000FFFFF,
|
||||
0x001FFFFF,
|
||||
0x003FFFFF,
|
||||
0x007FFFFF,
|
||||
0x00FFFFFF,
|
||||
0x01FFFFFF,
|
||||
0x03FFFFFF,
|
||||
0x07FFFFFF,
|
||||
0x0FFFFFFF,
|
||||
0x1FFFFFFF,
|
||||
0x3FFFFFFF,
|
||||
0x7FFFFFFF,
|
||||
0xFFFFFFFF,
|
||||
}
|
||||
|
||||
func bitMask(n uint32) uint32 {
|
||||
return kBitMask[n]
|
||||
}
|
||||
|
||||
type bitReader struct {
|
||||
val_ uint64
|
||||
bit_pos_ uint32
|
||||
input []byte
|
||||
input_len uint
|
||||
byte_pos uint
|
||||
}
|
||||
|
||||
type bitReaderState struct {
|
||||
val_ uint64
|
||||
bit_pos_ uint32
|
||||
input []byte
|
||||
input_len uint
|
||||
byte_pos uint
|
||||
}
|
||||
|
||||
/* Initializes the BrotliBitReader fields. */
|
||||
|
||||
/* Ensures that accumulator is not empty.
|
||||
May consume up to sizeof(brotli_reg_t) - 1 bytes of input.
|
||||
Returns false if data is required but there is no input available.
|
||||
For BROTLI_ALIGNED_READ this function also prepares bit reader for aligned
|
||||
reading. */
|
||||
func bitReaderSaveState(from *bitReader, to *bitReaderState) {
|
||||
to.val_ = from.val_
|
||||
to.bit_pos_ = from.bit_pos_
|
||||
to.input = from.input
|
||||
to.input_len = from.input_len
|
||||
to.byte_pos = from.byte_pos
|
||||
}
|
||||
|
||||
func bitReaderRestoreState(to *bitReader, from *bitReaderState) {
|
||||
to.val_ = from.val_
|
||||
to.bit_pos_ = from.bit_pos_
|
||||
to.input = from.input
|
||||
to.input_len = from.input_len
|
||||
to.byte_pos = from.byte_pos
|
||||
}
|
||||
|
||||
func getAvailableBits(br *bitReader) uint32 {
|
||||
return 64 - br.bit_pos_
|
||||
}
|
||||
|
||||
/* Returns amount of unread bytes the bit reader still has buffered from the
|
||||
BrotliInput, including whole bytes in br->val_. */
|
||||
func getRemainingBytes(br *bitReader) uint {
|
||||
return uint(uint32(br.input_len-br.byte_pos) + (getAvailableBits(br) >> 3))
|
||||
}
|
||||
|
||||
/* Checks if there is at least |num| bytes left in the input ring-buffer
|
||||
(excluding the bits remaining in br->val_). */
|
||||
func checkInputAmount(br *bitReader, num uint) bool {
|
||||
return br.input_len-br.byte_pos >= num
|
||||
}
|
||||
|
||||
/* Guarantees that there are at least |n_bits| + 1 bits in accumulator.
|
||||
Precondition: accumulator contains at least 1 bit.
|
||||
|n_bits| should be in the range [1..24] for regular build. For portable
|
||||
non-64-bit little-endian build only 16 bits are safe to request. */
|
||||
func fillBitWindow(br *bitReader, n_bits uint32) {
|
||||
if br.bit_pos_ >= 32 {
|
||||
br.val_ >>= 32
|
||||
br.bit_pos_ ^= 32 /* here same as -= 32 because of the if condition */
|
||||
br.val_ |= (uint64(binary.LittleEndian.Uint32(br.input[br.byte_pos:]))) << 32
|
||||
br.byte_pos += 4
|
||||
}
|
||||
}
|
||||
|
||||
/* Mostly like BrotliFillBitWindow, but guarantees only 16 bits and reads no
|
||||
more than BROTLI_SHORT_FILL_BIT_WINDOW_READ bytes of input. */
|
||||
func fillBitWindow16(br *bitReader) {
|
||||
fillBitWindow(br, 17)
|
||||
}
|
||||
|
||||
/* Tries to pull one byte of input to accumulator.
|
||||
Returns false if there is no input available. */
|
||||
func pullByte(br *bitReader) bool {
|
||||
if br.byte_pos == br.input_len {
|
||||
return false
|
||||
}
|
||||
|
||||
br.val_ >>= 8
|
||||
br.val_ |= (uint64(br.input[br.byte_pos])) << 56
|
||||
br.bit_pos_ -= 8
|
||||
br.byte_pos++
|
||||
return true
|
||||
}
|
||||
|
||||
/* Returns currently available bits.
|
||||
The number of valid bits could be calculated by BrotliGetAvailableBits. */
|
||||
func getBitsUnmasked(br *bitReader) uint64 {
|
||||
return br.val_ >> br.bit_pos_
|
||||
}
|
||||
|
||||
/* Like BrotliGetBits, but does not mask the result.
|
||||
The result contains at least 16 valid bits. */
|
||||
func get16BitsUnmasked(br *bitReader) uint32 {
|
||||
fillBitWindow(br, 16)
|
||||
return uint32(getBitsUnmasked(br))
|
||||
}
|
||||
|
||||
/* Returns the specified number of bits from |br| without advancing bit
|
||||
position. */
|
||||
func getBits(br *bitReader, n_bits uint32) uint32 {
|
||||
fillBitWindow(br, n_bits)
|
||||
return uint32(getBitsUnmasked(br)) & bitMask(n_bits)
|
||||
}
|
||||
|
||||
/* Tries to peek the specified amount of bits. Returns false, if there
|
||||
is not enough input. */
|
||||
func safeGetBits(br *bitReader, n_bits uint32, val *uint32) bool {
|
||||
for getAvailableBits(br) < n_bits {
|
||||
if !pullByte(br) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
*val = uint32(getBitsUnmasked(br)) & bitMask(n_bits)
|
||||
return true
|
||||
}
|
||||
|
||||
/* Advances the bit pos by |n_bits|. */
|
||||
func dropBits(br *bitReader, n_bits uint32) {
|
||||
br.bit_pos_ += n_bits
|
||||
}
|
||||
|
||||
func bitReaderUnload(br *bitReader) {
|
||||
var unused_bytes uint32 = getAvailableBits(br) >> 3
|
||||
var unused_bits uint32 = unused_bytes << 3
|
||||
br.byte_pos -= uint(unused_bytes)
|
||||
if unused_bits == 64 {
|
||||
br.val_ = 0
|
||||
} else {
|
||||
br.val_ <<= unused_bits
|
||||
}
|
||||
|
||||
br.bit_pos_ += unused_bits
|
||||
}
|
||||
|
||||
/* Reads the specified number of bits from |br| and advances the bit pos.
|
||||
Precondition: accumulator MUST contain at least |n_bits|. */
|
||||
func takeBits(br *bitReader, n_bits uint32, val *uint32) {
|
||||
*val = uint32(getBitsUnmasked(br)) & bitMask(n_bits)
|
||||
dropBits(br, n_bits)
|
||||
}
|
||||
|
||||
/* Reads the specified number of bits from |br| and advances the bit pos.
|
||||
Assumes that there is enough input to perform BrotliFillBitWindow. */
|
||||
func readBits(br *bitReader, n_bits uint32) uint32 {
|
||||
var val uint32
|
||||
fillBitWindow(br, n_bits)
|
||||
takeBits(br, n_bits, &val)
|
||||
return val
|
||||
}
|
||||
|
||||
/* Tries to read the specified amount of bits. Returns false, if there
|
||||
is not enough input. |n_bits| MUST be positive. */
|
||||
func safeReadBits(br *bitReader, n_bits uint32, val *uint32) bool {
|
||||
for getAvailableBits(br) < n_bits {
|
||||
if !pullByte(br) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
takeBits(br, n_bits, val)
|
||||
return true
|
||||
}
|
||||
|
||||
/* Advances the bit reader position to the next byte boundary and verifies
|
||||
that any skipped bits are set to zero. */
|
||||
func bitReaderJumpToByteBoundary(br *bitReader) bool {
|
||||
var pad_bits_count uint32 = getAvailableBits(br) & 0x7
|
||||
var pad_bits uint32 = 0
|
||||
if pad_bits_count != 0 {
|
||||
takeBits(br, pad_bits_count, &pad_bits)
|
||||
}
|
||||
|
||||
return pad_bits == 0
|
||||
}
|
||||
|
||||
/* Copies remaining input bytes stored in the bit reader to the output. Value
|
||||
|num| may not be larger than BrotliGetRemainingBytes. The bit reader must be
|
||||
warmed up again after this. */
|
||||
func copyBytes(dest []byte, br *bitReader, num uint) {
|
||||
for getAvailableBits(br) >= 8 && num > 0 {
|
||||
dest[0] = byte(getBitsUnmasked(br))
|
||||
dropBits(br, 8)
|
||||
dest = dest[1:]
|
||||
num--
|
||||
}
|
||||
|
||||
copy(dest, br.input[br.byte_pos:][:num])
|
||||
br.byte_pos += num
|
||||
}
|
||||
|
||||
func initBitReader(br *bitReader) {
|
||||
br.val_ = 0
|
||||
br.bit_pos_ = 64
|
||||
}
|
||||
|
||||
func warmupBitReader(br *bitReader) bool {
|
||||
/* Fixing alignment after unaligned BrotliFillWindow would result accumulator
|
||||
overflow. If unalignment is caused by BrotliSafeReadBits, then there is
|
||||
enough space in accumulator to fix alignment. */
|
||||
if getAvailableBits(br) == 0 {
|
||||
if !pullByte(br) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2010 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Write bits into a byte array. */
|
||||
|
||||
type bitWriter struct {
|
||||
dst []byte
|
||||
|
||||
// Data waiting to be written is the low nbits of bits.
|
||||
bits uint64
|
||||
nbits uint
|
||||
}
|
||||
|
||||
func (w *bitWriter) writeBits(nb uint, b uint64) {
|
||||
w.bits |= b << w.nbits
|
||||
w.nbits += nb
|
||||
if w.nbits >= 32 {
|
||||
bits := w.bits
|
||||
w.bits >>= 32
|
||||
w.nbits -= 32
|
||||
w.dst = append(w.dst,
|
||||
byte(bits),
|
||||
byte(bits>>8),
|
||||
byte(bits>>16),
|
||||
byte(bits>>24),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
func (w *bitWriter) writeSingleBit(bit bool) {
|
||||
if bit {
|
||||
w.writeBits(1, 1)
|
||||
} else {
|
||||
w.writeBits(1, 0)
|
||||
}
|
||||
}
|
||||
|
||||
func (w *bitWriter) jumpToByteBoundary() {
|
||||
dst := w.dst
|
||||
for w.nbits != 0 {
|
||||
dst = append(dst, byte(w.bits))
|
||||
w.bits >>= 8
|
||||
if w.nbits > 8 { // Avoid underflow
|
||||
w.nbits -= 8
|
||||
} else {
|
||||
w.nbits = 0
|
||||
}
|
||||
}
|
||||
w.bits = 0
|
||||
w.dst = dst
|
||||
}
|
||||
+144
@@ -0,0 +1,144 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Block split point selection utilities. */
|
||||
|
||||
type blockSplit struct {
|
||||
num_types uint
|
||||
num_blocks uint
|
||||
types []byte
|
||||
lengths []uint32
|
||||
types_alloc_size uint
|
||||
lengths_alloc_size uint
|
||||
}
|
||||
|
||||
const (
|
||||
kMaxLiteralHistograms uint = 100
|
||||
kMaxCommandHistograms uint = 50
|
||||
kLiteralBlockSwitchCost float64 = 28.1
|
||||
kCommandBlockSwitchCost float64 = 13.5
|
||||
kDistanceBlockSwitchCost float64 = 14.6
|
||||
kLiteralStrideLength uint = 70
|
||||
kCommandStrideLength uint = 40
|
||||
kSymbolsPerLiteralHistogram uint = 544
|
||||
kSymbolsPerCommandHistogram uint = 530
|
||||
kSymbolsPerDistanceHistogram uint = 544
|
||||
kMinLengthForBlockSplitting uint = 128
|
||||
kIterMulForRefining uint = 2
|
||||
kMinItersForRefining uint = 100
|
||||
)
|
||||
|
||||
func countLiterals(cmds []command) uint {
|
||||
var total_length uint = 0
|
||||
/* Count how many we have. */
|
||||
|
||||
for i := range cmds {
|
||||
total_length += uint(cmds[i].insert_len_)
|
||||
}
|
||||
|
||||
return total_length
|
||||
}
|
||||
|
||||
func copyLiteralsToByteArray(cmds []command, data []byte, offset uint, mask uint, literals []byte) {
|
||||
var pos uint = 0
|
||||
var from_pos uint = offset & mask
|
||||
for i := range cmds {
|
||||
var insert_len uint = uint(cmds[i].insert_len_)
|
||||
if from_pos+insert_len > mask {
|
||||
var head_size uint = mask + 1 - from_pos
|
||||
copy(literals[pos:], data[from_pos:][:head_size])
|
||||
from_pos = 0
|
||||
pos += head_size
|
||||
insert_len -= head_size
|
||||
}
|
||||
|
||||
if insert_len > 0 {
|
||||
copy(literals[pos:], data[from_pos:][:insert_len])
|
||||
pos += insert_len
|
||||
}
|
||||
|
||||
from_pos = uint((uint32(from_pos+insert_len) + commandCopyLen(&cmds[i])) & uint32(mask))
|
||||
}
|
||||
}
|
||||
|
||||
func myRand(seed *uint32) uint32 {
|
||||
/* Initial seed should be 7. In this case, loop length is (1 << 29). */
|
||||
*seed *= 16807
|
||||
|
||||
return *seed
|
||||
}
|
||||
|
||||
func bitCost(count uint) float64 {
|
||||
if count == 0 {
|
||||
return -2.0
|
||||
} else {
|
||||
return fastLog2(count)
|
||||
}
|
||||
}
|
||||
|
||||
const histogramsPerBatch = 64
|
||||
|
||||
const clustersPerBatch = 16
|
||||
|
||||
func initBlockSplit(self *blockSplit) {
|
||||
self.num_types = 0
|
||||
self.num_blocks = 0
|
||||
self.types = self.types[:0]
|
||||
self.lengths = self.lengths[:0]
|
||||
self.types_alloc_size = 0
|
||||
self.lengths_alloc_size = 0
|
||||
}
|
||||
|
||||
func splitBlock(cmds []command, data []byte, pos uint, mask uint, params *encoderParams, literal_split *blockSplit, insert_and_copy_split *blockSplit, dist_split *blockSplit) {
|
||||
{
|
||||
var literals_count uint = countLiterals(cmds)
|
||||
var literals []byte = make([]byte, literals_count)
|
||||
|
||||
/* Create a continuous array of literals. */
|
||||
copyLiteralsToByteArray(cmds, data, pos, mask, literals)
|
||||
|
||||
/* Create the block split on the array of literals.
|
||||
Literal histograms have alphabet size 256. */
|
||||
splitByteVectorLiteral(literals, literals_count, kSymbolsPerLiteralHistogram, kMaxLiteralHistograms, kLiteralStrideLength, kLiteralBlockSwitchCost, params, literal_split)
|
||||
|
||||
literals = nil
|
||||
}
|
||||
{
|
||||
var insert_and_copy_codes []uint16 = make([]uint16, len(cmds))
|
||||
/* Compute prefix codes for commands. */
|
||||
|
||||
for i := range cmds {
|
||||
insert_and_copy_codes[i] = cmds[i].cmd_prefix_
|
||||
}
|
||||
|
||||
/* Create the block split on the array of command prefixes. */
|
||||
splitByteVectorCommand(insert_and_copy_codes, kSymbolsPerCommandHistogram, kMaxCommandHistograms, kCommandStrideLength, kCommandBlockSwitchCost, params, insert_and_copy_split)
|
||||
|
||||
/* TODO: reuse for distances? */
|
||||
|
||||
insert_and_copy_codes = nil
|
||||
}
|
||||
{
|
||||
var distance_prefixes []uint16 = make([]uint16, len(cmds))
|
||||
var j uint = 0
|
||||
/* Create a continuous array of distance prefixes. */
|
||||
|
||||
for i := range cmds {
|
||||
var cmd *command = &cmds[i]
|
||||
if commandCopyLen(cmd) != 0 && cmd.cmd_prefix_ >= 128 {
|
||||
distance_prefixes[j] = cmd.dist_prefix_ & 0x3FF
|
||||
j++
|
||||
}
|
||||
}
|
||||
|
||||
/* Create the block split on the array of distance prefixes. */
|
||||
splitByteVectorDistance(distance_prefixes, j, kSymbolsPerDistanceHistogram, kMaxCommandHistograms, kCommandStrideLength, kDistanceBlockSwitchCost, params, dist_split)
|
||||
|
||||
distance_prefixes = nil
|
||||
}
|
||||
}
|
||||
+434
@@ -0,0 +1,434 @@
|
||||
package brotli
|
||||
|
||||
import "math"
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
func initialEntropyCodesCommand(data []uint16, length uint, stride uint, num_histograms uint, histograms []histogramCommand) {
|
||||
var seed uint32 = 7
|
||||
var block_length uint = length / num_histograms
|
||||
var i uint
|
||||
clearHistogramsCommand(histograms, num_histograms)
|
||||
for i = 0; i < num_histograms; i++ {
|
||||
var pos uint = length * i / num_histograms
|
||||
if i != 0 {
|
||||
pos += uint(myRand(&seed) % uint32(block_length))
|
||||
}
|
||||
|
||||
if pos+stride >= length {
|
||||
pos = length - stride - 1
|
||||
}
|
||||
|
||||
histogramAddVectorCommand(&histograms[i], data[pos:], stride)
|
||||
}
|
||||
}
|
||||
|
||||
func randomSampleCommand(seed *uint32, data []uint16, length uint, stride uint, sample *histogramCommand) {
|
||||
var pos uint = 0
|
||||
if stride >= length {
|
||||
stride = length
|
||||
} else {
|
||||
pos = uint(myRand(seed) % uint32(length-stride+1))
|
||||
}
|
||||
|
||||
histogramAddVectorCommand(sample, data[pos:], stride)
|
||||
}
|
||||
|
||||
func refineEntropyCodesCommand(data []uint16, length uint, stride uint, num_histograms uint, histograms []histogramCommand) {
|
||||
var iters uint = kIterMulForRefining*length/stride + kMinItersForRefining
|
||||
var seed uint32 = 7
|
||||
var iter uint
|
||||
iters = ((iters + num_histograms - 1) / num_histograms) * num_histograms
|
||||
for iter = 0; iter < iters; iter++ {
|
||||
var sample histogramCommand
|
||||
histogramClearCommand(&sample)
|
||||
randomSampleCommand(&seed, data, length, stride, &sample)
|
||||
histogramAddHistogramCommand(&histograms[iter%num_histograms], &sample)
|
||||
}
|
||||
}
|
||||
|
||||
/* Assigns a block id from the range [0, num_histograms) to each data element
|
||||
in data[0..length) and fills in block_id[0..length) with the assigned values.
|
||||
Returns the number of blocks, i.e. one plus the number of block switches. */
|
||||
func findBlocksCommand(data []uint16, length uint, block_switch_bitcost float64, num_histograms uint, histograms []histogramCommand, insert_cost []float64, cost []float64, switch_signal []byte, block_id []byte) uint {
|
||||
var data_size uint = histogramDataSizeCommand()
|
||||
var bitmaplen uint = (num_histograms + 7) >> 3
|
||||
var num_blocks uint = 1
|
||||
var i uint
|
||||
var j uint
|
||||
assert(num_histograms <= 256)
|
||||
if num_histograms <= 1 {
|
||||
for i = 0; i < length; i++ {
|
||||
block_id[i] = 0
|
||||
}
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
for i := 0; i < int(data_size*num_histograms); i++ {
|
||||
insert_cost[i] = 0
|
||||
}
|
||||
for i = 0; i < num_histograms; i++ {
|
||||
insert_cost[i] = fastLog2(uint(uint32(histograms[i].total_count_)))
|
||||
}
|
||||
|
||||
for i = data_size; i != 0; {
|
||||
i--
|
||||
for j = 0; j < num_histograms; j++ {
|
||||
insert_cost[i*num_histograms+j] = insert_cost[j] - bitCost(uint(histograms[j].data_[i]))
|
||||
}
|
||||
}
|
||||
|
||||
for i := 0; i < int(num_histograms); i++ {
|
||||
cost[i] = 0
|
||||
}
|
||||
for i := 0; i < int(length*bitmaplen); i++ {
|
||||
switch_signal[i] = 0
|
||||
}
|
||||
|
||||
/* After each iteration of this loop, cost[k] will contain the difference
|
||||
between the minimum cost of arriving at the current byte position using
|
||||
entropy code k, and the minimum cost of arriving at the current byte
|
||||
position. This difference is capped at the block switch cost, and if it
|
||||
reaches block switch cost, it means that when we trace back from the last
|
||||
position, we need to switch here. */
|
||||
for i = 0; i < length; i++ {
|
||||
var byte_ix uint = i
|
||||
var ix uint = byte_ix * bitmaplen
|
||||
var insert_cost_ix uint = uint(data[byte_ix]) * num_histograms
|
||||
var min_cost float64 = 1e99
|
||||
var block_switch_cost float64 = block_switch_bitcost
|
||||
var k uint
|
||||
for k = 0; k < num_histograms; k++ {
|
||||
/* We are coding the symbol in data[byte_ix] with entropy code k. */
|
||||
cost[k] += insert_cost[insert_cost_ix+k]
|
||||
|
||||
if cost[k] < min_cost {
|
||||
min_cost = cost[k]
|
||||
block_id[byte_ix] = byte(k)
|
||||
}
|
||||
}
|
||||
|
||||
/* More blocks for the beginning. */
|
||||
if byte_ix < 2000 {
|
||||
block_switch_cost *= 0.77 + 0.07*float64(byte_ix)/2000
|
||||
}
|
||||
|
||||
for k = 0; k < num_histograms; k++ {
|
||||
cost[k] -= min_cost
|
||||
if cost[k] >= block_switch_cost {
|
||||
var mask byte = byte(1 << (k & 7))
|
||||
cost[k] = block_switch_cost
|
||||
assert(k>>3 < bitmaplen)
|
||||
switch_signal[ix+(k>>3)] |= mask
|
||||
/* Trace back from the last position and switch at the marked places. */
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
var byte_ix uint = length - 1
|
||||
var ix uint = byte_ix * bitmaplen
|
||||
var cur_id byte = block_id[byte_ix]
|
||||
for byte_ix > 0 {
|
||||
var mask byte = byte(1 << (cur_id & 7))
|
||||
assert(uint(cur_id)>>3 < bitmaplen)
|
||||
byte_ix--
|
||||
ix -= bitmaplen
|
||||
if switch_signal[ix+uint(cur_id>>3)]&mask != 0 {
|
||||
if cur_id != block_id[byte_ix] {
|
||||
cur_id = block_id[byte_ix]
|
||||
num_blocks++
|
||||
}
|
||||
}
|
||||
|
||||
block_id[byte_ix] = cur_id
|
||||
}
|
||||
}
|
||||
|
||||
return num_blocks
|
||||
}
|
||||
|
||||
var remapBlockIdsCommand_kInvalidId uint16 = 256
|
||||
|
||||
func remapBlockIdsCommand(block_ids []byte, length uint, new_id []uint16, num_histograms uint) uint {
|
||||
var next_id uint16 = 0
|
||||
var i uint
|
||||
for i = 0; i < num_histograms; i++ {
|
||||
new_id[i] = remapBlockIdsCommand_kInvalidId
|
||||
}
|
||||
|
||||
for i = 0; i < length; i++ {
|
||||
assert(uint(block_ids[i]) < num_histograms)
|
||||
if new_id[block_ids[i]] == remapBlockIdsCommand_kInvalidId {
|
||||
new_id[block_ids[i]] = next_id
|
||||
next_id++
|
||||
}
|
||||
}
|
||||
|
||||
for i = 0; i < length; i++ {
|
||||
block_ids[i] = byte(new_id[block_ids[i]])
|
||||
assert(uint(block_ids[i]) < num_histograms)
|
||||
}
|
||||
|
||||
assert(uint(next_id) <= num_histograms)
|
||||
return uint(next_id)
|
||||
}
|
||||
|
||||
func buildBlockHistogramsCommand(data []uint16, length uint, block_ids []byte, num_histograms uint, histograms []histogramCommand) {
|
||||
var i uint
|
||||
clearHistogramsCommand(histograms, num_histograms)
|
||||
for i = 0; i < length; i++ {
|
||||
histogramAddCommand(&histograms[block_ids[i]], uint(data[i]))
|
||||
}
|
||||
}
|
||||
|
||||
var clusterBlocksCommand_kInvalidIndex uint32 = math.MaxUint32
|
||||
|
||||
func clusterBlocksCommand(data []uint16, length uint, num_blocks uint, block_ids []byte, split *blockSplit) {
|
||||
var histogram_symbols []uint32 = make([]uint32, num_blocks)
|
||||
var block_lengths []uint32 = make([]uint32, num_blocks)
|
||||
var expected_num_clusters uint = clustersPerBatch * (num_blocks + histogramsPerBatch - 1) / histogramsPerBatch
|
||||
var all_histograms_size uint = 0
|
||||
var all_histograms_capacity uint = expected_num_clusters
|
||||
var all_histograms []histogramCommand = make([]histogramCommand, all_histograms_capacity)
|
||||
var cluster_size_size uint = 0
|
||||
var cluster_size_capacity uint = expected_num_clusters
|
||||
var cluster_size []uint32 = make([]uint32, cluster_size_capacity)
|
||||
var num_clusters uint = 0
|
||||
var histograms []histogramCommand = make([]histogramCommand, brotli_min_size_t(num_blocks, histogramsPerBatch))
|
||||
var max_num_pairs uint = histogramsPerBatch * histogramsPerBatch / 2
|
||||
var pairs_capacity uint = max_num_pairs + 1
|
||||
var pairs []histogramPair = make([]histogramPair, pairs_capacity)
|
||||
var pos uint = 0
|
||||
var clusters []uint32
|
||||
var num_final_clusters uint
|
||||
var new_index []uint32
|
||||
var i uint
|
||||
var sizes = [histogramsPerBatch]uint32{0}
|
||||
var new_clusters = [histogramsPerBatch]uint32{0}
|
||||
var symbols = [histogramsPerBatch]uint32{0}
|
||||
var remap = [histogramsPerBatch]uint32{0}
|
||||
|
||||
for i := 0; i < int(num_blocks); i++ {
|
||||
block_lengths[i] = 0
|
||||
}
|
||||
{
|
||||
var block_idx uint = 0
|
||||
for i = 0; i < length; i++ {
|
||||
assert(block_idx < num_blocks)
|
||||
block_lengths[block_idx]++
|
||||
if i+1 == length || block_ids[i] != block_ids[i+1] {
|
||||
block_idx++
|
||||
}
|
||||
}
|
||||
|
||||
assert(block_idx == num_blocks)
|
||||
}
|
||||
|
||||
for i = 0; i < num_blocks; i += histogramsPerBatch {
|
||||
var num_to_combine uint = brotli_min_size_t(num_blocks-i, histogramsPerBatch)
|
||||
var num_new_clusters uint
|
||||
var j uint
|
||||
for j = 0; j < num_to_combine; j++ {
|
||||
var k uint
|
||||
histogramClearCommand(&histograms[j])
|
||||
for k = 0; uint32(k) < block_lengths[i+j]; k++ {
|
||||
histogramAddCommand(&histograms[j], uint(data[pos]))
|
||||
pos++
|
||||
}
|
||||
|
||||
histograms[j].bit_cost_ = populationCostCommand(&histograms[j])
|
||||
new_clusters[j] = uint32(j)
|
||||
symbols[j] = uint32(j)
|
||||
sizes[j] = 1
|
||||
}
|
||||
|
||||
num_new_clusters = histogramCombineCommand(histograms, sizes[:], symbols[:], new_clusters[:], []histogramPair(pairs), num_to_combine, num_to_combine, histogramsPerBatch, max_num_pairs)
|
||||
if all_histograms_capacity < (all_histograms_size + num_new_clusters) {
|
||||
var _new_size uint
|
||||
if all_histograms_capacity == 0 {
|
||||
_new_size = all_histograms_size + num_new_clusters
|
||||
} else {
|
||||
_new_size = all_histograms_capacity
|
||||
}
|
||||
var new_array []histogramCommand
|
||||
for _new_size < (all_histograms_size + num_new_clusters) {
|
||||
_new_size *= 2
|
||||
}
|
||||
new_array = make([]histogramCommand, _new_size)
|
||||
if all_histograms_capacity != 0 {
|
||||
copy(new_array, all_histograms[:all_histograms_capacity])
|
||||
}
|
||||
|
||||
all_histograms = new_array
|
||||
all_histograms_capacity = _new_size
|
||||
}
|
||||
|
||||
brotli_ensure_capacity_uint32_t(&cluster_size, &cluster_size_capacity, cluster_size_size+num_new_clusters)
|
||||
for j = 0; j < num_new_clusters; j++ {
|
||||
all_histograms[all_histograms_size] = histograms[new_clusters[j]]
|
||||
all_histograms_size++
|
||||
cluster_size[cluster_size_size] = sizes[new_clusters[j]]
|
||||
cluster_size_size++
|
||||
remap[new_clusters[j]] = uint32(j)
|
||||
}
|
||||
|
||||
for j = 0; j < num_to_combine; j++ {
|
||||
histogram_symbols[i+j] = uint32(num_clusters) + remap[symbols[j]]
|
||||
}
|
||||
|
||||
num_clusters += num_new_clusters
|
||||
assert(num_clusters == cluster_size_size)
|
||||
assert(num_clusters == all_histograms_size)
|
||||
}
|
||||
|
||||
histograms = nil
|
||||
|
||||
max_num_pairs = brotli_min_size_t(64*num_clusters, (num_clusters/2)*num_clusters)
|
||||
if pairs_capacity < max_num_pairs+1 {
|
||||
pairs = nil
|
||||
pairs = make([]histogramPair, (max_num_pairs + 1))
|
||||
}
|
||||
|
||||
clusters = make([]uint32, num_clusters)
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
clusters[i] = uint32(i)
|
||||
}
|
||||
|
||||
num_final_clusters = histogramCombineCommand(all_histograms, cluster_size, histogram_symbols, clusters, pairs, num_clusters, num_blocks, maxNumberOfBlockTypes, max_num_pairs)
|
||||
pairs = nil
|
||||
cluster_size = nil
|
||||
|
||||
new_index = make([]uint32, num_clusters)
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
new_index[i] = clusterBlocksCommand_kInvalidIndex
|
||||
}
|
||||
pos = 0
|
||||
{
|
||||
var next_index uint32 = 0
|
||||
for i = 0; i < num_blocks; i++ {
|
||||
var histo histogramCommand
|
||||
var j uint
|
||||
var best_out uint32
|
||||
var best_bits float64
|
||||
histogramClearCommand(&histo)
|
||||
for j = 0; uint32(j) < block_lengths[i]; j++ {
|
||||
histogramAddCommand(&histo, uint(data[pos]))
|
||||
pos++
|
||||
}
|
||||
|
||||
if i == 0 {
|
||||
best_out = histogram_symbols[0]
|
||||
} else {
|
||||
best_out = histogram_symbols[i-1]
|
||||
}
|
||||
best_bits = histogramBitCostDistanceCommand(&histo, &all_histograms[best_out])
|
||||
for j = 0; j < num_final_clusters; j++ {
|
||||
var cur_bits float64 = histogramBitCostDistanceCommand(&histo, &all_histograms[clusters[j]])
|
||||
if cur_bits < best_bits {
|
||||
best_bits = cur_bits
|
||||
best_out = clusters[j]
|
||||
}
|
||||
}
|
||||
|
||||
histogram_symbols[i] = best_out
|
||||
if new_index[best_out] == clusterBlocksCommand_kInvalidIndex {
|
||||
new_index[best_out] = next_index
|
||||
next_index++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
clusters = nil
|
||||
all_histograms = nil
|
||||
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, num_blocks)
|
||||
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, num_blocks)
|
||||
{
|
||||
var cur_length uint32 = 0
|
||||
var block_idx uint = 0
|
||||
var max_type byte = 0
|
||||
for i = 0; i < num_blocks; i++ {
|
||||
cur_length += block_lengths[i]
|
||||
if i+1 == num_blocks || histogram_symbols[i] != histogram_symbols[i+1] {
|
||||
var id byte = byte(new_index[histogram_symbols[i]])
|
||||
split.types[block_idx] = id
|
||||
split.lengths[block_idx] = cur_length
|
||||
max_type = brotli_max_uint8_t(max_type, id)
|
||||
cur_length = 0
|
||||
block_idx++
|
||||
}
|
||||
}
|
||||
|
||||
split.num_blocks = block_idx
|
||||
split.num_types = uint(max_type) + 1
|
||||
}
|
||||
|
||||
new_index = nil
|
||||
block_lengths = nil
|
||||
histogram_symbols = nil
|
||||
}
|
||||
|
||||
func splitByteVectorCommand(data []uint16, literals_per_histogram uint, max_histograms uint, sampling_stride_length uint, block_switch_cost float64, params *encoderParams, split *blockSplit) {
|
||||
length := uint(len(data))
|
||||
var data_size uint = histogramDataSizeCommand()
|
||||
var num_histograms uint = length/literals_per_histogram + 1
|
||||
var histograms []histogramCommand
|
||||
if num_histograms > max_histograms {
|
||||
num_histograms = max_histograms
|
||||
}
|
||||
|
||||
if length == 0 {
|
||||
split.num_types = 1
|
||||
return
|
||||
} else if length < kMinLengthForBlockSplitting {
|
||||
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, split.num_blocks+1)
|
||||
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, split.num_blocks+1)
|
||||
split.num_types = 1
|
||||
split.types[split.num_blocks] = 0
|
||||
split.lengths[split.num_blocks] = uint32(length)
|
||||
split.num_blocks++
|
||||
return
|
||||
}
|
||||
|
||||
histograms = make([]histogramCommand, num_histograms)
|
||||
|
||||
/* Find good entropy codes. */
|
||||
initialEntropyCodesCommand(data, length, sampling_stride_length, num_histograms, histograms)
|
||||
|
||||
refineEntropyCodesCommand(data, length, sampling_stride_length, num_histograms, histograms)
|
||||
{
|
||||
var block_ids []byte = make([]byte, length)
|
||||
var num_blocks uint = 0
|
||||
var bitmaplen uint = (num_histograms + 7) >> 3
|
||||
var insert_cost []float64 = make([]float64, (data_size * num_histograms))
|
||||
var cost []float64 = make([]float64, num_histograms)
|
||||
var switch_signal []byte = make([]byte, (length * bitmaplen))
|
||||
var new_id []uint16 = make([]uint16, num_histograms)
|
||||
var iters uint
|
||||
if params.quality < hqZopflificationQuality {
|
||||
iters = 3
|
||||
} else {
|
||||
iters = 10
|
||||
}
|
||||
/* Find a good path through literals with the good entropy codes. */
|
||||
|
||||
var i uint
|
||||
for i = 0; i < iters; i++ {
|
||||
num_blocks = findBlocksCommand(data, length, block_switch_cost, num_histograms, histograms, insert_cost, cost, switch_signal, block_ids)
|
||||
num_histograms = remapBlockIdsCommand(block_ids, length, new_id, num_histograms)
|
||||
buildBlockHistogramsCommand(data, length, block_ids, num_histograms, histograms)
|
||||
}
|
||||
|
||||
insert_cost = nil
|
||||
cost = nil
|
||||
switch_signal = nil
|
||||
new_id = nil
|
||||
histograms = nil
|
||||
clusterBlocksCommand(data, length, num_blocks, block_ids, split)
|
||||
block_ids = nil
|
||||
}
|
||||
}
|
||||
+433
@@ -0,0 +1,433 @@
|
||||
package brotli
|
||||
|
||||
import "math"
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
func initialEntropyCodesDistance(data []uint16, length uint, stride uint, num_histograms uint, histograms []histogramDistance) {
|
||||
var seed uint32 = 7
|
||||
var block_length uint = length / num_histograms
|
||||
var i uint
|
||||
clearHistogramsDistance(histograms, num_histograms)
|
||||
for i = 0; i < num_histograms; i++ {
|
||||
var pos uint = length * i / num_histograms
|
||||
if i != 0 {
|
||||
pos += uint(myRand(&seed) % uint32(block_length))
|
||||
}
|
||||
|
||||
if pos+stride >= length {
|
||||
pos = length - stride - 1
|
||||
}
|
||||
|
||||
histogramAddVectorDistance(&histograms[i], data[pos:], stride)
|
||||
}
|
||||
}
|
||||
|
||||
func randomSampleDistance(seed *uint32, data []uint16, length uint, stride uint, sample *histogramDistance) {
|
||||
var pos uint = 0
|
||||
if stride >= length {
|
||||
stride = length
|
||||
} else {
|
||||
pos = uint(myRand(seed) % uint32(length-stride+1))
|
||||
}
|
||||
|
||||
histogramAddVectorDistance(sample, data[pos:], stride)
|
||||
}
|
||||
|
||||
func refineEntropyCodesDistance(data []uint16, length uint, stride uint, num_histograms uint, histograms []histogramDistance) {
|
||||
var iters uint = kIterMulForRefining*length/stride + kMinItersForRefining
|
||||
var seed uint32 = 7
|
||||
var iter uint
|
||||
iters = ((iters + num_histograms - 1) / num_histograms) * num_histograms
|
||||
for iter = 0; iter < iters; iter++ {
|
||||
var sample histogramDistance
|
||||
histogramClearDistance(&sample)
|
||||
randomSampleDistance(&seed, data, length, stride, &sample)
|
||||
histogramAddHistogramDistance(&histograms[iter%num_histograms], &sample)
|
||||
}
|
||||
}
|
||||
|
||||
/* Assigns a block id from the range [0, num_histograms) to each data element
|
||||
in data[0..length) and fills in block_id[0..length) with the assigned values.
|
||||
Returns the number of blocks, i.e. one plus the number of block switches. */
|
||||
func findBlocksDistance(data []uint16, length uint, block_switch_bitcost float64, num_histograms uint, histograms []histogramDistance, insert_cost []float64, cost []float64, switch_signal []byte, block_id []byte) uint {
|
||||
var data_size uint = histogramDataSizeDistance()
|
||||
var bitmaplen uint = (num_histograms + 7) >> 3
|
||||
var num_blocks uint = 1
|
||||
var i uint
|
||||
var j uint
|
||||
assert(num_histograms <= 256)
|
||||
if num_histograms <= 1 {
|
||||
for i = 0; i < length; i++ {
|
||||
block_id[i] = 0
|
||||
}
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
for i := 0; i < int(data_size*num_histograms); i++ {
|
||||
insert_cost[i] = 0
|
||||
}
|
||||
for i = 0; i < num_histograms; i++ {
|
||||
insert_cost[i] = fastLog2(uint(uint32(histograms[i].total_count_)))
|
||||
}
|
||||
|
||||
for i = data_size; i != 0; {
|
||||
i--
|
||||
for j = 0; j < num_histograms; j++ {
|
||||
insert_cost[i*num_histograms+j] = insert_cost[j] - bitCost(uint(histograms[j].data_[i]))
|
||||
}
|
||||
}
|
||||
|
||||
for i := 0; i < int(num_histograms); i++ {
|
||||
cost[i] = 0
|
||||
}
|
||||
for i := 0; i < int(length*bitmaplen); i++ {
|
||||
switch_signal[i] = 0
|
||||
}
|
||||
|
||||
/* After each iteration of this loop, cost[k] will contain the difference
|
||||
between the minimum cost of arriving at the current byte position using
|
||||
entropy code k, and the minimum cost of arriving at the current byte
|
||||
position. This difference is capped at the block switch cost, and if it
|
||||
reaches block switch cost, it means that when we trace back from the last
|
||||
position, we need to switch here. */
|
||||
for i = 0; i < length; i++ {
|
||||
var byte_ix uint = i
|
||||
var ix uint = byte_ix * bitmaplen
|
||||
var insert_cost_ix uint = uint(data[byte_ix]) * num_histograms
|
||||
var min_cost float64 = 1e99
|
||||
var block_switch_cost float64 = block_switch_bitcost
|
||||
var k uint
|
||||
for k = 0; k < num_histograms; k++ {
|
||||
/* We are coding the symbol in data[byte_ix] with entropy code k. */
|
||||
cost[k] += insert_cost[insert_cost_ix+k]
|
||||
|
||||
if cost[k] < min_cost {
|
||||
min_cost = cost[k]
|
||||
block_id[byte_ix] = byte(k)
|
||||
}
|
||||
}
|
||||
|
||||
/* More blocks for the beginning. */
|
||||
if byte_ix < 2000 {
|
||||
block_switch_cost *= 0.77 + 0.07*float64(byte_ix)/2000
|
||||
}
|
||||
|
||||
for k = 0; k < num_histograms; k++ {
|
||||
cost[k] -= min_cost
|
||||
if cost[k] >= block_switch_cost {
|
||||
var mask byte = byte(1 << (k & 7))
|
||||
cost[k] = block_switch_cost
|
||||
assert(k>>3 < bitmaplen)
|
||||
switch_signal[ix+(k>>3)] |= mask
|
||||
/* Trace back from the last position and switch at the marked places. */
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
var byte_ix uint = length - 1
|
||||
var ix uint = byte_ix * bitmaplen
|
||||
var cur_id byte = block_id[byte_ix]
|
||||
for byte_ix > 0 {
|
||||
var mask byte = byte(1 << (cur_id & 7))
|
||||
assert(uint(cur_id)>>3 < bitmaplen)
|
||||
byte_ix--
|
||||
ix -= bitmaplen
|
||||
if switch_signal[ix+uint(cur_id>>3)]&mask != 0 {
|
||||
if cur_id != block_id[byte_ix] {
|
||||
cur_id = block_id[byte_ix]
|
||||
num_blocks++
|
||||
}
|
||||
}
|
||||
|
||||
block_id[byte_ix] = cur_id
|
||||
}
|
||||
}
|
||||
|
||||
return num_blocks
|
||||
}
|
||||
|
||||
var remapBlockIdsDistance_kInvalidId uint16 = 256
|
||||
|
||||
func remapBlockIdsDistance(block_ids []byte, length uint, new_id []uint16, num_histograms uint) uint {
|
||||
var next_id uint16 = 0
|
||||
var i uint
|
||||
for i = 0; i < num_histograms; i++ {
|
||||
new_id[i] = remapBlockIdsDistance_kInvalidId
|
||||
}
|
||||
|
||||
for i = 0; i < length; i++ {
|
||||
assert(uint(block_ids[i]) < num_histograms)
|
||||
if new_id[block_ids[i]] == remapBlockIdsDistance_kInvalidId {
|
||||
new_id[block_ids[i]] = next_id
|
||||
next_id++
|
||||
}
|
||||
}
|
||||
|
||||
for i = 0; i < length; i++ {
|
||||
block_ids[i] = byte(new_id[block_ids[i]])
|
||||
assert(uint(block_ids[i]) < num_histograms)
|
||||
}
|
||||
|
||||
assert(uint(next_id) <= num_histograms)
|
||||
return uint(next_id)
|
||||
}
|
||||
|
||||
func buildBlockHistogramsDistance(data []uint16, length uint, block_ids []byte, num_histograms uint, histograms []histogramDistance) {
|
||||
var i uint
|
||||
clearHistogramsDistance(histograms, num_histograms)
|
||||
for i = 0; i < length; i++ {
|
||||
histogramAddDistance(&histograms[block_ids[i]], uint(data[i]))
|
||||
}
|
||||
}
|
||||
|
||||
var clusterBlocksDistance_kInvalidIndex uint32 = math.MaxUint32
|
||||
|
||||
func clusterBlocksDistance(data []uint16, length uint, num_blocks uint, block_ids []byte, split *blockSplit) {
|
||||
var histogram_symbols []uint32 = make([]uint32, num_blocks)
|
||||
var block_lengths []uint32 = make([]uint32, num_blocks)
|
||||
var expected_num_clusters uint = clustersPerBatch * (num_blocks + histogramsPerBatch - 1) / histogramsPerBatch
|
||||
var all_histograms_size uint = 0
|
||||
var all_histograms_capacity uint = expected_num_clusters
|
||||
var all_histograms []histogramDistance = make([]histogramDistance, all_histograms_capacity)
|
||||
var cluster_size_size uint = 0
|
||||
var cluster_size_capacity uint = expected_num_clusters
|
||||
var cluster_size []uint32 = make([]uint32, cluster_size_capacity)
|
||||
var num_clusters uint = 0
|
||||
var histograms []histogramDistance = make([]histogramDistance, brotli_min_size_t(num_blocks, histogramsPerBatch))
|
||||
var max_num_pairs uint = histogramsPerBatch * histogramsPerBatch / 2
|
||||
var pairs_capacity uint = max_num_pairs + 1
|
||||
var pairs []histogramPair = make([]histogramPair, pairs_capacity)
|
||||
var pos uint = 0
|
||||
var clusters []uint32
|
||||
var num_final_clusters uint
|
||||
var new_index []uint32
|
||||
var i uint
|
||||
var sizes = [histogramsPerBatch]uint32{0}
|
||||
var new_clusters = [histogramsPerBatch]uint32{0}
|
||||
var symbols = [histogramsPerBatch]uint32{0}
|
||||
var remap = [histogramsPerBatch]uint32{0}
|
||||
|
||||
for i := 0; i < int(num_blocks); i++ {
|
||||
block_lengths[i] = 0
|
||||
}
|
||||
{
|
||||
var block_idx uint = 0
|
||||
for i = 0; i < length; i++ {
|
||||
assert(block_idx < num_blocks)
|
||||
block_lengths[block_idx]++
|
||||
if i+1 == length || block_ids[i] != block_ids[i+1] {
|
||||
block_idx++
|
||||
}
|
||||
}
|
||||
|
||||
assert(block_idx == num_blocks)
|
||||
}
|
||||
|
||||
for i = 0; i < num_blocks; i += histogramsPerBatch {
|
||||
var num_to_combine uint = brotli_min_size_t(num_blocks-i, histogramsPerBatch)
|
||||
var num_new_clusters uint
|
||||
var j uint
|
||||
for j = 0; j < num_to_combine; j++ {
|
||||
var k uint
|
||||
histogramClearDistance(&histograms[j])
|
||||
for k = 0; uint32(k) < block_lengths[i+j]; k++ {
|
||||
histogramAddDistance(&histograms[j], uint(data[pos]))
|
||||
pos++
|
||||
}
|
||||
|
||||
histograms[j].bit_cost_ = populationCostDistance(&histograms[j])
|
||||
new_clusters[j] = uint32(j)
|
||||
symbols[j] = uint32(j)
|
||||
sizes[j] = 1
|
||||
}
|
||||
|
||||
num_new_clusters = histogramCombineDistance(histograms, sizes[:], symbols[:], new_clusters[:], []histogramPair(pairs), num_to_combine, num_to_combine, histogramsPerBatch, max_num_pairs)
|
||||
if all_histograms_capacity < (all_histograms_size + num_new_clusters) {
|
||||
var _new_size uint
|
||||
if all_histograms_capacity == 0 {
|
||||
_new_size = all_histograms_size + num_new_clusters
|
||||
} else {
|
||||
_new_size = all_histograms_capacity
|
||||
}
|
||||
var new_array []histogramDistance
|
||||
for _new_size < (all_histograms_size + num_new_clusters) {
|
||||
_new_size *= 2
|
||||
}
|
||||
new_array = make([]histogramDistance, _new_size)
|
||||
if all_histograms_capacity != 0 {
|
||||
copy(new_array, all_histograms[:all_histograms_capacity])
|
||||
}
|
||||
|
||||
all_histograms = new_array
|
||||
all_histograms_capacity = _new_size
|
||||
}
|
||||
|
||||
brotli_ensure_capacity_uint32_t(&cluster_size, &cluster_size_capacity, cluster_size_size+num_new_clusters)
|
||||
for j = 0; j < num_new_clusters; j++ {
|
||||
all_histograms[all_histograms_size] = histograms[new_clusters[j]]
|
||||
all_histograms_size++
|
||||
cluster_size[cluster_size_size] = sizes[new_clusters[j]]
|
||||
cluster_size_size++
|
||||
remap[new_clusters[j]] = uint32(j)
|
||||
}
|
||||
|
||||
for j = 0; j < num_to_combine; j++ {
|
||||
histogram_symbols[i+j] = uint32(num_clusters) + remap[symbols[j]]
|
||||
}
|
||||
|
||||
num_clusters += num_new_clusters
|
||||
assert(num_clusters == cluster_size_size)
|
||||
assert(num_clusters == all_histograms_size)
|
||||
}
|
||||
|
||||
histograms = nil
|
||||
|
||||
max_num_pairs = brotli_min_size_t(64*num_clusters, (num_clusters/2)*num_clusters)
|
||||
if pairs_capacity < max_num_pairs+1 {
|
||||
pairs = nil
|
||||
pairs = make([]histogramPair, (max_num_pairs + 1))
|
||||
}
|
||||
|
||||
clusters = make([]uint32, num_clusters)
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
clusters[i] = uint32(i)
|
||||
}
|
||||
|
||||
num_final_clusters = histogramCombineDistance(all_histograms, cluster_size, histogram_symbols, clusters, pairs, num_clusters, num_blocks, maxNumberOfBlockTypes, max_num_pairs)
|
||||
pairs = nil
|
||||
cluster_size = nil
|
||||
|
||||
new_index = make([]uint32, num_clusters)
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
new_index[i] = clusterBlocksDistance_kInvalidIndex
|
||||
}
|
||||
pos = 0
|
||||
{
|
||||
var next_index uint32 = 0
|
||||
for i = 0; i < num_blocks; i++ {
|
||||
var histo histogramDistance
|
||||
var j uint
|
||||
var best_out uint32
|
||||
var best_bits float64
|
||||
histogramClearDistance(&histo)
|
||||
for j = 0; uint32(j) < block_lengths[i]; j++ {
|
||||
histogramAddDistance(&histo, uint(data[pos]))
|
||||
pos++
|
||||
}
|
||||
|
||||
if i == 0 {
|
||||
best_out = histogram_symbols[0]
|
||||
} else {
|
||||
best_out = histogram_symbols[i-1]
|
||||
}
|
||||
best_bits = histogramBitCostDistanceDistance(&histo, &all_histograms[best_out])
|
||||
for j = 0; j < num_final_clusters; j++ {
|
||||
var cur_bits float64 = histogramBitCostDistanceDistance(&histo, &all_histograms[clusters[j]])
|
||||
if cur_bits < best_bits {
|
||||
best_bits = cur_bits
|
||||
best_out = clusters[j]
|
||||
}
|
||||
}
|
||||
|
||||
histogram_symbols[i] = best_out
|
||||
if new_index[best_out] == clusterBlocksDistance_kInvalidIndex {
|
||||
new_index[best_out] = next_index
|
||||
next_index++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
clusters = nil
|
||||
all_histograms = nil
|
||||
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, num_blocks)
|
||||
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, num_blocks)
|
||||
{
|
||||
var cur_length uint32 = 0
|
||||
var block_idx uint = 0
|
||||
var max_type byte = 0
|
||||
for i = 0; i < num_blocks; i++ {
|
||||
cur_length += block_lengths[i]
|
||||
if i+1 == num_blocks || histogram_symbols[i] != histogram_symbols[i+1] {
|
||||
var id byte = byte(new_index[histogram_symbols[i]])
|
||||
split.types[block_idx] = id
|
||||
split.lengths[block_idx] = cur_length
|
||||
max_type = brotli_max_uint8_t(max_type, id)
|
||||
cur_length = 0
|
||||
block_idx++
|
||||
}
|
||||
}
|
||||
|
||||
split.num_blocks = block_idx
|
||||
split.num_types = uint(max_type) + 1
|
||||
}
|
||||
|
||||
new_index = nil
|
||||
block_lengths = nil
|
||||
histogram_symbols = nil
|
||||
}
|
||||
|
||||
func splitByteVectorDistance(data []uint16, length uint, literals_per_histogram uint, max_histograms uint, sampling_stride_length uint, block_switch_cost float64, params *encoderParams, split *blockSplit) {
|
||||
var data_size uint = histogramDataSizeDistance()
|
||||
var num_histograms uint = length/literals_per_histogram + 1
|
||||
var histograms []histogramDistance
|
||||
if num_histograms > max_histograms {
|
||||
num_histograms = max_histograms
|
||||
}
|
||||
|
||||
if length == 0 {
|
||||
split.num_types = 1
|
||||
return
|
||||
} else if length < kMinLengthForBlockSplitting {
|
||||
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, split.num_blocks+1)
|
||||
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, split.num_blocks+1)
|
||||
split.num_types = 1
|
||||
split.types[split.num_blocks] = 0
|
||||
split.lengths[split.num_blocks] = uint32(length)
|
||||
split.num_blocks++
|
||||
return
|
||||
}
|
||||
|
||||
histograms = make([]histogramDistance, num_histograms)
|
||||
|
||||
/* Find good entropy codes. */
|
||||
initialEntropyCodesDistance(data, length, sampling_stride_length, num_histograms, histograms)
|
||||
|
||||
refineEntropyCodesDistance(data, length, sampling_stride_length, num_histograms, histograms)
|
||||
{
|
||||
var block_ids []byte = make([]byte, length)
|
||||
var num_blocks uint = 0
|
||||
var bitmaplen uint = (num_histograms + 7) >> 3
|
||||
var insert_cost []float64 = make([]float64, (data_size * num_histograms))
|
||||
var cost []float64 = make([]float64, num_histograms)
|
||||
var switch_signal []byte = make([]byte, (length * bitmaplen))
|
||||
var new_id []uint16 = make([]uint16, num_histograms)
|
||||
var iters uint
|
||||
if params.quality < hqZopflificationQuality {
|
||||
iters = 3
|
||||
} else {
|
||||
iters = 10
|
||||
}
|
||||
/* Find a good path through literals with the good entropy codes. */
|
||||
|
||||
var i uint
|
||||
for i = 0; i < iters; i++ {
|
||||
num_blocks = findBlocksDistance(data, length, block_switch_cost, num_histograms, histograms, insert_cost, cost, switch_signal, block_ids)
|
||||
num_histograms = remapBlockIdsDistance(block_ids, length, new_id, num_histograms)
|
||||
buildBlockHistogramsDistance(data, length, block_ids, num_histograms, histograms)
|
||||
}
|
||||
|
||||
insert_cost = nil
|
||||
cost = nil
|
||||
switch_signal = nil
|
||||
new_id = nil
|
||||
histograms = nil
|
||||
clusterBlocksDistance(data, length, num_blocks, block_ids, split)
|
||||
block_ids = nil
|
||||
}
|
||||
}
|
||||
+433
@@ -0,0 +1,433 @@
|
||||
package brotli
|
||||
|
||||
import "math"
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
func initialEntropyCodesLiteral(data []byte, length uint, stride uint, num_histograms uint, histograms []histogramLiteral) {
|
||||
var seed uint32 = 7
|
||||
var block_length uint = length / num_histograms
|
||||
var i uint
|
||||
clearHistogramsLiteral(histograms, num_histograms)
|
||||
for i = 0; i < num_histograms; i++ {
|
||||
var pos uint = length * i / num_histograms
|
||||
if i != 0 {
|
||||
pos += uint(myRand(&seed) % uint32(block_length))
|
||||
}
|
||||
|
||||
if pos+stride >= length {
|
||||
pos = length - stride - 1
|
||||
}
|
||||
|
||||
histogramAddVectorLiteral(&histograms[i], data[pos:], stride)
|
||||
}
|
||||
}
|
||||
|
||||
func randomSampleLiteral(seed *uint32, data []byte, length uint, stride uint, sample *histogramLiteral) {
|
||||
var pos uint = 0
|
||||
if stride >= length {
|
||||
stride = length
|
||||
} else {
|
||||
pos = uint(myRand(seed) % uint32(length-stride+1))
|
||||
}
|
||||
|
||||
histogramAddVectorLiteral(sample, data[pos:], stride)
|
||||
}
|
||||
|
||||
func refineEntropyCodesLiteral(data []byte, length uint, stride uint, num_histograms uint, histograms []histogramLiteral) {
|
||||
var iters uint = kIterMulForRefining*length/stride + kMinItersForRefining
|
||||
var seed uint32 = 7
|
||||
var iter uint
|
||||
iters = ((iters + num_histograms - 1) / num_histograms) * num_histograms
|
||||
for iter = 0; iter < iters; iter++ {
|
||||
var sample histogramLiteral
|
||||
histogramClearLiteral(&sample)
|
||||
randomSampleLiteral(&seed, data, length, stride, &sample)
|
||||
histogramAddHistogramLiteral(&histograms[iter%num_histograms], &sample)
|
||||
}
|
||||
}
|
||||
|
||||
/* Assigns a block id from the range [0, num_histograms) to each data element
|
||||
in data[0..length) and fills in block_id[0..length) with the assigned values.
|
||||
Returns the number of blocks, i.e. one plus the number of block switches. */
|
||||
func findBlocksLiteral(data []byte, length uint, block_switch_bitcost float64, num_histograms uint, histograms []histogramLiteral, insert_cost []float64, cost []float64, switch_signal []byte, block_id []byte) uint {
|
||||
var data_size uint = histogramDataSizeLiteral()
|
||||
var bitmaplen uint = (num_histograms + 7) >> 3
|
||||
var num_blocks uint = 1
|
||||
var i uint
|
||||
var j uint
|
||||
assert(num_histograms <= 256)
|
||||
if num_histograms <= 1 {
|
||||
for i = 0; i < length; i++ {
|
||||
block_id[i] = 0
|
||||
}
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
for i := 0; i < int(data_size*num_histograms); i++ {
|
||||
insert_cost[i] = 0
|
||||
}
|
||||
for i = 0; i < num_histograms; i++ {
|
||||
insert_cost[i] = fastLog2(uint(uint32(histograms[i].total_count_)))
|
||||
}
|
||||
|
||||
for i = data_size; i != 0; {
|
||||
i--
|
||||
for j = 0; j < num_histograms; j++ {
|
||||
insert_cost[i*num_histograms+j] = insert_cost[j] - bitCost(uint(histograms[j].data_[i]))
|
||||
}
|
||||
}
|
||||
|
||||
for i := 0; i < int(num_histograms); i++ {
|
||||
cost[i] = 0
|
||||
}
|
||||
for i := 0; i < int(length*bitmaplen); i++ {
|
||||
switch_signal[i] = 0
|
||||
}
|
||||
|
||||
/* After each iteration of this loop, cost[k] will contain the difference
|
||||
between the minimum cost of arriving at the current byte position using
|
||||
entropy code k, and the minimum cost of arriving at the current byte
|
||||
position. This difference is capped at the block switch cost, and if it
|
||||
reaches block switch cost, it means that when we trace back from the last
|
||||
position, we need to switch here. */
|
||||
for i = 0; i < length; i++ {
|
||||
var byte_ix uint = i
|
||||
var ix uint = byte_ix * bitmaplen
|
||||
var insert_cost_ix uint = uint(data[byte_ix]) * num_histograms
|
||||
var min_cost float64 = 1e99
|
||||
var block_switch_cost float64 = block_switch_bitcost
|
||||
var k uint
|
||||
for k = 0; k < num_histograms; k++ {
|
||||
/* We are coding the symbol in data[byte_ix] with entropy code k. */
|
||||
cost[k] += insert_cost[insert_cost_ix+k]
|
||||
|
||||
if cost[k] < min_cost {
|
||||
min_cost = cost[k]
|
||||
block_id[byte_ix] = byte(k)
|
||||
}
|
||||
}
|
||||
|
||||
/* More blocks for the beginning. */
|
||||
if byte_ix < 2000 {
|
||||
block_switch_cost *= 0.77 + 0.07*float64(byte_ix)/2000
|
||||
}
|
||||
|
||||
for k = 0; k < num_histograms; k++ {
|
||||
cost[k] -= min_cost
|
||||
if cost[k] >= block_switch_cost {
|
||||
var mask byte = byte(1 << (k & 7))
|
||||
cost[k] = block_switch_cost
|
||||
assert(k>>3 < bitmaplen)
|
||||
switch_signal[ix+(k>>3)] |= mask
|
||||
/* Trace back from the last position and switch at the marked places. */
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
var byte_ix uint = length - 1
|
||||
var ix uint = byte_ix * bitmaplen
|
||||
var cur_id byte = block_id[byte_ix]
|
||||
for byte_ix > 0 {
|
||||
var mask byte = byte(1 << (cur_id & 7))
|
||||
assert(uint(cur_id)>>3 < bitmaplen)
|
||||
byte_ix--
|
||||
ix -= bitmaplen
|
||||
if switch_signal[ix+uint(cur_id>>3)]&mask != 0 {
|
||||
if cur_id != block_id[byte_ix] {
|
||||
cur_id = block_id[byte_ix]
|
||||
num_blocks++
|
||||
}
|
||||
}
|
||||
|
||||
block_id[byte_ix] = cur_id
|
||||
}
|
||||
}
|
||||
|
||||
return num_blocks
|
||||
}
|
||||
|
||||
var remapBlockIdsLiteral_kInvalidId uint16 = 256
|
||||
|
||||
func remapBlockIdsLiteral(block_ids []byte, length uint, new_id []uint16, num_histograms uint) uint {
|
||||
var next_id uint16 = 0
|
||||
var i uint
|
||||
for i = 0; i < num_histograms; i++ {
|
||||
new_id[i] = remapBlockIdsLiteral_kInvalidId
|
||||
}
|
||||
|
||||
for i = 0; i < length; i++ {
|
||||
assert(uint(block_ids[i]) < num_histograms)
|
||||
if new_id[block_ids[i]] == remapBlockIdsLiteral_kInvalidId {
|
||||
new_id[block_ids[i]] = next_id
|
||||
next_id++
|
||||
}
|
||||
}
|
||||
|
||||
for i = 0; i < length; i++ {
|
||||
block_ids[i] = byte(new_id[block_ids[i]])
|
||||
assert(uint(block_ids[i]) < num_histograms)
|
||||
}
|
||||
|
||||
assert(uint(next_id) <= num_histograms)
|
||||
return uint(next_id)
|
||||
}
|
||||
|
||||
func buildBlockHistogramsLiteral(data []byte, length uint, block_ids []byte, num_histograms uint, histograms []histogramLiteral) {
|
||||
var i uint
|
||||
clearHistogramsLiteral(histograms, num_histograms)
|
||||
for i = 0; i < length; i++ {
|
||||
histogramAddLiteral(&histograms[block_ids[i]], uint(data[i]))
|
||||
}
|
||||
}
|
||||
|
||||
var clusterBlocksLiteral_kInvalidIndex uint32 = math.MaxUint32
|
||||
|
||||
func clusterBlocksLiteral(data []byte, length uint, num_blocks uint, block_ids []byte, split *blockSplit) {
|
||||
var histogram_symbols []uint32 = make([]uint32, num_blocks)
|
||||
var block_lengths []uint32 = make([]uint32, num_blocks)
|
||||
var expected_num_clusters uint = clustersPerBatch * (num_blocks + histogramsPerBatch - 1) / histogramsPerBatch
|
||||
var all_histograms_size uint = 0
|
||||
var all_histograms_capacity uint = expected_num_clusters
|
||||
var all_histograms []histogramLiteral = make([]histogramLiteral, all_histograms_capacity)
|
||||
var cluster_size_size uint = 0
|
||||
var cluster_size_capacity uint = expected_num_clusters
|
||||
var cluster_size []uint32 = make([]uint32, cluster_size_capacity)
|
||||
var num_clusters uint = 0
|
||||
var histograms []histogramLiteral = make([]histogramLiteral, brotli_min_size_t(num_blocks, histogramsPerBatch))
|
||||
var max_num_pairs uint = histogramsPerBatch * histogramsPerBatch / 2
|
||||
var pairs_capacity uint = max_num_pairs + 1
|
||||
var pairs []histogramPair = make([]histogramPair, pairs_capacity)
|
||||
var pos uint = 0
|
||||
var clusters []uint32
|
||||
var num_final_clusters uint
|
||||
var new_index []uint32
|
||||
var i uint
|
||||
var sizes = [histogramsPerBatch]uint32{0}
|
||||
var new_clusters = [histogramsPerBatch]uint32{0}
|
||||
var symbols = [histogramsPerBatch]uint32{0}
|
||||
var remap = [histogramsPerBatch]uint32{0}
|
||||
|
||||
for i := 0; i < int(num_blocks); i++ {
|
||||
block_lengths[i] = 0
|
||||
}
|
||||
{
|
||||
var block_idx uint = 0
|
||||
for i = 0; i < length; i++ {
|
||||
assert(block_idx < num_blocks)
|
||||
block_lengths[block_idx]++
|
||||
if i+1 == length || block_ids[i] != block_ids[i+1] {
|
||||
block_idx++
|
||||
}
|
||||
}
|
||||
|
||||
assert(block_idx == num_blocks)
|
||||
}
|
||||
|
||||
for i = 0; i < num_blocks; i += histogramsPerBatch {
|
||||
var num_to_combine uint = brotli_min_size_t(num_blocks-i, histogramsPerBatch)
|
||||
var num_new_clusters uint
|
||||
var j uint
|
||||
for j = 0; j < num_to_combine; j++ {
|
||||
var k uint
|
||||
histogramClearLiteral(&histograms[j])
|
||||
for k = 0; uint32(k) < block_lengths[i+j]; k++ {
|
||||
histogramAddLiteral(&histograms[j], uint(data[pos]))
|
||||
pos++
|
||||
}
|
||||
|
||||
histograms[j].bit_cost_ = populationCostLiteral(&histograms[j])
|
||||
new_clusters[j] = uint32(j)
|
||||
symbols[j] = uint32(j)
|
||||
sizes[j] = 1
|
||||
}
|
||||
|
||||
num_new_clusters = histogramCombineLiteral(histograms, sizes[:], symbols[:], new_clusters[:], []histogramPair(pairs), num_to_combine, num_to_combine, histogramsPerBatch, max_num_pairs)
|
||||
if all_histograms_capacity < (all_histograms_size + num_new_clusters) {
|
||||
var _new_size uint
|
||||
if all_histograms_capacity == 0 {
|
||||
_new_size = all_histograms_size + num_new_clusters
|
||||
} else {
|
||||
_new_size = all_histograms_capacity
|
||||
}
|
||||
var new_array []histogramLiteral
|
||||
for _new_size < (all_histograms_size + num_new_clusters) {
|
||||
_new_size *= 2
|
||||
}
|
||||
new_array = make([]histogramLiteral, _new_size)
|
||||
if all_histograms_capacity != 0 {
|
||||
copy(new_array, all_histograms[:all_histograms_capacity])
|
||||
}
|
||||
|
||||
all_histograms = new_array
|
||||
all_histograms_capacity = _new_size
|
||||
}
|
||||
|
||||
brotli_ensure_capacity_uint32_t(&cluster_size, &cluster_size_capacity, cluster_size_size+num_new_clusters)
|
||||
for j = 0; j < num_new_clusters; j++ {
|
||||
all_histograms[all_histograms_size] = histograms[new_clusters[j]]
|
||||
all_histograms_size++
|
||||
cluster_size[cluster_size_size] = sizes[new_clusters[j]]
|
||||
cluster_size_size++
|
||||
remap[new_clusters[j]] = uint32(j)
|
||||
}
|
||||
|
||||
for j = 0; j < num_to_combine; j++ {
|
||||
histogram_symbols[i+j] = uint32(num_clusters) + remap[symbols[j]]
|
||||
}
|
||||
|
||||
num_clusters += num_new_clusters
|
||||
assert(num_clusters == cluster_size_size)
|
||||
assert(num_clusters == all_histograms_size)
|
||||
}
|
||||
|
||||
histograms = nil
|
||||
|
||||
max_num_pairs = brotli_min_size_t(64*num_clusters, (num_clusters/2)*num_clusters)
|
||||
if pairs_capacity < max_num_pairs+1 {
|
||||
pairs = nil
|
||||
pairs = make([]histogramPair, (max_num_pairs + 1))
|
||||
}
|
||||
|
||||
clusters = make([]uint32, num_clusters)
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
clusters[i] = uint32(i)
|
||||
}
|
||||
|
||||
num_final_clusters = histogramCombineLiteral(all_histograms, cluster_size, histogram_symbols, clusters, pairs, num_clusters, num_blocks, maxNumberOfBlockTypes, max_num_pairs)
|
||||
pairs = nil
|
||||
cluster_size = nil
|
||||
|
||||
new_index = make([]uint32, num_clusters)
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
new_index[i] = clusterBlocksLiteral_kInvalidIndex
|
||||
}
|
||||
pos = 0
|
||||
{
|
||||
var next_index uint32 = 0
|
||||
for i = 0; i < num_blocks; i++ {
|
||||
var histo histogramLiteral
|
||||
var j uint
|
||||
var best_out uint32
|
||||
var best_bits float64
|
||||
histogramClearLiteral(&histo)
|
||||
for j = 0; uint32(j) < block_lengths[i]; j++ {
|
||||
histogramAddLiteral(&histo, uint(data[pos]))
|
||||
pos++
|
||||
}
|
||||
|
||||
if i == 0 {
|
||||
best_out = histogram_symbols[0]
|
||||
} else {
|
||||
best_out = histogram_symbols[i-1]
|
||||
}
|
||||
best_bits = histogramBitCostDistanceLiteral(&histo, &all_histograms[best_out])
|
||||
for j = 0; j < num_final_clusters; j++ {
|
||||
var cur_bits float64 = histogramBitCostDistanceLiteral(&histo, &all_histograms[clusters[j]])
|
||||
if cur_bits < best_bits {
|
||||
best_bits = cur_bits
|
||||
best_out = clusters[j]
|
||||
}
|
||||
}
|
||||
|
||||
histogram_symbols[i] = best_out
|
||||
if new_index[best_out] == clusterBlocksLiteral_kInvalidIndex {
|
||||
new_index[best_out] = next_index
|
||||
next_index++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
clusters = nil
|
||||
all_histograms = nil
|
||||
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, num_blocks)
|
||||
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, num_blocks)
|
||||
{
|
||||
var cur_length uint32 = 0
|
||||
var block_idx uint = 0
|
||||
var max_type byte = 0
|
||||
for i = 0; i < num_blocks; i++ {
|
||||
cur_length += block_lengths[i]
|
||||
if i+1 == num_blocks || histogram_symbols[i] != histogram_symbols[i+1] {
|
||||
var id byte = byte(new_index[histogram_symbols[i]])
|
||||
split.types[block_idx] = id
|
||||
split.lengths[block_idx] = cur_length
|
||||
max_type = brotli_max_uint8_t(max_type, id)
|
||||
cur_length = 0
|
||||
block_idx++
|
||||
}
|
||||
}
|
||||
|
||||
split.num_blocks = block_idx
|
||||
split.num_types = uint(max_type) + 1
|
||||
}
|
||||
|
||||
new_index = nil
|
||||
block_lengths = nil
|
||||
histogram_symbols = nil
|
||||
}
|
||||
|
||||
func splitByteVectorLiteral(data []byte, length uint, literals_per_histogram uint, max_histograms uint, sampling_stride_length uint, block_switch_cost float64, params *encoderParams, split *blockSplit) {
|
||||
var data_size uint = histogramDataSizeLiteral()
|
||||
var num_histograms uint = length/literals_per_histogram + 1
|
||||
var histograms []histogramLiteral
|
||||
if num_histograms > max_histograms {
|
||||
num_histograms = max_histograms
|
||||
}
|
||||
|
||||
if length == 0 {
|
||||
split.num_types = 1
|
||||
return
|
||||
} else if length < kMinLengthForBlockSplitting {
|
||||
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, split.num_blocks+1)
|
||||
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, split.num_blocks+1)
|
||||
split.num_types = 1
|
||||
split.types[split.num_blocks] = 0
|
||||
split.lengths[split.num_blocks] = uint32(length)
|
||||
split.num_blocks++
|
||||
return
|
||||
}
|
||||
|
||||
histograms = make([]histogramLiteral, num_histograms)
|
||||
|
||||
/* Find good entropy codes. */
|
||||
initialEntropyCodesLiteral(data, length, sampling_stride_length, num_histograms, histograms)
|
||||
|
||||
refineEntropyCodesLiteral(data, length, sampling_stride_length, num_histograms, histograms)
|
||||
{
|
||||
var block_ids []byte = make([]byte, length)
|
||||
var num_blocks uint = 0
|
||||
var bitmaplen uint = (num_histograms + 7) >> 3
|
||||
var insert_cost []float64 = make([]float64, (data_size * num_histograms))
|
||||
var cost []float64 = make([]float64, num_histograms)
|
||||
var switch_signal []byte = make([]byte, (length * bitmaplen))
|
||||
var new_id []uint16 = make([]uint16, num_histograms)
|
||||
var iters uint
|
||||
if params.quality < hqZopflificationQuality {
|
||||
iters = 3
|
||||
} else {
|
||||
iters = 10
|
||||
}
|
||||
/* Find a good path through literals with the good entropy codes. */
|
||||
|
||||
var i uint
|
||||
for i = 0; i < iters; i++ {
|
||||
num_blocks = findBlocksLiteral(data, length, block_switch_cost, num_histograms, histograms, insert_cost, cost, switch_signal, block_ids)
|
||||
num_histograms = remapBlockIdsLiteral(block_ids, length, new_id, num_histograms)
|
||||
buildBlockHistogramsLiteral(data, length, block_ids, num_histograms, histograms)
|
||||
}
|
||||
|
||||
insert_cost = nil
|
||||
cost = nil
|
||||
switch_signal = nil
|
||||
new_id = nil
|
||||
histograms = nil
|
||||
clusterBlocksLiteral(data, length, num_blocks, block_ids, split)
|
||||
block_ids = nil
|
||||
}
|
||||
}
|
||||
+1539
File diff suppressed because it is too large.
Load diff
+30
@@ -0,0 +1,30 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Functions for clustering similar histograms together. */
|
||||
|
||||
type histogramPair struct {
|
||||
idx1 uint32
|
||||
idx2 uint32
|
||||
cost_combo float64
|
||||
cost_diff float64
|
||||
}
|
||||
|
||||
func histogramPairIsLess(p1 *histogramPair, p2 *histogramPair) bool {
|
||||
if p1.cost_diff != p2.cost_diff {
|
||||
return p1.cost_diff > p2.cost_diff
|
||||
}
|
||||
|
||||
return (p1.idx2 - p1.idx1) > (p2.idx2 - p2.idx1)
|
||||
}
|
||||
|
||||
/* Returns entropy reduction of the context map when we combine two clusters. */
|
||||
func clusterCostDiff(size_a uint, size_b uint) float64 {
|
||||
var size_c uint = size_a + size_b
|
||||
return float64(size_a)*fastLog2(size_a) + float64(size_b)*fastLog2(size_b) - float64(size_c)*fastLog2(size_c)
|
||||
}
|
||||
+164
@@ -0,0 +1,164 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Computes the bit cost reduction by combining out[idx1] and out[idx2] and if
|
||||
it is below a threshold, stores the pair (idx1, idx2) in the *pairs queue. */
|
||||
func compareAndPushToQueueCommand(out []histogramCommand, cluster_size []uint32, idx1 uint32, idx2 uint32, max_num_pairs uint, pairs []histogramPair, num_pairs *uint) {
|
||||
var is_good_pair bool = false
|
||||
var p histogramPair
|
||||
p.idx2 = 0
|
||||
p.idx1 = p.idx2
|
||||
p.cost_combo = 0
|
||||
p.cost_diff = p.cost_combo
|
||||
if idx1 == idx2 {
|
||||
return
|
||||
}
|
||||
|
||||
if idx2 < idx1 {
|
||||
var t uint32 = idx2
|
||||
idx2 = idx1
|
||||
idx1 = t
|
||||
}
|
||||
|
||||
p.idx1 = idx1
|
||||
p.idx2 = idx2
|
||||
p.cost_diff = 0.5 * clusterCostDiff(uint(cluster_size[idx1]), uint(cluster_size[idx2]))
|
||||
p.cost_diff -= out[idx1].bit_cost_
|
||||
p.cost_diff -= out[idx2].bit_cost_
|
||||
|
||||
if out[idx1].total_count_ == 0 {
|
||||
p.cost_combo = out[idx2].bit_cost_
|
||||
is_good_pair = true
|
||||
} else if out[idx2].total_count_ == 0 {
|
||||
p.cost_combo = out[idx1].bit_cost_
|
||||
is_good_pair = true
|
||||
} else {
|
||||
var threshold float64
|
||||
if *num_pairs == 0 {
|
||||
threshold = 1e99
|
||||
} else {
|
||||
threshold = brotli_max_double(0.0, pairs[0].cost_diff)
|
||||
}
|
||||
var combo histogramCommand = out[idx1]
|
||||
var cost_combo float64
|
||||
histogramAddHistogramCommand(&combo, &out[idx2])
|
||||
cost_combo = populationCostCommand(&combo)
|
||||
if cost_combo < threshold-p.cost_diff {
|
||||
p.cost_combo = cost_combo
|
||||
is_good_pair = true
|
||||
}
|
||||
}
|
||||
|
||||
if is_good_pair {
|
||||
p.cost_diff += p.cost_combo
|
||||
if *num_pairs > 0 && histogramPairIsLess(&pairs[0], &p) {
|
||||
/* Replace the top of the queue if needed. */
|
||||
if *num_pairs < max_num_pairs {
|
||||
pairs[*num_pairs] = pairs[0]
|
||||
(*num_pairs)++
|
||||
}
|
||||
|
||||
pairs[0] = p
|
||||
} else if *num_pairs < max_num_pairs {
|
||||
pairs[*num_pairs] = p
|
||||
(*num_pairs)++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func histogramCombineCommand(out []histogramCommand, cluster_size []uint32, symbols []uint32, clusters []uint32, pairs []histogramPair, num_clusters uint, symbols_size uint, max_clusters uint, max_num_pairs uint) uint {
|
||||
var cost_diff_threshold float64 = 0.0
|
||||
var min_cluster_size uint = 1
|
||||
var num_pairs uint = 0
|
||||
{
|
||||
/* We maintain a vector of histogram pairs, with the property that the pair
|
||||
with the maximum bit cost reduction is the first. */
|
||||
var idx1 uint
|
||||
for idx1 = 0; idx1 < num_clusters; idx1++ {
|
||||
var idx2 uint
|
||||
for idx2 = idx1 + 1; idx2 < num_clusters; idx2++ {
|
||||
compareAndPushToQueueCommand(out, cluster_size, clusters[idx1], clusters[idx2], max_num_pairs, pairs[0:], &num_pairs)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for num_clusters > min_cluster_size {
|
||||
var best_idx1 uint32
|
||||
var best_idx2 uint32
|
||||
var i uint
|
||||
if pairs[0].cost_diff >= cost_diff_threshold {
|
||||
cost_diff_threshold = 1e99
|
||||
min_cluster_size = max_clusters
|
||||
continue
|
||||
}
|
||||
|
||||
/* Take the best pair from the top of heap. */
|
||||
best_idx1 = pairs[0].idx1
|
||||
|
||||
best_idx2 = pairs[0].idx2
|
||||
histogramAddHistogramCommand(&out[best_idx1], &out[best_idx2])
|
||||
out[best_idx1].bit_cost_ = pairs[0].cost_combo
|
||||
cluster_size[best_idx1] += cluster_size[best_idx2]
|
||||
for i = 0; i < symbols_size; i++ {
|
||||
if symbols[i] == best_idx2 {
|
||||
symbols[i] = best_idx1
|
||||
}
|
||||
}
|
||||
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
if clusters[i] == best_idx2 {
|
||||
copy(clusters[i:], clusters[i+1:][:num_clusters-i-1])
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
num_clusters--
|
||||
{
|
||||
/* Remove pairs intersecting the just combined best pair. */
|
||||
var copy_to_idx uint = 0
|
||||
for i = 0; i < num_pairs; i++ {
|
||||
var p *histogramPair = &pairs[i]
|
||||
if p.idx1 == best_idx1 || p.idx2 == best_idx1 || p.idx1 == best_idx2 || p.idx2 == best_idx2 {
|
||||
/* Remove invalid pair from the queue. */
|
||||
continue
|
||||
}
|
||||
|
||||
if histogramPairIsLess(&pairs[0], p) {
|
||||
/* Replace the top of the queue if needed. */
|
||||
var front histogramPair = pairs[0]
|
||||
pairs[0] = *p
|
||||
pairs[copy_to_idx] = front
|
||||
} else {
|
||||
pairs[copy_to_idx] = *p
|
||||
}
|
||||
|
||||
copy_to_idx++
|
||||
}
|
||||
|
||||
num_pairs = copy_to_idx
|
||||
}
|
||||
|
||||
/* Push new pairs formed with the combined histogram to the heap. */
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
compareAndPushToQueueCommand(out, cluster_size, best_idx1, clusters[i], max_num_pairs, pairs[0:], &num_pairs)
|
||||
}
|
||||
}
|
||||
|
||||
return num_clusters
|
||||
}
|
||||
|
||||
/* What is the bit cost of moving histogram from cur_symbol to candidate. */
|
||||
func histogramBitCostDistanceCommand(histogram *histogramCommand, candidate *histogramCommand) float64 {
|
||||
if histogram.total_count_ == 0 {
|
||||
return 0.0
|
||||
} else {
|
||||
var tmp histogramCommand = *histogram
|
||||
histogramAddHistogramCommand(&tmp, candidate)
|
||||
return populationCostCommand(&tmp) - candidate.bit_cost_
|
||||
}
|
||||
}
|
||||
+326
@@ -0,0 +1,326 @@
|
||||
package brotli
|
||||
|
||||
import "math"
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Computes the bit cost reduction by combining out[idx1] and out[idx2] and if
|
||||
it is below a threshold, stores the pair (idx1, idx2) in the *pairs queue. */
|
||||
func compareAndPushToQueueDistance(out []histogramDistance, cluster_size []uint32, idx1 uint32, idx2 uint32, max_num_pairs uint, pairs []histogramPair, num_pairs *uint) {
|
||||
var is_good_pair bool = false
|
||||
var p histogramPair
|
||||
p.idx2 = 0
|
||||
p.idx1 = p.idx2
|
||||
p.cost_combo = 0
|
||||
p.cost_diff = p.cost_combo
|
||||
if idx1 == idx2 {
|
||||
return
|
||||
}
|
||||
|
||||
if idx2 < idx1 {
|
||||
var t uint32 = idx2
|
||||
idx2 = idx1
|
||||
idx1 = t
|
||||
}
|
||||
|
||||
p.idx1 = idx1
|
||||
p.idx2 = idx2
|
||||
p.cost_diff = 0.5 * clusterCostDiff(uint(cluster_size[idx1]), uint(cluster_size[idx2]))
|
||||
p.cost_diff -= out[idx1].bit_cost_
|
||||
p.cost_diff -= out[idx2].bit_cost_
|
||||
|
||||
if out[idx1].total_count_ == 0 {
|
||||
p.cost_combo = out[idx2].bit_cost_
|
||||
is_good_pair = true
|
||||
} else if out[idx2].total_count_ == 0 {
|
||||
p.cost_combo = out[idx1].bit_cost_
|
||||
is_good_pair = true
|
||||
} else {
|
||||
var threshold float64
|
||||
if *num_pairs == 0 {
|
||||
threshold = 1e99
|
||||
} else {
|
||||
threshold = brotli_max_double(0.0, pairs[0].cost_diff)
|
||||
}
|
||||
var combo histogramDistance = out[idx1]
|
||||
var cost_combo float64
|
||||
histogramAddHistogramDistance(&combo, &out[idx2])
|
||||
cost_combo = populationCostDistance(&combo)
|
||||
if cost_combo < threshold-p.cost_diff {
|
||||
p.cost_combo = cost_combo
|
||||
is_good_pair = true
|
||||
}
|
||||
}
|
||||
|
||||
if is_good_pair {
|
||||
p.cost_diff += p.cost_combo
|
||||
if *num_pairs > 0 && histogramPairIsLess(&pairs[0], &p) {
|
||||
/* Replace the top of the queue if needed. */
|
||||
if *num_pairs < max_num_pairs {
|
||||
pairs[*num_pairs] = pairs[0]
|
||||
(*num_pairs)++
|
||||
}
|
||||
|
||||
pairs[0] = p
|
||||
} else if *num_pairs < max_num_pairs {
|
||||
pairs[*num_pairs] = p
|
||||
(*num_pairs)++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func histogramCombineDistance(out []histogramDistance, cluster_size []uint32, symbols []uint32, clusters []uint32, pairs []histogramPair, num_clusters uint, symbols_size uint, max_clusters uint, max_num_pairs uint) uint {
|
||||
var cost_diff_threshold float64 = 0.0
|
||||
var min_cluster_size uint = 1
|
||||
var num_pairs uint = 0
|
||||
{
|
||||
/* We maintain a vector of histogram pairs, with the property that the pair
|
||||
with the maximum bit cost reduction is the first. */
|
||||
var idx1 uint
|
||||
for idx1 = 0; idx1 < num_clusters; idx1++ {
|
||||
var idx2 uint
|
||||
for idx2 = idx1 + 1; idx2 < num_clusters; idx2++ {
|
||||
compareAndPushToQueueDistance(out, cluster_size, clusters[idx1], clusters[idx2], max_num_pairs, pairs[0:], &num_pairs)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for num_clusters > min_cluster_size {
|
||||
var best_idx1 uint32
|
||||
var best_idx2 uint32
|
||||
var i uint
|
||||
if pairs[0].cost_diff >= cost_diff_threshold {
|
||||
cost_diff_threshold = 1e99
|
||||
min_cluster_size = max_clusters
|
||||
continue
|
||||
}
|
||||
|
||||
/* Take the best pair from the top of heap. */
|
||||
best_idx1 = pairs[0].idx1
|
||||
|
||||
best_idx2 = pairs[0].idx2
|
||||
histogramAddHistogramDistance(&out[best_idx1], &out[best_idx2])
|
||||
out[best_idx1].bit_cost_ = pairs[0].cost_combo
|
||||
cluster_size[best_idx1] += cluster_size[best_idx2]
|
||||
for i = 0; i < symbols_size; i++ {
|
||||
if symbols[i] == best_idx2 {
|
||||
symbols[i] = best_idx1
|
||||
}
|
||||
}
|
||||
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
if clusters[i] == best_idx2 {
|
||||
copy(clusters[i:], clusters[i+1:][:num_clusters-i-1])
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
num_clusters--
|
||||
{
|
||||
/* Remove pairs intersecting the just combined best pair. */
|
||||
var copy_to_idx uint = 0
|
||||
for i = 0; i < num_pairs; i++ {
|
||||
var p *histogramPair = &pairs[i]
|
||||
if p.idx1 == best_idx1 || p.idx2 == best_idx1 || p.idx1 == best_idx2 || p.idx2 == best_idx2 {
|
||||
/* Remove invalid pair from the queue. */
|
||||
continue
|
||||
}
|
||||
|
||||
if histogramPairIsLess(&pairs[0], p) {
|
||||
/* Replace the top of the queue if needed. */
|
||||
var front histogramPair = pairs[0]
|
||||
pairs[0] = *p
|
||||
pairs[copy_to_idx] = front
|
||||
} else {
|
||||
pairs[copy_to_idx] = *p
|
||||
}
|
||||
|
||||
copy_to_idx++
|
||||
}
|
||||
|
||||
num_pairs = copy_to_idx
|
||||
}
|
||||
|
||||
/* Push new pairs formed with the combined histogram to the heap. */
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
compareAndPushToQueueDistance(out, cluster_size, best_idx1, clusters[i], max_num_pairs, pairs[0:], &num_pairs)
|
||||
}
|
||||
}
|
||||
|
||||
return num_clusters
|
||||
}
|
||||
|
||||
/* What is the bit cost of moving histogram from cur_symbol to candidate. */
|
||||
func histogramBitCostDistanceDistance(histogram *histogramDistance, candidate *histogramDistance) float64 {
|
||||
if histogram.total_count_ == 0 {
|
||||
return 0.0
|
||||
} else {
|
||||
var tmp histogramDistance = *histogram
|
||||
histogramAddHistogramDistance(&tmp, candidate)
|
||||
return populationCostDistance(&tmp) - candidate.bit_cost_
|
||||
}
|
||||
}
|
||||
|
||||
/* Find the best 'out' histogram for each of the 'in' histograms.
|
||||
When called, clusters[0..num_clusters) contains the unique values from
|
||||
symbols[0..in_size), but this property is not preserved in this function.
|
||||
Note: we assume that out[]->bit_cost_ is already up-to-date. */
|
||||
func histogramRemapDistance(in []histogramDistance, in_size uint, clusters []uint32, num_clusters uint, out []histogramDistance, symbols []uint32) {
|
||||
var i uint
|
||||
for i = 0; i < in_size; i++ {
|
||||
var best_out uint32
|
||||
if i == 0 {
|
||||
best_out = symbols[0]
|
||||
} else {
|
||||
best_out = symbols[i-1]
|
||||
}
|
||||
var best_bits float64 = histogramBitCostDistanceDistance(&in[i], &out[best_out])
|
||||
var j uint
|
||||
for j = 0; j < num_clusters; j++ {
|
||||
var cur_bits float64 = histogramBitCostDistanceDistance(&in[i], &out[clusters[j]])
|
||||
if cur_bits < best_bits {
|
||||
best_bits = cur_bits
|
||||
best_out = clusters[j]
|
||||
}
|
||||
}
|
||||
|
||||
symbols[i] = best_out
|
||||
}
|
||||
|
||||
/* Recompute each out based on raw and symbols. */
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
histogramClearDistance(&out[clusters[i]])
|
||||
}
|
||||
|
||||
for i = 0; i < in_size; i++ {
|
||||
histogramAddHistogramDistance(&out[symbols[i]], &in[i])
|
||||
}
|
||||
}
|
||||
|
||||
/* Reorders elements of the out[0..length) array and changes values in
|
||||
symbols[0..length) array in the following way:
|
||||
* when called, symbols[] contains indexes into out[], and has N unique
|
||||
values (possibly N < length)
|
||||
* on return, symbols'[i] = f(symbols[i]) and
|
||||
out'[symbols'[i]] = out[symbols[i]], for each 0 <= i < length,
|
||||
where f is a bijection between the range of symbols[] and [0..N), and
|
||||
the first occurrences of values in symbols'[i] come in consecutive
|
||||
increasing order.
|
||||
Returns N, the number of unique values in symbols[]. */
|
||||
|
||||
var histogramReindexDistance_kInvalidIndex uint32 = math.MaxUint32
|
||||
|
||||
func histogramReindexDistance(out []histogramDistance, symbols []uint32, length uint) uint {
|
||||
var new_index []uint32 = make([]uint32, length)
|
||||
var next_index uint32
|
||||
var tmp []histogramDistance
|
||||
var i uint
|
||||
for i = 0; i < length; i++ {
|
||||
new_index[i] = histogramReindexDistance_kInvalidIndex
|
||||
}
|
||||
|
||||
next_index = 0
|
||||
for i = 0; i < length; i++ {
|
||||
if new_index[symbols[i]] == histogramReindexDistance_kInvalidIndex {
|
||||
new_index[symbols[i]] = next_index
|
||||
next_index++
|
||||
}
|
||||
}
|
||||
|
||||
/* TODO: by using idea of "cycle-sort" we can avoid allocation of
|
||||
tmp and reduce the number of copying by the factor of 2. */
|
||||
tmp = make([]histogramDistance, next_index)
|
||||
|
||||
next_index = 0
|
||||
for i = 0; i < length; i++ {
|
||||
if new_index[symbols[i]] == next_index {
|
||||
tmp[next_index] = out[symbols[i]]
|
||||
next_index++
|
||||
}
|
||||
|
||||
symbols[i] = new_index[symbols[i]]
|
||||
}
|
||||
|
||||
new_index = nil
|
||||
for i = 0; uint32(i) < next_index; i++ {
|
||||
out[i] = tmp[i]
|
||||
}
|
||||
|
||||
tmp = nil
|
||||
return uint(next_index)
|
||||
}
|
||||
|
||||
func clusterHistogramsDistance(in []histogramDistance, in_size uint, max_histograms uint, out []histogramDistance, out_size *uint, histogram_symbols []uint32) {
|
||||
var cluster_size []uint32 = make([]uint32, in_size)
|
||||
var clusters []uint32 = make([]uint32, in_size)
|
||||
var num_clusters uint = 0
|
||||
var max_input_histograms uint = 64
|
||||
var pairs_capacity uint = max_input_histograms * max_input_histograms / 2
|
||||
var pairs []histogramPair = make([]histogramPair, (pairs_capacity + 1))
|
||||
var i uint
|
||||
|
||||
/* For the first pass of clustering, we allow all pairs. */
|
||||
for i = 0; i < in_size; i++ {
|
||||
cluster_size[i] = 1
|
||||
}
|
||||
|
||||
for i = 0; i < in_size; i++ {
|
||||
out[i] = in[i]
|
||||
out[i].bit_cost_ = populationCostDistance(&in[i])
|
||||
histogram_symbols[i] = uint32(i)
|
||||
}
|
||||
|
||||
for i = 0; i < in_size; i += max_input_histograms {
|
||||
var num_to_combine uint = brotli_min_size_t(in_size-i, max_input_histograms)
|
||||
var num_new_clusters uint
|
||||
var j uint
|
||||
for j = 0; j < num_to_combine; j++ {
|
||||
clusters[num_clusters+j] = uint32(i + j)
|
||||
}
|
||||
|
||||
num_new_clusters = histogramCombineDistance(out, cluster_size, histogram_symbols[i:], clusters[num_clusters:], pairs, num_to_combine, num_to_combine, max_histograms, pairs_capacity)
|
||||
num_clusters += num_new_clusters
|
||||
}
|
||||
{
|
||||
/* For the second pass, we limit the total number of histogram pairs.
|
||||
After this limit is reached, we only keep searching for the best pair. */
|
||||
var max_num_pairs uint = brotli_min_size_t(64*num_clusters, (num_clusters/2)*num_clusters)
|
||||
if pairs_capacity < (max_num_pairs + 1) {
|
||||
var _new_size uint
|
||||
if pairs_capacity == 0 {
|
||||
_new_size = max_num_pairs + 1
|
||||
} else {
|
||||
_new_size = pairs_capacity
|
||||
}
|
||||
var new_array []histogramPair
|
||||
for _new_size < (max_num_pairs + 1) {
|
||||
_new_size *= 2
|
||||
}
|
||||
new_array = make([]histogramPair, _new_size)
|
||||
if pairs_capacity != 0 {
|
||||
copy(new_array, pairs[:pairs_capacity])
|
||||
}
|
||||
|
||||
pairs = new_array
|
||||
pairs_capacity = _new_size
|
||||
}
|
||||
|
||||
/* Collapse similar histograms. */
|
||||
num_clusters = histogramCombineDistance(out, cluster_size, histogram_symbols, clusters, pairs, num_clusters, in_size, max_histograms, max_num_pairs)
|
||||
}
|
||||
|
||||
pairs = nil
|
||||
cluster_size = nil
|
||||
|
||||
/* Find the optimal map from original histograms to the final ones. */
|
||||
histogramRemapDistance(in, in_size, clusters, num_clusters, out, histogram_symbols)
|
||||
|
||||
clusters = nil
|
||||
|
||||
/* Convert the context map to a canonical form. */
|
||||
*out_size = histogramReindexDistance(out, histogram_symbols, in_size)
|
||||
}
|
||||
+326
@@ -0,0 +1,326 @@
|
||||
package brotli
|
||||
|
||||
import "math"
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Computes the bit cost reduction by combining out[idx1] and out[idx2] and if
|
||||
it is below a threshold, stores the pair (idx1, idx2) in the *pairs queue. */
|
||||
func compareAndPushToQueueLiteral(out []histogramLiteral, cluster_size []uint32, idx1 uint32, idx2 uint32, max_num_pairs uint, pairs []histogramPair, num_pairs *uint) {
|
||||
var is_good_pair bool = false
|
||||
var p histogramPair
|
||||
p.idx2 = 0
|
||||
p.idx1 = p.idx2
|
||||
p.cost_combo = 0
|
||||
p.cost_diff = p.cost_combo
|
||||
if idx1 == idx2 {
|
||||
return
|
||||
}
|
||||
|
||||
if idx2 < idx1 {
|
||||
var t uint32 = idx2
|
||||
idx2 = idx1
|
||||
idx1 = t
|
||||
}
|
||||
|
||||
p.idx1 = idx1
|
||||
p.idx2 = idx2
|
||||
p.cost_diff = 0.5 * clusterCostDiff(uint(cluster_size[idx1]), uint(cluster_size[idx2]))
|
||||
p.cost_diff -= out[idx1].bit_cost_
|
||||
p.cost_diff -= out[idx2].bit_cost_
|
||||
|
||||
if out[idx1].total_count_ == 0 {
|
||||
p.cost_combo = out[idx2].bit_cost_
|
||||
is_good_pair = true
|
||||
} else if out[idx2].total_count_ == 0 {
|
||||
p.cost_combo = out[idx1].bit_cost_
|
||||
is_good_pair = true
|
||||
} else {
|
||||
var threshold float64
|
||||
if *num_pairs == 0 {
|
||||
threshold = 1e99
|
||||
} else {
|
||||
threshold = brotli_max_double(0.0, pairs[0].cost_diff)
|
||||
}
|
||||
var combo histogramLiteral = out[idx1]
|
||||
var cost_combo float64
|
||||
histogramAddHistogramLiteral(&combo, &out[idx2])
|
||||
cost_combo = populationCostLiteral(&combo)
|
||||
if cost_combo < threshold-p.cost_diff {
|
||||
p.cost_combo = cost_combo
|
||||
is_good_pair = true
|
||||
}
|
||||
}
|
||||
|
||||
if is_good_pair {
|
||||
p.cost_diff += p.cost_combo
|
||||
if *num_pairs > 0 && histogramPairIsLess(&pairs[0], &p) {
|
||||
/* Replace the top of the queue if needed. */
|
||||
if *num_pairs < max_num_pairs {
|
||||
pairs[*num_pairs] = pairs[0]
|
||||
(*num_pairs)++
|
||||
}
|
||||
|
||||
pairs[0] = p
|
||||
} else if *num_pairs < max_num_pairs {
|
||||
pairs[*num_pairs] = p
|
||||
(*num_pairs)++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func histogramCombineLiteral(out []histogramLiteral, cluster_size []uint32, symbols []uint32, clusters []uint32, pairs []histogramPair, num_clusters uint, symbols_size uint, max_clusters uint, max_num_pairs uint) uint {
|
||||
var cost_diff_threshold float64 = 0.0
|
||||
var min_cluster_size uint = 1
|
||||
var num_pairs uint = 0
|
||||
{
|
||||
/* We maintain a vector of histogram pairs, with the property that the pair
|
||||
with the maximum bit cost reduction is the first. */
|
||||
var idx1 uint
|
||||
for idx1 = 0; idx1 < num_clusters; idx1++ {
|
||||
var idx2 uint
|
||||
for idx2 = idx1 + 1; idx2 < num_clusters; idx2++ {
|
||||
compareAndPushToQueueLiteral(out, cluster_size, clusters[idx1], clusters[idx2], max_num_pairs, pairs[0:], &num_pairs)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for num_clusters > min_cluster_size {
|
||||
var best_idx1 uint32
|
||||
var best_idx2 uint32
|
||||
var i uint
|
||||
if pairs[0].cost_diff >= cost_diff_threshold {
|
||||
cost_diff_threshold = 1e99
|
||||
min_cluster_size = max_clusters
|
||||
continue
|
||||
}
|
||||
|
||||
/* Take the best pair from the top of heap. */
|
||||
best_idx1 = pairs[0].idx1
|
||||
|
||||
best_idx2 = pairs[0].idx2
|
||||
histogramAddHistogramLiteral(&out[best_idx1], &out[best_idx2])
|
||||
out[best_idx1].bit_cost_ = pairs[0].cost_combo
|
||||
cluster_size[best_idx1] += cluster_size[best_idx2]
|
||||
for i = 0; i < symbols_size; i++ {
|
||||
if symbols[i] == best_idx2 {
|
||||
symbols[i] = best_idx1
|
||||
}
|
||||
}
|
||||
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
if clusters[i] == best_idx2 {
|
||||
copy(clusters[i:], clusters[i+1:][:num_clusters-i-1])
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
num_clusters--
|
||||
{
|
||||
/* Remove pairs intersecting the just combined best pair. */
|
||||
var copy_to_idx uint = 0
|
||||
for i = 0; i < num_pairs; i++ {
|
||||
var p *histogramPair = &pairs[i]
|
||||
if p.idx1 == best_idx1 || p.idx2 == best_idx1 || p.idx1 == best_idx2 || p.idx2 == best_idx2 {
|
||||
/* Remove invalid pair from the queue. */
|
||||
continue
|
||||
}
|
||||
|
||||
if histogramPairIsLess(&pairs[0], p) {
|
||||
/* Replace the top of the queue if needed. */
|
||||
var front histogramPair = pairs[0]
|
||||
pairs[0] = *p
|
||||
pairs[copy_to_idx] = front
|
||||
} else {
|
||||
pairs[copy_to_idx] = *p
|
||||
}
|
||||
|
||||
copy_to_idx++
|
||||
}
|
||||
|
||||
num_pairs = copy_to_idx
|
||||
}
|
||||
|
||||
/* Push new pairs formed with the combined histogram to the heap. */
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
compareAndPushToQueueLiteral(out, cluster_size, best_idx1, clusters[i], max_num_pairs, pairs[0:], &num_pairs)
|
||||
}
|
||||
}
|
||||
|
||||
return num_clusters
|
||||
}
|
||||
|
||||
/* What is the bit cost of moving histogram from cur_symbol to candidate. */
|
||||
func histogramBitCostDistanceLiteral(histogram *histogramLiteral, candidate *histogramLiteral) float64 {
|
||||
if histogram.total_count_ == 0 {
|
||||
return 0.0
|
||||
} else {
|
||||
var tmp histogramLiteral = *histogram
|
||||
histogramAddHistogramLiteral(&tmp, candidate)
|
||||
return populationCostLiteral(&tmp) - candidate.bit_cost_
|
||||
}
|
||||
}
|
||||
|
||||
/* Find the best 'out' histogram for each of the 'in' histograms.
|
||||
When called, clusters[0..num_clusters) contains the unique values from
|
||||
symbols[0..in_size), but this property is not preserved in this function.
|
||||
Note: we assume that out[]->bit_cost_ is already up-to-date. */
|
||||
func histogramRemapLiteral(in []histogramLiteral, in_size uint, clusters []uint32, num_clusters uint, out []histogramLiteral, symbols []uint32) {
|
||||
var i uint
|
||||
for i = 0; i < in_size; i++ {
|
||||
var best_out uint32
|
||||
if i == 0 {
|
||||
best_out = symbols[0]
|
||||
} else {
|
||||
best_out = symbols[i-1]
|
||||
}
|
||||
var best_bits float64 = histogramBitCostDistanceLiteral(&in[i], &out[best_out])
|
||||
var j uint
|
||||
for j = 0; j < num_clusters; j++ {
|
||||
var cur_bits float64 = histogramBitCostDistanceLiteral(&in[i], &out[clusters[j]])
|
||||
if cur_bits < best_bits {
|
||||
best_bits = cur_bits
|
||||
best_out = clusters[j]
|
||||
}
|
||||
}
|
||||
|
||||
symbols[i] = best_out
|
||||
}
|
||||
|
||||
/* Recompute each out based on raw and symbols. */
|
||||
for i = 0; i < num_clusters; i++ {
|
||||
histogramClearLiteral(&out[clusters[i]])
|
||||
}
|
||||
|
||||
for i = 0; i < in_size; i++ {
|
||||
histogramAddHistogramLiteral(&out[symbols[i]], &in[i])
|
||||
}
|
||||
}
|
||||
|
||||
/* Reorders elements of the out[0..length) array and changes values in
|
||||
symbols[0..length) array in the following way:
|
||||
* when called, symbols[] contains indexes into out[], and has N unique
|
||||
values (possibly N < length)
|
||||
* on return, symbols'[i] = f(symbols[i]) and
|
||||
out'[symbols'[i]] = out[symbols[i]], for each 0 <= i < length,
|
||||
where f is a bijection between the range of symbols[] and [0..N), and
|
||||
the first occurrences of values in symbols'[i] come in consecutive
|
||||
increasing order.
|
||||
Returns N, the number of unique values in symbols[]. */
|
||||
|
||||
var histogramReindexLiteral_kInvalidIndex uint32 = math.MaxUint32
|
||||
|
||||
func histogramReindexLiteral(out []histogramLiteral, symbols []uint32, length uint) uint {
|
||||
var new_index []uint32 = make([]uint32, length)
|
||||
var next_index uint32
|
||||
var tmp []histogramLiteral
|
||||
var i uint
|
||||
for i = 0; i < length; i++ {
|
||||
new_index[i] = histogramReindexLiteral_kInvalidIndex
|
||||
}
|
||||
|
||||
next_index = 0
|
||||
for i = 0; i < length; i++ {
|
||||
if new_index[symbols[i]] == histogramReindexLiteral_kInvalidIndex {
|
||||
new_index[symbols[i]] = next_index
|
||||
next_index++
|
||||
}
|
||||
}
|
||||
|
||||
/* TODO: by using idea of "cycle-sort" we can avoid allocation of
|
||||
tmp and reduce the number of copying by the factor of 2. */
|
||||
tmp = make([]histogramLiteral, next_index)
|
||||
|
||||
next_index = 0
|
||||
for i = 0; i < length; i++ {
|
||||
if new_index[symbols[i]] == next_index {
|
||||
tmp[next_index] = out[symbols[i]]
|
||||
next_index++
|
||||
}
|
||||
|
||||
symbols[i] = new_index[symbols[i]]
|
||||
}
|
||||
|
||||
new_index = nil
|
||||
for i = 0; uint32(i) < next_index; i++ {
|
||||
out[i] = tmp[i]
|
||||
}
|
||||
|
||||
tmp = nil
|
||||
return uint(next_index)
|
||||
}
|
||||
|
||||
func clusterHistogramsLiteral(in []histogramLiteral, in_size uint, max_histograms uint, out []histogramLiteral, out_size *uint, histogram_symbols []uint32) {
|
||||
var cluster_size []uint32 = make([]uint32, in_size)
|
||||
var clusters []uint32 = make([]uint32, in_size)
|
||||
var num_clusters uint = 0
|
||||
var max_input_histograms uint = 64
|
||||
var pairs_capacity uint = max_input_histograms * max_input_histograms / 2
|
||||
var pairs []histogramPair = make([]histogramPair, (pairs_capacity + 1))
|
||||
var i uint
|
||||
|
||||
/* For the first pass of clustering, we allow all pairs. */
|
||||
for i = 0; i < in_size; i++ {
|
||||
cluster_size[i] = 1
|
||||
}
|
||||
|
||||
for i = 0; i < in_size; i++ {
|
||||
out[i] = in[i]
|
||||
out[i].bit_cost_ = populationCostLiteral(&in[i])
|
||||
histogram_symbols[i] = uint32(i)
|
||||
}
|
||||
|
||||
for i = 0; i < in_size; i += max_input_histograms {
|
||||
var num_to_combine uint = brotli_min_size_t(in_size-i, max_input_histograms)
|
||||
var num_new_clusters uint
|
||||
var j uint
|
||||
for j = 0; j < num_to_combine; j++ {
|
||||
clusters[num_clusters+j] = uint32(i + j)
|
||||
}
|
||||
|
||||
num_new_clusters = histogramCombineLiteral(out, cluster_size, histogram_symbols[i:], clusters[num_clusters:], pairs, num_to_combine, num_to_combine, max_histograms, pairs_capacity)
|
||||
num_clusters += num_new_clusters
|
||||
}
|
||||
{
|
||||
/* For the second pass, we limit the total number of histogram pairs.
|
||||
After this limit is reached, we only keep searching for the best pair. */
|
||||
var max_num_pairs uint = brotli_min_size_t(64*num_clusters, (num_clusters/2)*num_clusters)
|
||||
if pairs_capacity < (max_num_pairs + 1) {
|
||||
var _new_size uint
|
||||
if pairs_capacity == 0 {
|
||||
_new_size = max_num_pairs + 1
|
||||
} else {
|
||||
_new_size = pairs_capacity
|
||||
}
|
||||
var new_array []histogramPair
|
||||
for _new_size < (max_num_pairs + 1) {
|
||||
_new_size *= 2
|
||||
}
|
||||
new_array = make([]histogramPair, _new_size)
|
||||
if pairs_capacity != 0 {
|
||||
copy(new_array, pairs[:pairs_capacity])
|
||||
}
|
||||
|
||||
pairs = new_array
|
||||
pairs_capacity = _new_size
|
||||
}
|
||||
|
||||
/* Collapse similar histograms. */
|
||||
num_clusters = histogramCombineLiteral(out, cluster_size, histogram_symbols, clusters, pairs, num_clusters, in_size, max_histograms, max_num_pairs)
|
||||
}
|
||||
|
||||
pairs = nil
|
||||
cluster_size = nil
|
||||
|
||||
/* Find the optimal map from original histograms to the final ones. */
|
||||
histogramRemapLiteral(in, in_size, clusters, num_clusters, out, histogram_symbols)
|
||||
|
||||
clusters = nil
|
||||
|
||||
/* Convert the context map to a canonical form. */
|
||||
*out_size = histogramReindexLiteral(out, histogram_symbols, in_size)
|
||||
}
|
||||
+254
@@ -0,0 +1,254 @@
|
||||
package brotli
|
||||
|
||||
var kInsBase = []uint32{
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
8,
|
||||
10,
|
||||
14,
|
||||
18,
|
||||
26,
|
||||
34,
|
||||
50,
|
||||
66,
|
||||
98,
|
||||
130,
|
||||
194,
|
||||
322,
|
||||
578,
|
||||
1090,
|
||||
2114,
|
||||
6210,
|
||||
22594,
|
||||
}
|
||||
|
||||
var kInsExtra = []uint32{
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
3,
|
||||
3,
|
||||
4,
|
||||
4,
|
||||
5,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
12,
|
||||
14,
|
||||
24,
|
||||
}
|
||||
|
||||
var kCopyBase = []uint32{
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
12,
|
||||
14,
|
||||
18,
|
||||
22,
|
||||
30,
|
||||
38,
|
||||
54,
|
||||
70,
|
||||
102,
|
||||
134,
|
||||
198,
|
||||
326,
|
||||
582,
|
||||
1094,
|
||||
2118,
|
||||
}
|
||||
|
||||
var kCopyExtra = []uint32{
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
3,
|
||||
3,
|
||||
4,
|
||||
4,
|
||||
5,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
24,
|
||||
}
|
||||
|
||||
func getInsertLengthCode(insertlen uint) uint16 {
|
||||
if insertlen < 6 {
|
||||
return uint16(insertlen)
|
||||
} else if insertlen < 130 {
|
||||
var nbits uint32 = log2FloorNonZero(insertlen-2) - 1
|
||||
return uint16((nbits << 1) + uint32((insertlen-2)>>nbits) + 2)
|
||||
} else if insertlen < 2114 {
|
||||
return uint16(log2FloorNonZero(insertlen-66) + 10)
|
||||
} else if insertlen < 6210 {
|
||||
return 21
|
||||
} else if insertlen < 22594 {
|
||||
return 22
|
||||
} else {
|
||||
return 23
|
||||
}
|
||||
}
|
||||
|
||||
func getCopyLengthCode(copylen uint) uint16 {
|
||||
if copylen < 10 {
|
||||
return uint16(copylen - 2)
|
||||
} else if copylen < 134 {
|
||||
var nbits uint32 = log2FloorNonZero(copylen-6) - 1
|
||||
return uint16((nbits << 1) + uint32((copylen-6)>>nbits) + 4)
|
||||
} else if copylen < 2118 {
|
||||
return uint16(log2FloorNonZero(copylen-70) + 12)
|
||||
} else {
|
||||
return 23
|
||||
}
|
||||
}
|
||||
|
||||
func combineLengthCodes(inscode uint16, copycode uint16, use_last_distance bool) uint16 {
|
||||
var bits64 uint16 = uint16(copycode&0x7 | (inscode&0x7)<<3)
|
||||
if use_last_distance && inscode < 8 && copycode < 16 {
|
||||
if copycode < 8 {
|
||||
return bits64
|
||||
} else {
|
||||
return bits64 | 64
|
||||
}
|
||||
} else {
|
||||
/* Specification: 5 Encoding of ... (last table) */
|
||||
/* offset = 2 * index, where index is in range [0..8] */
|
||||
var offset uint32 = 2 * ((uint32(copycode) >> 3) + 3*(uint32(inscode)>>3))
|
||||
|
||||
/* All values in specification are K * 64,
|
||||
where K = [2, 3, 6, 4, 5, 8, 7, 9, 10],
|
||||
i + 1 = [1, 2, 3, 4, 5, 6, 7, 8, 9],
|
||||
K - i - 1 = [1, 1, 3, 0, 0, 2, 0, 1, 2] = D.
|
||||
All values in D require only 2 bits to encode.
|
||||
Magic constant is shifted 6 bits left, to avoid final multiplication. */
|
||||
offset = (offset << 5) + 0x40 + ((0x520D40 >> offset) & 0xC0)
|
||||
|
||||
return uint16(offset | uint32(bits64))
|
||||
}
|
||||
}
|
||||
|
||||
func getLengthCode(insertlen uint, copylen uint, use_last_distance bool, code *uint16) {
|
||||
var inscode uint16 = getInsertLengthCode(insertlen)
|
||||
var copycode uint16 = getCopyLengthCode(copylen)
|
||||
*code = combineLengthCodes(inscode, copycode, use_last_distance)
|
||||
}
|
||||
|
||||
func getInsertBase(inscode uint16) uint32 {
|
||||
return kInsBase[inscode]
|
||||
}
|
||||
|
||||
func getInsertExtra(inscode uint16) uint32 {
|
||||
return kInsExtra[inscode]
|
||||
}
|
||||
|
||||
func getCopyBase(copycode uint16) uint32 {
|
||||
return kCopyBase[copycode]
|
||||
}
|
||||
|
||||
func getCopyExtra(copycode uint16) uint32 {
|
||||
return kCopyExtra[copycode]
|
||||
}
|
||||
|
||||
type command struct {
|
||||
insert_len_ uint32
|
||||
copy_len_ uint32
|
||||
dist_extra_ uint32
|
||||
cmd_prefix_ uint16
|
||||
dist_prefix_ uint16
|
||||
}
|
||||
|
||||
/* distance_code is e.g. 0 for same-as-last short code, or 16 for offset 1. */
|
||||
func makeCommand(dist *distanceParams, insertlen uint, copylen uint, copylen_code_delta int, distance_code uint) (cmd command) {
|
||||
/* Don't rely on signed int representation, use honest casts. */
|
||||
var delta uint32 = uint32(byte(int8(copylen_code_delta)))
|
||||
cmd.insert_len_ = uint32(insertlen)
|
||||
cmd.copy_len_ = uint32(uint32(copylen) | delta<<25)
|
||||
|
||||
/* The distance prefix and extra bits are stored in this Command as if
|
||||
npostfix and ndirect were 0, they are only recomputed later after the
|
||||
clustering if needed. */
|
||||
prefixEncodeCopyDistance(distance_code, uint(dist.num_direct_distance_codes), uint(dist.distance_postfix_bits), &cmd.dist_prefix_, &cmd.dist_extra_)
|
||||
getLengthCode(insertlen, uint(int(copylen)+copylen_code_delta), (cmd.dist_prefix_&0x3FF == 0), &cmd.cmd_prefix_)
|
||||
|
||||
return cmd
|
||||
}
|
||||
|
||||
func makeInsertCommand(insertlen uint) (cmd command) {
|
||||
cmd.insert_len_ = uint32(insertlen)
|
||||
cmd.copy_len_ = 4 << 25
|
||||
cmd.dist_extra_ = 0
|
||||
cmd.dist_prefix_ = numDistanceShortCodes
|
||||
getLengthCode(insertlen, 4, false, &cmd.cmd_prefix_)
|
||||
return cmd
|
||||
}
|
||||
|
||||
func commandRestoreDistanceCode(self *command, dist *distanceParams) uint32 {
|
||||
if uint32(self.dist_prefix_&0x3FF) < numDistanceShortCodes+dist.num_direct_distance_codes {
|
||||
return uint32(self.dist_prefix_) & 0x3FF
|
||||
} else {
|
||||
var dcode uint32 = uint32(self.dist_prefix_) & 0x3FF
|
||||
var nbits uint32 = uint32(self.dist_prefix_) >> 10
|
||||
var extra uint32 = self.dist_extra_
|
||||
var postfix_mask uint32 = (1 << dist.distance_postfix_bits) - 1
|
||||
var hcode uint32 = (dcode - dist.num_direct_distance_codes - numDistanceShortCodes) >> dist.distance_postfix_bits
|
||||
var lcode uint32 = (dcode - dist.num_direct_distance_codes - numDistanceShortCodes) & postfix_mask
|
||||
var offset uint32 = ((2 + (hcode & 1)) << nbits) - 4
|
||||
return ((offset + extra) << dist.distance_postfix_bits) + lcode + dist.num_direct_distance_codes + numDistanceShortCodes
|
||||
}
|
||||
}
|
||||
|
||||
func commandDistanceContext(self *command) uint32 {
|
||||
var r uint32 = uint32(self.cmd_prefix_) >> 6
|
||||
var c uint32 = uint32(self.cmd_prefix_) & 7
|
||||
if (r == 0 || r == 2 || r == 4 || r == 7) && (c <= 2) {
|
||||
return c
|
||||
}
|
||||
|
||||
return 3
|
||||
}
|
||||
|
||||
func commandCopyLen(self *command) uint32 {
|
||||
return self.copy_len_ & 0x1FFFFFF
|
||||
}
|
||||
|
||||
func commandCopyLenCode(self *command) uint32 {
|
||||
var modifier uint32 = self.copy_len_ >> 25
|
||||
var delta int32 = int32(int8(byte(modifier | (modifier&0x40)<<1)))
|
||||
return uint32(int32(self.copy_len_&0x1FFFFFF) + delta)
|
||||
}
|
||||
+834
@@ -0,0 +1,834 @@
|
||||
package brotli
|
||||
|
||||
import "encoding/binary"
|
||||
|
||||
/* Copyright 2015 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Function for fast encoding of an input fragment, independently from the input
|
||||
history. This function uses one-pass processing: when we find a backward
|
||||
match, we immediately emit the corresponding command and literal codes to
|
||||
the bit stream.
|
||||
|
||||
Adapted from the CompressFragment() function in
|
||||
https://github.com/google/snappy/blob/master/snappy.cc */
|
||||
|
||||
const maxDistance_compress_fragment = 262128
|
||||
|
||||
func hash5(p []byte, shift uint) uint32 {
|
||||
var h uint64 = (binary.LittleEndian.Uint64(p) << 24) * uint64(kHashMul32)
|
||||
return uint32(h >> shift)
|
||||
}
|
||||
|
||||
func hashBytesAtOffset5(v uint64, offset int, shift uint) uint32 {
|
||||
assert(offset >= 0)
|
||||
assert(offset <= 3)
|
||||
{
|
||||
var h uint64 = ((v >> uint(8*offset)) << 24) * uint64(kHashMul32)
|
||||
return uint32(h >> shift)
|
||||
}
|
||||
}
|
||||
|
||||
func isMatch5(p1 []byte, p2 []byte) bool {
|
||||
return binary.LittleEndian.Uint32(p1) == binary.LittleEndian.Uint32(p2) &&
|
||||
p1[4] == p2[4]
|
||||
}
|
||||
|
||||
/* Builds a literal prefix code into "depths" and "bits" based on the statistics
|
||||
of the "input" string and stores it into the bit stream.
|
||||
Note that the prefix code here is built from the pre-LZ77 input, therefore
|
||||
we can only approximate the statistics of the actual literal stream.
|
||||
Moreover, for long inputs we build a histogram from a sample of the input
|
||||
and thus have to assign a non-zero depth for each literal.
|
||||
Returns estimated compression ratio millibytes/char for encoding given input
|
||||
with generated code. */
|
||||
func buildAndStoreLiteralPrefixCode(input []byte, input_size uint, depths []byte, bits []uint16, storage_ix *uint, storage []byte) uint {
|
||||
var histogram = [256]uint32{0}
|
||||
var histogram_total uint
|
||||
var i uint
|
||||
if input_size < 1<<15 {
|
||||
for i = 0; i < input_size; i++ {
|
||||
histogram[input[i]]++
|
||||
}
|
||||
|
||||
histogram_total = input_size
|
||||
for i = 0; i < 256; i++ {
|
||||
/* We weigh the first 11 samples with weight 3 to account for the
|
||||
balancing effect of the LZ77 phase on the histogram. */
|
||||
var adjust uint32 = 2 * brotli_min_uint32_t(histogram[i], 11)
|
||||
histogram[i] += adjust
|
||||
histogram_total += uint(adjust)
|
||||
}
|
||||
} else {
|
||||
const kSampleRate uint = 29
|
||||
for i = 0; i < input_size; i += kSampleRate {
|
||||
histogram[input[i]]++
|
||||
}
|
||||
|
||||
histogram_total = (input_size + kSampleRate - 1) / kSampleRate
|
||||
for i = 0; i < 256; i++ {
|
||||
/* We add 1 to each population count to avoid 0 bit depths (since this is
|
||||
only a sample and we don't know if the symbol appears or not), and we
|
||||
weigh the first 11 samples with weight 3 to account for the balancing
|
||||
effect of the LZ77 phase on the histogram (more frequent symbols are
|
||||
more likely to be in backward references instead as literals). */
|
||||
var adjust uint32 = 1 + 2*brotli_min_uint32_t(histogram[i], 11)
|
||||
histogram[i] += adjust
|
||||
histogram_total += uint(adjust)
|
||||
}
|
||||
}
|
||||
|
||||
buildAndStoreHuffmanTreeFast(histogram[:], histogram_total, /* max_bits = */
|
||||
8, depths, bits, storage_ix, storage)
|
||||
{
|
||||
var literal_ratio uint = 0
|
||||
for i = 0; i < 256; i++ {
|
||||
if histogram[i] != 0 {
|
||||
literal_ratio += uint(histogram[i] * uint32(depths[i]))
|
||||
}
|
||||
}
|
||||
|
||||
/* Estimated encoding ratio, millibytes per symbol. */
|
||||
return (literal_ratio * 125) / histogram_total
|
||||
}
|
||||
}
|
||||
|
||||
/* Builds a command and distance prefix code (each 64 symbols) into "depth" and
|
||||
"bits" based on "histogram" and stores it into the bit stream. */
|
||||
func buildAndStoreCommandPrefixCode1(histogram []uint32, depth []byte, bits []uint16, storage_ix *uint, storage []byte) {
|
||||
var tree [129]huffmanTree
|
||||
var cmd_depth = [numCommandSymbols]byte{0}
|
||||
/* Tree size for building a tree over 64 symbols is 2 * 64 + 1. */
|
||||
|
||||
var cmd_bits [64]uint16
|
||||
|
||||
createHuffmanTree(histogram, 64, 15, tree[:], depth)
|
||||
createHuffmanTree(histogram[64:], 64, 14, tree[:], depth[64:])
|
||||
|
||||
/* We have to jump through a few hoops here in order to compute
|
||||
the command bits because the symbols are in a different order than in
|
||||
the full alphabet. This looks complicated, but having the symbols
|
||||
in this order in the command bits saves a few branches in the Emit*
|
||||
functions. */
|
||||
copy(cmd_depth[:], depth[:24])
|
||||
|
||||
copy(cmd_depth[24:][:], depth[40:][:8])
|
||||
copy(cmd_depth[32:][:], depth[24:][:8])
|
||||
copy(cmd_depth[40:][:], depth[48:][:8])
|
||||
copy(cmd_depth[48:][:], depth[32:][:8])
|
||||
copy(cmd_depth[56:][:], depth[56:][:8])
|
||||
convertBitDepthsToSymbols(cmd_depth[:], 64, cmd_bits[:])
|
||||
copy(bits, cmd_bits[:24])
|
||||
copy(bits[24:], cmd_bits[32:][:8])
|
||||
copy(bits[32:], cmd_bits[48:][:8])
|
||||
copy(bits[40:], cmd_bits[24:][:8])
|
||||
copy(bits[48:], cmd_bits[40:][:8])
|
||||
copy(bits[56:], cmd_bits[56:][:8])
|
||||
convertBitDepthsToSymbols(depth[64:], 64, bits[64:])
|
||||
{
|
||||
/* Create the bit length array for the full command alphabet. */
|
||||
var i uint
|
||||
for i := 0; i < int(64); i++ {
|
||||
cmd_depth[i] = 0
|
||||
} /* only 64 first values were used */
|
||||
copy(cmd_depth[:], depth[:8])
|
||||
copy(cmd_depth[64:][:], depth[8:][:8])
|
||||
copy(cmd_depth[128:][:], depth[16:][:8])
|
||||
copy(cmd_depth[192:][:], depth[24:][:8])
|
||||
copy(cmd_depth[384:][:], depth[32:][:8])
|
||||
for i = 0; i < 8; i++ {
|
||||
cmd_depth[128+8*i] = depth[40+i]
|
||||
cmd_depth[256+8*i] = depth[48+i]
|
||||
cmd_depth[448+8*i] = depth[56+i]
|
||||
}
|
||||
|
||||
storeHuffmanTree(cmd_depth[:], numCommandSymbols, tree[:], storage_ix, storage)
|
||||
}
|
||||
|
||||
storeHuffmanTree(depth[64:], 64, tree[:], storage_ix, storage)
|
||||
}
|
||||
|
||||
/* REQUIRES: insertlen < 6210 */
|
||||
func emitInsertLen1(insertlen uint, depth []byte, bits []uint16, histo []uint32, storage_ix *uint, storage []byte) {
|
||||
if insertlen < 6 {
|
||||
var code uint = insertlen + 40
|
||||
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
|
||||
histo[code]++
|
||||
} else if insertlen < 130 {
|
||||
var tail uint = insertlen - 2
|
||||
var nbits uint32 = log2FloorNonZero(tail) - 1
|
||||
var prefix uint = tail >> nbits
|
||||
var inscode uint = uint((nbits << 1) + uint32(prefix) + 42)
|
||||
writeBits(uint(depth[inscode]), uint64(bits[inscode]), storage_ix, storage)
|
||||
writeBits(uint(nbits), uint64(tail)-(uint64(prefix)<<nbits), storage_ix, storage)
|
||||
histo[inscode]++
|
||||
} else if insertlen < 2114 {
|
||||
var tail uint = insertlen - 66
|
||||
var nbits uint32 = log2FloorNonZero(tail)
|
||||
var code uint = uint(nbits + 50)
|
||||
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
|
||||
writeBits(uint(nbits), uint64(tail)-(uint64(uint(1))<<nbits), storage_ix, storage)
|
||||
histo[code]++
|
||||
} else {
|
||||
writeBits(uint(depth[61]), uint64(bits[61]), storage_ix, storage)
|
||||
writeBits(12, uint64(insertlen)-2114, storage_ix, storage)
|
||||
histo[61]++
|
||||
}
|
||||
}
|
||||
|
||||
func emitLongInsertLen(insertlen uint, depth []byte, bits []uint16, histo []uint32, storage_ix *uint, storage []byte) {
|
||||
if insertlen < 22594 {
|
||||
writeBits(uint(depth[62]), uint64(bits[62]), storage_ix, storage)
|
||||
writeBits(14, uint64(insertlen)-6210, storage_ix, storage)
|
||||
histo[62]++
|
||||
} else {
|
||||
writeBits(uint(depth[63]), uint64(bits[63]), storage_ix, storage)
|
||||
writeBits(24, uint64(insertlen)-22594, storage_ix, storage)
|
||||
histo[63]++
|
||||
}
|
||||
}
|
||||
|
||||
func emitCopyLen1(copylen uint, depth []byte, bits []uint16, histo []uint32, storage_ix *uint, storage []byte) {
|
||||
if copylen < 10 {
|
||||
writeBits(uint(depth[copylen+14]), uint64(bits[copylen+14]), storage_ix, storage)
|
||||
histo[copylen+14]++
|
||||
} else if copylen < 134 {
|
||||
var tail uint = copylen - 6
|
||||
var nbits uint32 = log2FloorNonZero(tail) - 1
|
||||
var prefix uint = tail >> nbits
|
||||
var code uint = uint((nbits << 1) + uint32(prefix) + 20)
|
||||
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
|
||||
writeBits(uint(nbits), uint64(tail)-(uint64(prefix)<<nbits), storage_ix, storage)
|
||||
histo[code]++
|
||||
} else if copylen < 2118 {
|
||||
var tail uint = copylen - 70
|
||||
var nbits uint32 = log2FloorNonZero(tail)
|
||||
var code uint = uint(nbits + 28)
|
||||
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
|
||||
writeBits(uint(nbits), uint64(tail)-(uint64(uint(1))<<nbits), storage_ix, storage)
|
||||
histo[code]++
|
||||
} else {
|
||||
writeBits(uint(depth[39]), uint64(bits[39]), storage_ix, storage)
|
||||
writeBits(24, uint64(copylen)-2118, storage_ix, storage)
|
||||
histo[39]++
|
||||
}
|
||||
}
|
||||
|
||||
func emitCopyLenLastDistance1(copylen uint, depth []byte, bits []uint16, histo []uint32, storage_ix *uint, storage []byte) {
|
||||
if copylen < 12 {
|
||||
writeBits(uint(depth[copylen-4]), uint64(bits[copylen-4]), storage_ix, storage)
|
||||
histo[copylen-4]++
|
||||
} else if copylen < 72 {
|
||||
var tail uint = copylen - 8
|
||||
var nbits uint32 = log2FloorNonZero(tail) - 1
|
||||
var prefix uint = tail >> nbits
|
||||
var code uint = uint((nbits << 1) + uint32(prefix) + 4)
|
||||
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
|
||||
writeBits(uint(nbits), uint64(tail)-(uint64(prefix)<<nbits), storage_ix, storage)
|
||||
histo[code]++
|
||||
} else if copylen < 136 {
|
||||
var tail uint = copylen - 8
|
||||
var code uint = (tail >> 5) + 30
|
||||
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
|
||||
writeBits(5, uint64(tail)&31, storage_ix, storage)
|
||||
writeBits(uint(depth[64]), uint64(bits[64]), storage_ix, storage)
|
||||
histo[code]++
|
||||
histo[64]++
|
||||
} else if copylen < 2120 {
|
||||
var tail uint = copylen - 72
|
||||
var nbits uint32 = log2FloorNonZero(tail)
|
||||
var code uint = uint(nbits + 28)
|
||||
writeBits(uint(depth[code]), uint64(bits[code]), storage_ix, storage)
|
||||
writeBits(uint(nbits), uint64(tail)-(uint64(uint(1))<<nbits), storage_ix, storage)
|
||||
writeBits(uint(depth[64]), uint64(bits[64]), storage_ix, storage)
|
||||
histo[code]++
|
||||
histo[64]++
|
||||
} else {
|
||||
writeBits(uint(depth[39]), uint64(bits[39]), storage_ix, storage)
|
||||
writeBits(24, uint64(copylen)-2120, storage_ix, storage)
|
||||
writeBits(uint(depth[64]), uint64(bits[64]), storage_ix, storage)
|
||||
histo[39]++
|
||||
histo[64]++
|
||||
}
|
||||
}
|
||||
|
||||
func emitDistance1(distance uint, depth []byte, bits []uint16, histo []uint32, storage_ix *uint, storage []byte) {
|
||||
var d uint = distance + 3
|
||||
var nbits uint32 = log2FloorNonZero(d) - 1
|
||||
var prefix uint = (d >> nbits) & 1
|
||||
var offset uint = (2 + prefix) << nbits
|
||||
var distcode uint = uint(2*(nbits-1) + uint32(prefix) + 80)
|
||||
writeBits(uint(depth[distcode]), uint64(bits[distcode]), storage_ix, storage)
|
||||
writeBits(uint(nbits), uint64(d)-uint64(offset), storage_ix, storage)
|
||||
histo[distcode]++
|
||||
}
|
||||
|
||||
func emitLiterals(input []byte, len uint, depth []byte, bits []uint16, storage_ix *uint, storage []byte) {
|
||||
var j uint
|
||||
for j = 0; j < len; j++ {
|
||||
var lit byte = input[j]
|
||||
writeBits(uint(depth[lit]), uint64(bits[lit]), storage_ix, storage)
|
||||
}
|
||||
}
|
||||
|
||||
/* REQUIRES: len <= 1 << 24. */
|
||||
func storeMetaBlockHeader1(len uint, is_uncompressed bool, storage_ix *uint, storage []byte) {
|
||||
var nibbles uint = 6
|
||||
|
||||
/* ISLAST */
|
||||
writeBits(1, 0, storage_ix, storage)
|
||||
|
||||
if len <= 1<<16 {
|
||||
nibbles = 4
|
||||
} else if len <= 1<<20 {
|
||||
nibbles = 5
|
||||
}
|
||||
|
||||
writeBits(2, uint64(nibbles)-4, storage_ix, storage)
|
||||
writeBits(nibbles*4, uint64(len)-1, storage_ix, storage)
|
||||
|
||||
/* ISUNCOMPRESSED */
|
||||
writeSingleBit(is_uncompressed, storage_ix, storage)
|
||||
}
|
||||
|
||||
func updateBits(n_bits uint, bits uint32, pos uint, array []byte) {
|
||||
for n_bits > 0 {
|
||||
var byte_pos uint = pos >> 3
|
||||
var n_unchanged_bits uint = pos & 7
|
||||
var n_changed_bits uint = brotli_min_size_t(n_bits, 8-n_unchanged_bits)
|
||||
var total_bits uint = n_unchanged_bits + n_changed_bits
|
||||
var mask uint32 = (^((1 << total_bits) - 1)) | ((1 << n_unchanged_bits) - 1)
|
||||
var unchanged_bits uint32 = uint32(array[byte_pos]) & mask
|
||||
var changed_bits uint32 = bits & ((1 << n_changed_bits) - 1)
|
||||
array[byte_pos] = byte(changed_bits<<n_unchanged_bits | unchanged_bits)
|
||||
n_bits -= n_changed_bits
|
||||
bits >>= n_changed_bits
|
||||
pos += n_changed_bits
|
||||
}
|
||||
}
|
||||
|
||||
func rewindBitPosition1(new_storage_ix uint, storage_ix *uint, storage []byte) {
|
||||
var bitpos uint = new_storage_ix & 7
|
||||
var mask uint = (1 << bitpos) - 1
|
||||
storage[new_storage_ix>>3] &= byte(mask)
|
||||
*storage_ix = new_storage_ix
|
||||
}
|
||||
|
||||
var shouldMergeBlock_kSampleRate uint = 43
|
||||
|
||||
func shouldMergeBlock(data []byte, len uint, depths []byte) bool {
|
||||
var histo = [256]uint{0}
|
||||
var i uint
|
||||
for i = 0; i < len; i += shouldMergeBlock_kSampleRate {
|
||||
histo[data[i]]++
|
||||
}
|
||||
{
|
||||
var total uint = (len + shouldMergeBlock_kSampleRate - 1) / shouldMergeBlock_kSampleRate
|
||||
var r float64 = (fastLog2(total)+0.5)*float64(total) + 200
|
||||
for i = 0; i < 256; i++ {
|
||||
r -= float64(histo[i]) * (float64(depths[i]) + fastLog2(histo[i]))
|
||||
}
|
||||
|
||||
return r >= 0.0
|
||||
}
|
||||
}
|
||||
|
||||
func shouldUseUncompressedMode(metablock_start []byte, next_emit []byte, insertlen uint, literal_ratio uint) bool {
|
||||
var compressed uint = uint(-cap(next_emit) + cap(metablock_start))
|
||||
if compressed*50 > insertlen {
|
||||
return false
|
||||
} else {
|
||||
return literal_ratio > 980
|
||||
}
|
||||
}
|
||||
|
||||
func emitUncompressedMetaBlock1(begin []byte, end []byte, storage_ix_start uint, storage_ix *uint, storage []byte) {
|
||||
var len uint = uint(-cap(end) + cap(begin))
|
||||
rewindBitPosition1(storage_ix_start, storage_ix, storage)
|
||||
storeMetaBlockHeader1(uint(len), true, storage_ix, storage)
|
||||
*storage_ix = (*storage_ix + 7) &^ 7
|
||||
copy(storage[*storage_ix>>3:], begin[:len])
|
||||
*storage_ix += uint(len << 3)
|
||||
storage[*storage_ix>>3] = 0
|
||||
}
|
||||
|
||||
var kCmdHistoSeed = [128]uint32{
|
||||
0,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
0,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
}
|
||||
|
||||
var compressFragmentFastImpl_kFirstBlockSize uint = 3 << 15
|
||||
var compressFragmentFastImpl_kMergeBlockSize uint = 1 << 16
|
||||
|
||||
func compressFragmentFastImpl(in []byte, input_size uint, is_last bool, table []int, table_bits uint, cmd_depth []byte, cmd_bits []uint16, cmd_code_numbits *uint, cmd_code []byte, storage_ix *uint, storage []byte) {
|
||||
var cmd_histo [128]uint32
|
||||
var ip_end int
|
||||
var next_emit int = 0
|
||||
var base_ip int = 0
|
||||
var input int = 0
|
||||
const kInputMarginBytes uint = windowGap
|
||||
const kMinMatchLen uint = 5
|
||||
var metablock_start int = input
|
||||
var block_size uint = brotli_min_size_t(input_size, compressFragmentFastImpl_kFirstBlockSize)
|
||||
var total_block_size uint = block_size
|
||||
var mlen_storage_ix uint = *storage_ix + 3
|
||||
var lit_depth [256]byte
|
||||
var lit_bits [256]uint16
|
||||
var literal_ratio uint
|
||||
var ip int
|
||||
var last_distance int
|
||||
var shift uint = 64 - table_bits
|
||||
|
||||
/* "next_emit" is a pointer to the first byte that is not covered by a
|
||||
previous copy. Bytes between "next_emit" and the start of the next copy or
|
||||
the end of the input will be emitted as literal bytes. */
|
||||
|
||||
/* Save the start of the first block for position and distance computations.
|
||||
*/
|
||||
|
||||
/* Save the bit position of the MLEN field of the meta-block header, so that
|
||||
we can update it later if we decide to extend this meta-block. */
|
||||
storeMetaBlockHeader1(block_size, false, storage_ix, storage)
|
||||
|
||||
/* No block splits, no contexts. */
|
||||
writeBits(13, 0, storage_ix, storage)
|
||||
|
||||
literal_ratio = buildAndStoreLiteralPrefixCode(in[input:], block_size, lit_depth[:], lit_bits[:], storage_ix, storage)
|
||||
{
|
||||
/* Store the pre-compressed command and distance prefix codes. */
|
||||
var i uint
|
||||
for i = 0; i+7 < *cmd_code_numbits; i += 8 {
|
||||
writeBits(8, uint64(cmd_code[i>>3]), storage_ix, storage)
|
||||
}
|
||||
}
|
||||
|
||||
writeBits(*cmd_code_numbits&7, uint64(cmd_code[*cmd_code_numbits>>3]), storage_ix, storage)
|
||||
|
||||
/* Initialize the command and distance histograms. We will gather
|
||||
statistics of command and distance codes during the processing
|
||||
of this block and use it to update the command and distance
|
||||
prefix codes for the next block. */
|
||||
emit_commands:
|
||||
copy(cmd_histo[:], kCmdHistoSeed[:])
|
||||
|
||||
/* "ip" is the input pointer. */
|
||||
ip = input
|
||||
|
||||
last_distance = -1
|
||||
ip_end = int(uint(input) + block_size)
|
||||
|
||||
if block_size >= kInputMarginBytes {
|
||||
var len_limit uint = brotli_min_size_t(block_size-kMinMatchLen, input_size-kInputMarginBytes)
|
||||
var ip_limit int = int(uint(input) + len_limit)
|
||||
/* For the last block, we need to keep a 16 bytes margin so that we can be
|
||||
sure that all distances are at most window size - 16.
|
||||
For all other blocks, we only need to keep a margin of 5 bytes so that
|
||||
we don't go over the block size with a copy. */
|
||||
|
||||
var next_hash uint32
|
||||
ip++
|
||||
for next_hash = hash5(in[ip:], shift); ; {
|
||||
var skip uint32 = 32
|
||||
var next_ip int = ip
|
||||
/* Step 1: Scan forward in the input looking for a 5-byte-long match.
|
||||
If we get close to exhausting the input then goto emit_remainder.
|
||||
|
||||
Heuristic match skipping: If 32 bytes are scanned with no matches
|
||||
found, start looking only at every other byte. If 32 more bytes are
|
||||
scanned, look at every third byte, etc.. When a match is found,
|
||||
immediately go back to looking at every byte. This is a small loss
|
||||
(~5% performance, ~0.1% density) for compressible data due to more
|
||||
bookkeeping, but for non-compressible data (such as JPEG) it's a huge
|
||||
win since the compressor quickly "realizes" the data is incompressible
|
||||
and doesn't bother looking for matches everywhere.
|
||||
|
||||
The "skip" variable keeps track of how many bytes there are since the
|
||||
last match; dividing it by 32 (i.e. right-shifting by five) gives the
|
||||
number of bytes to move ahead for each iteration. */
|
||||
|
||||
var candidate int
|
||||
assert(next_emit < ip)
|
||||
|
||||
trawl:
|
||||
for {
|
||||
var hash uint32 = next_hash
|
||||
var bytes_between_hash_lookups uint32 = skip >> 5
|
||||
skip++
|
||||
assert(hash == hash5(in[next_ip:], shift))
|
||||
ip = next_ip
|
||||
next_ip = int(uint32(ip) + bytes_between_hash_lookups)
|
||||
if next_ip > ip_limit {
|
||||
goto emit_remainder
|
||||
}
|
||||
|
||||
next_hash = hash5(in[next_ip:], shift)
|
||||
candidate = ip - last_distance
|
||||
if isMatch5(in[ip:], in[candidate:]) {
|
||||
if candidate < ip {
|
||||
table[hash] = int(ip - base_ip)
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
candidate = base_ip + table[hash]
|
||||
assert(candidate >= base_ip)
|
||||
assert(candidate < ip)
|
||||
|
||||
table[hash] = int(ip - base_ip)
|
||||
if isMatch5(in[ip:], in[candidate:]) {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
/* Check copy distance. If candidate is not feasible, continue search.
|
||||
Checking is done outside of hot loop to reduce overhead. */
|
||||
if ip-candidate > maxDistance_compress_fragment {
|
||||
goto trawl
|
||||
}
|
||||
|
||||
/* Step 2: Emit the found match together with the literal bytes from
|
||||
"next_emit" to the bit stream, and then see if we can find a next match
|
||||
immediately afterwards. Repeat until we find no match for the input
|
||||
without emitting some literal bytes. */
|
||||
{
|
||||
var base int = ip
|
||||
/* > 0 */
|
||||
var matched uint = 5 + findMatchLengthWithLimit(in[candidate+5:], in[ip+5:], uint(ip_end-ip)-5)
|
||||
var distance int = int(base - candidate)
|
||||
/* We have a 5-byte match at ip, and we need to emit bytes in
|
||||
[next_emit, ip). */
|
||||
|
||||
var insert uint = uint(base - next_emit)
|
||||
ip += int(matched)
|
||||
if insert < 6210 {
|
||||
emitInsertLen1(insert, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
|
||||
} else if shouldUseUncompressedMode(in[metablock_start:], in[next_emit:], insert, literal_ratio) {
|
||||
emitUncompressedMetaBlock1(in[metablock_start:], in[base:], mlen_storage_ix-3, storage_ix, storage)
|
||||
input_size -= uint(base - input)
|
||||
input = base
|
||||
next_emit = input
|
||||
goto next_block
|
||||
} else {
|
||||
emitLongInsertLen(insert, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
|
||||
}
|
||||
|
||||
emitLiterals(in[next_emit:], insert, lit_depth[:], lit_bits[:], storage_ix, storage)
|
||||
if distance == last_distance {
|
||||
writeBits(uint(cmd_depth[64]), uint64(cmd_bits[64]), storage_ix, storage)
|
||||
cmd_histo[64]++
|
||||
} else {
|
||||
emitDistance1(uint(distance), cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
|
||||
last_distance = distance
|
||||
}
|
||||
|
||||
emitCopyLenLastDistance1(matched, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
|
||||
|
||||
next_emit = ip
|
||||
if ip >= ip_limit {
|
||||
goto emit_remainder
|
||||
}
|
||||
|
||||
/* We could immediately start working at ip now, but to improve
|
||||
compression we first update "table" with the hashes of some positions
|
||||
within the last copy. */
|
||||
{
|
||||
var input_bytes uint64 = binary.LittleEndian.Uint64(in[ip-3:])
|
||||
var prev_hash uint32 = hashBytesAtOffset5(input_bytes, 0, shift)
|
||||
var cur_hash uint32 = hashBytesAtOffset5(input_bytes, 3, shift)
|
||||
table[prev_hash] = int(ip - base_ip - 3)
|
||||
prev_hash = hashBytesAtOffset5(input_bytes, 1, shift)
|
||||
table[prev_hash] = int(ip - base_ip - 2)
|
||||
prev_hash = hashBytesAtOffset5(input_bytes, 2, shift)
|
||||
table[prev_hash] = int(ip - base_ip - 1)
|
||||
|
||||
candidate = base_ip + table[cur_hash]
|
||||
table[cur_hash] = int(ip - base_ip)
|
||||
}
|
||||
}
|
||||
|
||||
for isMatch5(in[ip:], in[candidate:]) {
|
||||
var base int = ip
|
||||
/* We have a 5-byte match at ip, and no need to emit any literal bytes
|
||||
prior to ip. */
|
||||
|
||||
var matched uint = 5 + findMatchLengthWithLimit(in[candidate+5:], in[ip+5:], uint(ip_end-ip)-5)
|
||||
if ip-candidate > maxDistance_compress_fragment {
|
||||
break
|
||||
}
|
||||
ip += int(matched)
|
||||
last_distance = int(base - candidate) /* > 0 */
|
||||
emitCopyLen1(matched, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
|
||||
emitDistance1(uint(last_distance), cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
|
||||
|
||||
next_emit = ip
|
||||
if ip >= ip_limit {
|
||||
goto emit_remainder
|
||||
}
|
||||
|
||||
/* We could immediately start working at ip now, but to improve
|
||||
compression we first update "table" with the hashes of some positions
|
||||
within the last copy. */
|
||||
{
|
||||
var input_bytes uint64 = binary.LittleEndian.Uint64(in[ip-3:])
|
||||
var prev_hash uint32 = hashBytesAtOffset5(input_bytes, 0, shift)
|
||||
var cur_hash uint32 = hashBytesAtOffset5(input_bytes, 3, shift)
|
||||
table[prev_hash] = int(ip - base_ip - 3)
|
||||
prev_hash = hashBytesAtOffset5(input_bytes, 1, shift)
|
||||
table[prev_hash] = int(ip - base_ip - 2)
|
||||
prev_hash = hashBytesAtOffset5(input_bytes, 2, shift)
|
||||
table[prev_hash] = int(ip - base_ip - 1)
|
||||
|
||||
candidate = base_ip + table[cur_hash]
|
||||
table[cur_hash] = int(ip - base_ip)
|
||||
}
|
||||
}
|
||||
|
||||
ip++
|
||||
next_hash = hash5(in[ip:], shift)
|
||||
}
|
||||
}
|
||||
|
||||
emit_remainder:
|
||||
assert(next_emit <= ip_end)
|
||||
input += int(block_size)
|
||||
input_size -= block_size
|
||||
block_size = brotli_min_size_t(input_size, compressFragmentFastImpl_kMergeBlockSize)
|
||||
|
||||
/* Decide if we want to continue this meta-block instead of emitting the
|
||||
last insert-only command. */
|
||||
if input_size > 0 && total_block_size+block_size <= 1<<20 && shouldMergeBlock(in[input:], block_size, lit_depth[:]) {
|
||||
assert(total_block_size > 1<<16)
|
||||
|
||||
/* Update the size of the current meta-block and continue emitting commands.
|
||||
We can do this because the current size and the new size both have 5
|
||||
nibbles. */
|
||||
total_block_size += block_size
|
||||
|
||||
updateBits(20, uint32(total_block_size-1), mlen_storage_ix, storage)
|
||||
goto emit_commands
|
||||
}
|
||||
|
||||
/* Emit the remaining bytes as literals. */
|
||||
if next_emit < ip_end {
|
||||
var insert uint = uint(ip_end - next_emit)
|
||||
if insert < 6210 {
|
||||
emitInsertLen1(insert, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
|
||||
emitLiterals(in[next_emit:], insert, lit_depth[:], lit_bits[:], storage_ix, storage)
|
||||
} else if shouldUseUncompressedMode(in[metablock_start:], in[next_emit:], insert, literal_ratio) {
|
||||
emitUncompressedMetaBlock1(in[metablock_start:], in[ip_end:], mlen_storage_ix-3, storage_ix, storage)
|
||||
} else {
|
||||
emitLongInsertLen(insert, cmd_depth, cmd_bits, cmd_histo[:], storage_ix, storage)
|
||||
emitLiterals(in[next_emit:], insert, lit_depth[:], lit_bits[:], storage_ix, storage)
|
||||
}
|
||||
}
|
||||
|
||||
next_emit = ip_end
|
||||
|
||||
/* If we have more data, write a new meta-block header and prefix codes and
|
||||
then continue emitting commands. */
|
||||
next_block:
|
||||
if input_size > 0 {
|
||||
metablock_start = input
|
||||
block_size = brotli_min_size_t(input_size, compressFragmentFastImpl_kFirstBlockSize)
|
||||
total_block_size = block_size
|
||||
|
||||
/* Save the bit position of the MLEN field of the meta-block header, so that
|
||||
we can update it later if we decide to extend this meta-block. */
|
||||
mlen_storage_ix = *storage_ix + 3
|
||||
|
||||
storeMetaBlockHeader1(block_size, false, storage_ix, storage)
|
||||
|
||||
/* No block splits, no contexts. */
|
||||
writeBits(13, 0, storage_ix, storage)
|
||||
|
||||
literal_ratio = buildAndStoreLiteralPrefixCode(in[input:], block_size, lit_depth[:], lit_bits[:], storage_ix, storage)
|
||||
buildAndStoreCommandPrefixCode1(cmd_histo[:], cmd_depth, cmd_bits, storage_ix, storage)
|
||||
goto emit_commands
|
||||
}
|
||||
|
||||
if !is_last {
|
||||
/* If this is not the last block, update the command and distance prefix
|
||||
codes for the next block and store the compressed forms. */
|
||||
cmd_code[0] = 0
|
||||
|
||||
*cmd_code_numbits = 0
|
||||
buildAndStoreCommandPrefixCode1(cmd_histo[:], cmd_depth, cmd_bits, cmd_code_numbits, cmd_code)
|
||||
}
|
||||
}
|
||||
|
||||
/* Compresses "input" string to the "*storage" buffer as one or more complete
|
||||
meta-blocks, and updates the "*storage_ix" bit position.
|
||||
|
||||
If "is_last" is 1, emits an additional empty last meta-block.
|
||||
|
||||
"cmd_depth" and "cmd_bits" contain the command and distance prefix codes
|
||||
(see comment in encode.h) used for the encoding of this input fragment.
|
||||
If "is_last" is 0, they are updated to reflect the statistics
|
||||
of this input fragment, to be used for the encoding of the next fragment.
|
||||
|
||||
"*cmd_code_numbits" is the number of bits of the compressed representation
|
||||
of the command and distance prefix codes, and "cmd_code" is an array of
|
||||
at least "(*cmd_code_numbits + 7) >> 3" size that contains the compressed
|
||||
command and distance prefix codes. If "is_last" is 0, these are also
|
||||
updated to represent the updated "cmd_depth" and "cmd_bits".
|
||||
|
||||
REQUIRES: "input_size" is greater than zero, or "is_last" is 1.
|
||||
REQUIRES: "input_size" is less or equal to maximal metablock size (1 << 24).
|
||||
REQUIRES: All elements in "table[0..table_size-1]" are initialized to zero.
|
||||
REQUIRES: "table_size" is an odd (9, 11, 13, 15) power of two
|
||||
OUTPUT: maximal copy distance <= |input_size|
|
||||
OUTPUT: maximal copy distance <= BROTLI_MAX_BACKWARD_LIMIT(18) */
|
||||
func compressFragmentFast(input []byte, input_size uint, is_last bool, table []int, table_size uint, cmd_depth []byte, cmd_bits []uint16, cmd_code_numbits *uint, cmd_code []byte, storage_ix *uint, storage []byte) {
|
||||
var initial_storage_ix uint = *storage_ix
|
||||
var table_bits uint = uint(log2FloorNonZero(table_size))
|
||||
|
||||
if input_size == 0 {
|
||||
assert(is_last)
|
||||
writeBits(1, 1, storage_ix, storage) /* islast */
|
||||
writeBits(1, 1, storage_ix, storage) /* isempty */
|
||||
*storage_ix = (*storage_ix + 7) &^ 7
|
||||
return
|
||||
}
|
||||
|
||||
compressFragmentFastImpl(input, input_size, is_last, table, table_bits, cmd_depth, cmd_bits, cmd_code_numbits, cmd_code, storage_ix, storage)
|
||||
|
||||
/* If output is larger than single uncompressed block, rewrite it. */
|
||||
if *storage_ix-initial_storage_ix > 31+(input_size<<3) {
|
||||
emitUncompressedMetaBlock1(input, input[input_size:], initial_storage_ix, storage_ix, storage)
|
||||
}
|
||||
|
||||
if is_last {
|
||||
writeBits(1, 1, storage_ix, storage) /* islast */
|
||||
writeBits(1, 1, storage_ix, storage) /* isempty */
|
||||
*storage_ix = (*storage_ix + 7) &^ 7
|
||||
}
|
||||
}
|
||||
+773
@@ -0,0 +1,773 @@
|
||||
package brotli
|
||||
|
||||
import "encoding/binary"
|
||||
|
||||
/* Copyright 2015 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Function for fast encoding of an input fragment, independently from the input
|
||||
history. This function uses two-pass processing: in the first pass we save
|
||||
the found backward matches and literal bytes into a buffer, and in the
|
||||
second pass we emit them into the bit stream using prefix codes built based
|
||||
on the actual command and literal byte histograms. */
|
||||
|
||||
const kCompressFragmentTwoPassBlockSize uint = 1 << 17
|
||||
|
||||
func hash1(p []byte, shift uint, length uint) uint32 {
|
||||
var h uint64 = (binary.LittleEndian.Uint64(p) << ((8 - length) * 8)) * uint64(kHashMul32)
|
||||
return uint32(h >> shift)
|
||||
}
|
||||
|
||||
func hashBytesAtOffset(v uint64, offset uint, shift uint, length uint) uint32 {
|
||||
assert(offset <= 8-length)
|
||||
{
|
||||
var h uint64 = ((v >> (8 * offset)) << ((8 - length) * 8)) * uint64(kHashMul32)
|
||||
return uint32(h >> shift)
|
||||
}
|
||||
}
|
||||
|
||||
func isMatch1(p1 []byte, p2 []byte, length uint) bool {
|
||||
if binary.LittleEndian.Uint32(p1) != binary.LittleEndian.Uint32(p2) {
|
||||
return false
|
||||
}
|
||||
if length == 4 {
|
||||
return true
|
||||
}
|
||||
return p1[4] == p2[4] && p1[5] == p2[5]
|
||||
}
|
||||
|
||||
/*
|
||||
Builds a command and distance prefix code (each 64 symbols) into "depth" and
|
||||
|
||||
"bits" based on "histogram" and stores it into the bit stream.
|
||||
*/
|
||||
func buildAndStoreCommandPrefixCode(histogram []uint32, depth []byte, bits []uint16, storage_ix *uint, storage []byte) {
|
||||
var tree [129]huffmanTree
|
||||
var cmd_depth = [numCommandSymbols]byte{0}
|
||||
/* Tree size for building a tree over 64 symbols is 2 * 64 + 1. */
|
||||
|
||||
var cmd_bits [64]uint16
|
||||
createHuffmanTree(histogram, 64, 15, tree[:], depth)
|
||||
createHuffmanTree(histogram[64:], 64, 14, tree[:], depth[64:])
|
||||
|
||||
/* We have to jump through a few hoops here in order to compute
|
||||
the command bits because the symbols are in a different order than in
|
||||
the full alphabet. This looks complicated, but having the symbols
|
||||
in this order in the command bits saves a few branches in the Emit*
|
||||
functions. */
|
||||
copy(cmd_depth[:], depth[24:][:24])
|
||||
|
||||
copy(cmd_depth[24:][:], depth[:8])
|
||||
copy(cmd_depth[32:][:], depth[48:][:8])
|
||||
copy(cmd_depth[40:][:], depth[8:][:8])
|
||||
copy(cmd_depth[48:][:], depth[56:][:8])
|
||||
copy(cmd_depth[56:][:], depth[16:][:8])
|
||||
convertBitDepthsToSymbols(cmd_depth[:], 64, cmd_bits[:])
|
||||
copy(bits, cmd_bits[24:][:8])
|
||||
copy(bits[8:], cmd_bits[40:][:8])
|
||||
copy(bits[16:], cmd_bits[56:][:8])
|
||||
copy(bits[24:], cmd_bits[:24])
|
||||
copy(bits[48:], cmd_bits[32:][:8])
|
||||
copy(bits[56:], cmd_bits[48:][:8])
|
||||
convertBitDepthsToSymbols(depth[64:], 64, bits[64:])
|
||||
{
|
||||
/* Create the bit length array for the full command alphabet. */
|
||||
var i uint
|
||||
for i := 0; i < int(64); i++ {
|
||||
cmd_depth[i] = 0
|
||||
} /* only 64 first values were used */
|
||||
copy(cmd_depth[:], depth[24:][:8])
|
||||
copy(cmd_depth[64:][:], depth[32:][:8])
|
||||
copy(cmd_depth[128:][:], depth[40:][:8])
|
||||
copy(cmd_depth[192:][:], depth[48:][:8])
|
||||
copy(cmd_depth[384:][:], depth[56:][:8])
|
||||
for i = 0; i < 8; i++ {
|
||||
cmd_depth[128+8*i] = depth[i]
|
||||
cmd_depth[256+8*i] = depth[8+i]
|
||||
cmd_depth[448+8*i] = depth[16+i]
|
||||
}
|
||||
|
||||
storeHuffmanTree(cmd_depth[:], numCommandSymbols, tree[:], storage_ix, storage)
|
||||
}
|
||||
|
||||
storeHuffmanTree(depth[64:], 64, tree[:], storage_ix, storage)
|
||||
}
|
||||
|
||||
func emitInsertLen(insertlen uint32, commands *[]uint32) {
|
||||
if insertlen < 6 {
|
||||
(*commands)[0] = insertlen
|
||||
} else if insertlen < 130 {
|
||||
var tail uint32 = insertlen - 2
|
||||
var nbits uint32 = log2FloorNonZero(uint(tail)) - 1
|
||||
var prefix uint32 = tail >> nbits
|
||||
var inscode uint32 = (nbits << 1) + prefix + 2
|
||||
var extra uint32 = tail - (prefix << nbits)
|
||||
(*commands)[0] = inscode | extra<<8
|
||||
} else if insertlen < 2114 {
|
||||
var tail uint32 = insertlen - 66
|
||||
var nbits uint32 = log2FloorNonZero(uint(tail))
|
||||
var code uint32 = nbits + 10
|
||||
var extra uint32 = tail - (1 << nbits)
|
||||
(*commands)[0] = code | extra<<8
|
||||
} else if insertlen < 6210 {
|
||||
var extra uint32 = insertlen - 2114
|
||||
(*commands)[0] = 21 | extra<<8
|
||||
} else if insertlen < 22594 {
|
||||
var extra uint32 = insertlen - 6210
|
||||
(*commands)[0] = 22 | extra<<8
|
||||
} else {
|
||||
var extra uint32 = insertlen - 22594
|
||||
(*commands)[0] = 23 | extra<<8
|
||||
}
|
||||
|
||||
*commands = (*commands)[1:]
|
||||
}
|
||||
|
||||
func emitCopyLen(copylen uint, commands *[]uint32) {
|
||||
if copylen < 10 {
|
||||
(*commands)[0] = uint32(copylen + 38)
|
||||
} else if copylen < 134 {
|
||||
var tail uint = copylen - 6
|
||||
var nbits uint = uint(log2FloorNonZero(tail) - 1)
|
||||
var prefix uint = tail >> nbits
|
||||
var code uint = (nbits << 1) + prefix + 44
|
||||
var extra uint = tail - (prefix << nbits)
|
||||
(*commands)[0] = uint32(code | extra<<8)
|
||||
} else if copylen < 2118 {
|
||||
var tail uint = copylen - 70
|
||||
var nbits uint = uint(log2FloorNonZero(tail))
|
||||
var code uint = nbits + 52
|
||||
var extra uint = tail - (uint(1) << nbits)
|
||||
(*commands)[0] = uint32(code | extra<<8)
|
||||
} else {
|
||||
var extra uint = copylen - 2118
|
||||
(*commands)[0] = uint32(63 | extra<<8)
|
||||
}
|
||||
|
||||
*commands = (*commands)[1:]
|
||||
}
|
||||
|
||||
func emitCopyLenLastDistance(copylen uint, commands *[]uint32) {
|
||||
if copylen < 12 {
|
||||
(*commands)[0] = uint32(copylen + 20)
|
||||
*commands = (*commands)[1:]
|
||||
} else if copylen < 72 {
|
||||
var tail uint = copylen - 8
|
||||
var nbits uint = uint(log2FloorNonZero(tail) - 1)
|
||||
var prefix uint = tail >> nbits
|
||||
var code uint = (nbits << 1) + prefix + 28
|
||||
var extra uint = tail - (prefix << nbits)
|
||||
(*commands)[0] = uint32(code | extra<<8)
|
||||
*commands = (*commands)[1:]
|
||||
} else if copylen < 136 {
|
||||
var tail uint = copylen - 8
|
||||
var code uint = (tail >> 5) + 54
|
||||
var extra uint = tail & 31
|
||||
(*commands)[0] = uint32(code | extra<<8)
|
||||
*commands = (*commands)[1:]
|
||||
(*commands)[0] = 64
|
||||
*commands = (*commands)[1:]
|
||||
} else if copylen < 2120 {
|
||||
var tail uint = copylen - 72
|
||||
var nbits uint = uint(log2FloorNonZero(tail))
|
||||
var code uint = nbits + 52
|
||||
var extra uint = tail - (uint(1) << nbits)
|
||||
(*commands)[0] = uint32(code | extra<<8)
|
||||
*commands = (*commands)[1:]
|
||||
(*commands)[0] = 64
|
||||
*commands = (*commands)[1:]
|
||||
} else {
|
||||
var extra uint = copylen - 2120
|
||||
(*commands)[0] = uint32(63 | extra<<8)
|
||||
*commands = (*commands)[1:]
|
||||
(*commands)[0] = 64
|
||||
*commands = (*commands)[1:]
|
||||
}
|
||||
}
|
||||
|
||||
func emitDistance(distance uint32, commands *[]uint32) {
|
||||
var d uint32 = distance + 3
|
||||
var nbits uint32 = log2FloorNonZero(uint(d)) - 1
|
||||
var prefix uint32 = (d >> nbits) & 1
|
||||
var offset uint32 = (2 + prefix) << nbits
|
||||
var distcode uint32 = 2*(nbits-1) + prefix + 80
|
||||
var extra uint32 = d - offset
|
||||
(*commands)[0] = distcode | extra<<8
|
||||
*commands = (*commands)[1:]
|
||||
}
|
||||
|
||||
/* REQUIRES: len <= 1 << 24. */
|
||||
func storeMetaBlockHeader(len uint, is_uncompressed bool, storage_ix *uint, storage []byte) {
|
||||
var nibbles uint = 6
|
||||
|
||||
/* ISLAST */
|
||||
writeBits(1, 0, storage_ix, storage)
|
||||
|
||||
if len <= 1<<16 {
|
||||
nibbles = 4
|
||||
} else if len <= 1<<20 {
|
||||
nibbles = 5
|
||||
}
|
||||
|
||||
writeBits(2, uint64(nibbles)-4, storage_ix, storage)
|
||||
writeBits(nibbles*4, uint64(len)-1, storage_ix, storage)
|
||||
|
||||
/* ISUNCOMPRESSED */
|
||||
writeSingleBit(is_uncompressed, storage_ix, storage)
|
||||
}
|
||||
|
||||
func storeMetaBlockHeaderBW(len uint, is_uncompressed bool, bw *bitWriter) {
|
||||
var nibbles uint = 6
|
||||
|
||||
/* ISLAST */
|
||||
bw.writeBits(1, 0)
|
||||
|
||||
if len <= 1<<16 {
|
||||
nibbles = 4
|
||||
} else if len <= 1<<20 {
|
||||
nibbles = 5
|
||||
}
|
||||
|
||||
bw.writeBits(2, uint64(nibbles)-4)
|
||||
bw.writeBits(nibbles*4, uint64(len)-1)
|
||||
|
||||
/* ISUNCOMPRESSED */
|
||||
bw.writeSingleBit(is_uncompressed)
|
||||
}
|
||||
|
||||
func createCommands(input []byte, block_size uint, input_size uint, base_ip_ptr []byte, table []int, table_bits uint, min_match uint, literals *[]byte, commands *[]uint32) {
|
||||
var ip int = 0
|
||||
var shift uint = 64 - table_bits
|
||||
var ip_end int = int(block_size)
|
||||
var base_ip int = -cap(base_ip_ptr) + cap(input)
|
||||
var next_emit int = 0
|
||||
var last_distance int = -1
|
||||
/* "ip" is the input pointer. */
|
||||
|
||||
const kInputMarginBytes uint = windowGap
|
||||
|
||||
/* "next_emit" is a pointer to the first byte that is not covered by a
|
||||
previous copy. Bytes between "next_emit" and the start of the next copy or
|
||||
the end of the input will be emitted as literal bytes. */
|
||||
if block_size >= kInputMarginBytes {
|
||||
var len_limit uint = brotli_min_size_t(block_size-min_match, input_size-kInputMarginBytes)
|
||||
var ip_limit int = int(len_limit)
|
||||
/* For the last block, we need to keep a 16 bytes margin so that we can be
|
||||
sure that all distances are at most window size - 16.
|
||||
For all other blocks, we only need to keep a margin of 5 bytes so that
|
||||
we don't go over the block size with a copy. */
|
||||
|
||||
var next_hash uint32
|
||||
ip++
|
||||
for next_hash = hash1(input[ip:], shift, min_match); ; {
|
||||
var skip uint32 = 32
|
||||
var next_ip int = ip
|
||||
/* Step 1: Scan forward in the input looking for a 6-byte-long match.
|
||||
If we get close to exhausting the input then goto emit_remainder.
|
||||
|
||||
Heuristic match skipping: If 32 bytes are scanned with no matches
|
||||
found, start looking only at every other byte. If 32 more bytes are
|
||||
scanned, look at every third byte, etc.. When a match is found,
|
||||
immediately go back to looking at every byte. This is a small loss
|
||||
(~5% performance, ~0.1% density) for compressible data due to more
|
||||
bookkeeping, but for non-compressible data (such as JPEG) it's a huge
|
||||
win since the compressor quickly "realizes" the data is incompressible
|
||||
and doesn't bother looking for matches everywhere.
|
||||
|
||||
The "skip" variable keeps track of how many bytes there are since the
|
||||
last match; dividing it by 32 (ie. right-shifting by five) gives the
|
||||
number of bytes to move ahead for each iteration. */
|
||||
|
||||
var candidate int
|
||||
|
||||
assert(next_emit < ip)
|
||||
|
||||
trawl:
|
||||
for {
|
||||
var hash uint32 = next_hash
|
||||
var bytes_between_hash_lookups uint32 = skip >> 5
|
||||
skip++
|
||||
ip = next_ip
|
||||
assert(hash == hash1(input[ip:], shift, min_match))
|
||||
next_ip = int(uint32(ip) + bytes_between_hash_lookups)
|
||||
if next_ip > ip_limit {
|
||||
goto emit_remainder
|
||||
}
|
||||
|
||||
next_hash = hash1(input[next_ip:], shift, min_match)
|
||||
candidate = ip - last_distance
|
||||
if isMatch1(input[ip:], base_ip_ptr[candidate-base_ip:], min_match) {
|
||||
if candidate < ip {
|
||||
table[hash] = int(ip - base_ip)
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
candidate = base_ip + table[hash]
|
||||
assert(candidate >= base_ip)
|
||||
assert(candidate < ip)
|
||||
|
||||
table[hash] = int(ip - base_ip)
|
||||
if isMatch1(input[ip:], base_ip_ptr[candidate-base_ip:], min_match) {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
/* Check copy distance. If candidate is not feasible, continue search.
|
||||
Checking is done outside of hot loop to reduce overhead. */
|
||||
if ip-candidate > maxDistance_compress_fragment {
|
||||
goto trawl
|
||||
}
|
||||
|
||||
/* Step 2: Emit the found match together with the literal bytes from
|
||||
"next_emit", and then see if we can find a next match immediately
|
||||
afterwards. Repeat until we find no match for the input
|
||||
without emitting some literal bytes. */
|
||||
{
|
||||
var base int = ip
|
||||
/* > 0 */
|
||||
var matched uint = min_match + findMatchLengthWithLimit(base_ip_ptr[uint(candidate-base_ip)+min_match:], input[uint(ip)+min_match:], uint(ip_end-ip)-min_match)
|
||||
var distance int = int(base - candidate)
|
||||
/* We have a 6-byte match at ip, and we need to emit bytes in
|
||||
[next_emit, ip). */
|
||||
|
||||
var insert int = int(base - next_emit)
|
||||
ip += int(matched)
|
||||
emitInsertLen(uint32(insert), commands)
|
||||
copy(*literals, input[next_emit:][:uint(insert)])
|
||||
*literals = (*literals)[insert:]
|
||||
if distance == last_distance {
|
||||
(*commands)[0] = 64
|
||||
*commands = (*commands)[1:]
|
||||
} else {
|
||||
emitDistance(uint32(distance), commands)
|
||||
last_distance = distance
|
||||
}
|
||||
|
||||
emitCopyLenLastDistance(matched, commands)
|
||||
|
||||
next_emit = ip
|
||||
if ip >= ip_limit {
|
||||
goto emit_remainder
|
||||
}
|
||||
{
|
||||
var input_bytes uint64
|
||||
var cur_hash uint32
|
||||
/* We could immediately start working at ip now, but to improve
|
||||
compression we first update "table" with the hashes of some
|
||||
positions within the last copy. */
|
||||
|
||||
var prev_hash uint32
|
||||
if min_match == 4 {
|
||||
input_bytes = binary.LittleEndian.Uint64(input[ip-3:])
|
||||
cur_hash = hashBytesAtOffset(input_bytes, 3, shift, min_match)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 3)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 2)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 1)
|
||||
} else {
|
||||
input_bytes = binary.LittleEndian.Uint64(input[ip-5:])
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 5)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 4)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 2, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 3)
|
||||
input_bytes = binary.LittleEndian.Uint64(input[ip-2:])
|
||||
cur_hash = hashBytesAtOffset(input_bytes, 2, shift, min_match)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 2)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 1)
|
||||
}
|
||||
|
||||
candidate = base_ip + table[cur_hash]
|
||||
table[cur_hash] = int(ip - base_ip)
|
||||
}
|
||||
}
|
||||
|
||||
for ip-candidate <= maxDistance_compress_fragment && isMatch1(input[ip:], base_ip_ptr[candidate-base_ip:], min_match) {
|
||||
var base int = ip
|
||||
/* We have a 6-byte match at ip, and no need to emit any
|
||||
literal bytes prior to ip. */
|
||||
|
||||
var matched uint = min_match + findMatchLengthWithLimit(base_ip_ptr[uint(candidate-base_ip)+min_match:], input[uint(ip)+min_match:], uint(ip_end-ip)-min_match)
|
||||
ip += int(matched)
|
||||
last_distance = int(base - candidate) /* > 0 */
|
||||
emitCopyLen(matched, commands)
|
||||
emitDistance(uint32(last_distance), commands)
|
||||
|
||||
next_emit = ip
|
||||
if ip >= ip_limit {
|
||||
goto emit_remainder
|
||||
}
|
||||
{
|
||||
var input_bytes uint64
|
||||
var cur_hash uint32
|
||||
/* We could immediately start working at ip now, but to improve
|
||||
compression we first update "table" with the hashes of some
|
||||
positions within the last copy. */
|
||||
|
||||
var prev_hash uint32
|
||||
if min_match == 4 {
|
||||
input_bytes = binary.LittleEndian.Uint64(input[ip-3:])
|
||||
cur_hash = hashBytesAtOffset(input_bytes, 3, shift, min_match)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 3)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 2)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 2, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 1)
|
||||
} else {
|
||||
input_bytes = binary.LittleEndian.Uint64(input[ip-5:])
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 5)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 4)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 2, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 3)
|
||||
input_bytes = binary.LittleEndian.Uint64(input[ip-2:])
|
||||
cur_hash = hashBytesAtOffset(input_bytes, 2, shift, min_match)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 0, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 2)
|
||||
prev_hash = hashBytesAtOffset(input_bytes, 1, shift, min_match)
|
||||
table[prev_hash] = int(ip - base_ip - 1)
|
||||
}
|
||||
|
||||
candidate = base_ip + table[cur_hash]
|
||||
table[cur_hash] = int(ip - base_ip)
|
||||
}
|
||||
}
|
||||
|
||||
ip++
|
||||
next_hash = hash1(input[ip:], shift, min_match)
|
||||
}
|
||||
}
|
||||
|
||||
emit_remainder:
|
||||
assert(next_emit <= ip_end)
|
||||
|
||||
/* Emit the remaining bytes as literals. */
|
||||
if next_emit < ip_end {
|
||||
var insert uint32 = uint32(ip_end - next_emit)
|
||||
emitInsertLen(insert, commands)
|
||||
copy(*literals, input[next_emit:][:insert])
|
||||
*literals = (*literals)[insert:]
|
||||
}
|
||||
}
|
||||
|
||||
var storeCommands_kNumExtraBits = [128]uint32{
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
3,
|
||||
3,
|
||||
4,
|
||||
4,
|
||||
5,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
12,
|
||||
14,
|
||||
24,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
3,
|
||||
3,
|
||||
4,
|
||||
4,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
3,
|
||||
3,
|
||||
4,
|
||||
4,
|
||||
5,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
24,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
3,
|
||||
3,
|
||||
4,
|
||||
4,
|
||||
5,
|
||||
5,
|
||||
6,
|
||||
6,
|
||||
7,
|
||||
7,
|
||||
8,
|
||||
8,
|
||||
9,
|
||||
9,
|
||||
10,
|
||||
10,
|
||||
11,
|
||||
11,
|
||||
12,
|
||||
12,
|
||||
13,
|
||||
13,
|
||||
14,
|
||||
14,
|
||||
15,
|
||||
15,
|
||||
16,
|
||||
16,
|
||||
17,
|
||||
17,
|
||||
18,
|
||||
18,
|
||||
19,
|
||||
19,
|
||||
20,
|
||||
20,
|
||||
21,
|
||||
21,
|
||||
22,
|
||||
22,
|
||||
23,
|
||||
23,
|
||||
24,
|
||||
24,
|
||||
}
|
||||
var storeCommands_kInsertOffset = [24]uint32{
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
8,
|
||||
10,
|
||||
14,
|
||||
18,
|
||||
26,
|
||||
34,
|
||||
50,
|
||||
66,
|
||||
98,
|
||||
130,
|
||||
194,
|
||||
322,
|
||||
578,
|
||||
1090,
|
||||
2114,
|
||||
6210,
|
||||
22594,
|
||||
}
|
||||
|
||||
func storeCommands(literals []byte, num_literals uint, commands []uint32, num_commands uint, storage_ix *uint, storage []byte) {
|
||||
var lit_depths [256]byte
|
||||
var lit_bits [256]uint16
|
||||
var lit_histo = [256]uint32{0}
|
||||
var cmd_depths = [128]byte{0}
|
||||
var cmd_bits = [128]uint16{0}
|
||||
var cmd_histo = [128]uint32{0}
|
||||
var i uint
|
||||
for i = 0; i < num_literals; i++ {
|
||||
lit_histo[literals[i]]++
|
||||
}
|
||||
|
||||
buildAndStoreHuffmanTreeFast(lit_histo[:], num_literals, /* max_bits = */
|
||||
8, lit_depths[:], lit_bits[:], storage_ix, storage)
|
||||
|
||||
for i = 0; i < num_commands; i++ {
|
||||
var code uint32 = commands[i] & 0xFF
|
||||
assert(code < 128)
|
||||
cmd_histo[code]++
|
||||
}
|
||||
|
||||
cmd_histo[1] += 1
|
||||
cmd_histo[2] += 1
|
||||
cmd_histo[64] += 1
|
||||
cmd_histo[84] += 1
|
||||
buildAndStoreCommandPrefixCode(cmd_histo[:], cmd_depths[:], cmd_bits[:], storage_ix, storage)
|
||||
|
||||
for i = 0; i < num_commands; i++ {
|
||||
var cmd uint32 = commands[i]
|
||||
var code uint32 = cmd & 0xFF
|
||||
var extra uint32 = cmd >> 8
|
||||
assert(code < 128)
|
||||
writeBits(uint(cmd_depths[code]), uint64(cmd_bits[code]), storage_ix, storage)
|
||||
writeBits(uint(storeCommands_kNumExtraBits[code]), uint64(extra), storage_ix, storage)
|
||||
if code < 24 {
|
||||
var insert uint32 = storeCommands_kInsertOffset[code] + extra
|
||||
var j uint32
|
||||
for j = 0; j < insert; j++ {
|
||||
var lit byte = literals[0]
|
||||
writeBits(uint(lit_depths[lit]), uint64(lit_bits[lit]), storage_ix, storage)
|
||||
literals = literals[1:]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Acceptable loss for uncompressible speedup is 2% */
|
||||
const minRatio = 0.98
|
||||
|
||||
const sampleRate = 43
|
||||
|
||||
func shouldCompress(input []byte, input_size uint, num_literals uint) bool {
|
||||
var corpus_size float64 = float64(input_size)
|
||||
if float64(num_literals) < minRatio*corpus_size {
|
||||
return true
|
||||
} else {
|
||||
var literal_histo = [256]uint32{0}
|
||||
var max_total_bit_cost float64 = corpus_size * 8 * minRatio / sampleRate
|
||||
var i uint
|
||||
for i = 0; i < input_size; i += sampleRate {
|
||||
literal_histo[input[i]]++
|
||||
}
|
||||
|
||||
return bitsEntropy(literal_histo[:], 256) < max_total_bit_cost
|
||||
}
|
||||
}
|
||||
|
||||
func rewindBitPosition(new_storage_ix uint, storage_ix *uint, storage []byte) {
|
||||
var bitpos uint = new_storage_ix & 7
|
||||
var mask uint = (1 << bitpos) - 1
|
||||
storage[new_storage_ix>>3] &= byte(mask)
|
||||
*storage_ix = new_storage_ix
|
||||
}
|
||||
|
||||
func emitUncompressedMetaBlock(input []byte, input_size uint, storage_ix *uint, storage []byte) {
|
||||
storeMetaBlockHeader(input_size, true, storage_ix, storage)
|
||||
*storage_ix = (*storage_ix + 7) &^ 7
|
||||
copy(storage[*storage_ix>>3:], input[:input_size])
|
||||
*storage_ix += input_size << 3
|
||||
storage[*storage_ix>>3] = 0
|
||||
}
|
||||
|
||||
func compressFragmentTwoPassImpl(input []byte, input_size uint, is_last bool, command_buf []uint32, literal_buf []byte, table []int, table_bits uint, min_match uint, storage_ix *uint, storage []byte) {
|
||||
/* Save the start of the first block for position and distance computations.
|
||||
*/
|
||||
var base_ip []byte = input
|
||||
|
||||
for input_size > 0 {
|
||||
var block_size uint = brotli_min_size_t(input_size, kCompressFragmentTwoPassBlockSize)
|
||||
var commands []uint32 = command_buf
|
||||
var literals []byte = literal_buf
|
||||
var num_literals uint
|
||||
createCommands(input, block_size, input_size, base_ip, table, table_bits, min_match, &literals, &commands)
|
||||
num_literals = uint(-cap(literals) + cap(literal_buf))
|
||||
if shouldCompress(input, block_size, num_literals) {
|
||||
var num_commands uint = uint(-cap(commands) + cap(command_buf))
|
||||
storeMetaBlockHeader(block_size, false, storage_ix, storage)
|
||||
|
||||
/* No block splits, no contexts. */
|
||||
writeBits(13, 0, storage_ix, storage)
|
||||
|
||||
storeCommands(literal_buf, num_literals, command_buf, num_commands, storage_ix, storage)
|
||||
} else {
|
||||
/* Since we did not find many backward references and the entropy of
|
||||
the data is close to 8 bits, we can simply emit an uncompressed block.
|
||||
This makes compression speed of uncompressible data about 3x faster. */
|
||||
emitUncompressedMetaBlock(input, block_size, storage_ix, storage)
|
||||
}
|
||||
|
||||
input = input[block_size:]
|
||||
input_size -= block_size
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
Compresses "input" string to the "*storage" buffer as one or more complete
|
||||
|
||||
meta-blocks, and updates the "*storage_ix" bit position.
|
||||
|
||||
If "is_last" is 1, emits an additional empty last meta-block.
|
||||
|
||||
REQUIRES: "input_size" is greater than zero, or "is_last" is 1.
|
||||
REQUIRES: "input_size" is less or equal to maximal metablock size (1 << 24).
|
||||
REQUIRES: "command_buf" and "literal_buf" point to at least
|
||||
kCompressFragmentTwoPassBlockSize long arrays.
|
||||
REQUIRES: All elements in "table[0..table_size-1]" are initialized to zero.
|
||||
REQUIRES: "table_size" is a power of two
|
||||
OUTPUT: maximal copy distance <= |input_size|
|
||||
OUTPUT: maximal copy distance <= BROTLI_MAX_BACKWARD_LIMIT(18)
|
||||
*/
|
||||
func compressFragmentTwoPass(input []byte, input_size uint, is_last bool, command_buf []uint32, literal_buf []byte, table []int, table_size uint, storage_ix *uint, storage []byte) {
|
||||
var initial_storage_ix uint = *storage_ix
|
||||
var table_bits uint = uint(log2FloorNonZero(table_size))
|
||||
var min_match uint
|
||||
if table_bits <= 15 {
|
||||
min_match = 4
|
||||
} else {
|
||||
min_match = 6
|
||||
}
|
||||
compressFragmentTwoPassImpl(input, input_size, is_last, command_buf, literal_buf, table, table_bits, min_match, storage_ix, storage)
|
||||
|
||||
/* If output is larger than single uncompressed block, rewrite it. */
|
||||
if *storage_ix-initial_storage_ix > 31+(input_size<<3) {
|
||||
rewindBitPosition(initial_storage_ix, storage_ix, storage)
|
||||
emitUncompressedMetaBlock(input, input_size, storage_ix, storage)
|
||||
}
|
||||
|
||||
if is_last {
|
||||
writeBits(1, 1, storage_ix, storage) /* islast */
|
||||
writeBits(1, 1, storage_ix, storage) /* isempty */
|
||||
*storage_ix = (*storage_ix + 7) &^ 7
|
||||
}
|
||||
}
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2016 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Specification: 7.3. Encoding of the context map */
|
||||
const contextMapMaxRle = 16
|
||||
|
||||
/* Specification: 2. Compressed representation overview */
|
||||
const maxNumberOfBlockTypes = 256
|
||||
|
||||
/* Specification: 3.3. Alphabet sizes: insert-and-copy length */
|
||||
const numLiteralSymbols = 256
|
||||
|
||||
const numCommandSymbols = 704
|
||||
|
||||
const numBlockLenSymbols = 26
|
||||
|
||||
const maxContextMapSymbols = (maxNumberOfBlockTypes + contextMapMaxRle)
|
||||
|
||||
const maxBlockTypeSymbols = (maxNumberOfBlockTypes + 2)
|
||||
|
||||
/* Specification: 3.5. Complex prefix codes */
|
||||
const repeatPreviousCodeLength = 16
|
||||
|
||||
const repeatZeroCodeLength = 17
|
||||
|
||||
const codeLengthCodes = (repeatZeroCodeLength + 1)
|
||||
|
||||
/* "code length of 8 is repeated" */
|
||||
const initialRepeatedCodeLength = 8
|
||||
|
||||
/* "Large Window Brotli" */
|
||||
const largeMaxDistanceBits = 62
|
||||
|
||||
const largeMinWbits = 10
|
||||
|
||||
const largeMaxWbits = 30
|
||||
|
||||
/* Specification: 4. Encoding of distances */
|
||||
const numDistanceShortCodes = 16
|
||||
|
||||
const maxNpostfix = 3
|
||||
|
||||
const maxNdirect = 120
|
||||
|
||||
const maxDistanceBits = 24
|
||||
|
||||
func distanceAlphabetSize(NPOSTFIX uint, NDIRECT uint, MAXNBITS uint) uint {
|
||||
return numDistanceShortCodes + NDIRECT + uint(MAXNBITS<<(NPOSTFIX+1))
|
||||
}
|
||||
|
||||
/* numDistanceSymbols == 1128 */
|
||||
const numDistanceSymbols = 1128
|
||||
|
||||
const maxDistance = 0x3FFFFFC
|
||||
|
||||
const maxAllowedDistance = 0x7FFFFFFC
|
||||
|
||||
/* 7.1. Context modes and context ID lookup for literals */
|
||||
/* "context IDs for literals are in the range of 0..63" */
|
||||
const literalContextBits = 6
|
||||
|
||||
/* 7.2. Context ID for distances */
|
||||
const distanceContextBits = 2
|
||||
|
||||
/* 9.1. Format of the Stream Header */
|
||||
/* Number of slack bytes for window size. Don't confuse
|
||||
with BROTLI_NUM_DISTANCE_SHORT_CODES. */
|
||||
const windowGap = 16
|
||||
|
||||
func maxBackwardLimit(W uint) uint {
|
||||
return (uint(1) << W) - windowGap
|
||||
}
|
||||
+2176
File diff suppressed because it is too large.
Load diff
+2581
File diff suppressed because it is too large.
Load diff
+122890
File diff suppressed because it is too large.
Load diff
+32779
File diff suppressed because it is too large.
Load diff
+1220
File diff suppressed because it is too large.
Load diff
+177
@@ -0,0 +1,177 @@
|
||||
package brotli
|
||||
|
||||
import "github.com/andybalholm/brotli/matchfinder"
|
||||
|
||||
// An Encoder implements the matchfinder.Encoder interface, writing in Brotli format.
|
||||
type Encoder struct {
|
||||
wroteHeader bool
|
||||
bw bitWriter
|
||||
distCache []distanceCode
|
||||
}
|
||||
|
||||
func (e *Encoder) Reset() {
|
||||
e.wroteHeader = false
|
||||
e.bw = bitWriter{}
|
||||
}
|
||||
|
||||
func (e *Encoder) Encode(dst []byte, src []byte, matches []matchfinder.Match, lastBlock bool) []byte {
|
||||
e.bw.dst = dst
|
||||
if !e.wroteHeader {
|
||||
e.bw.writeBits(4, 15)
|
||||
e.wroteHeader = true
|
||||
}
|
||||
|
||||
if len(src) == 0 {
|
||||
if lastBlock {
|
||||
e.bw.writeBits(2, 3) // islast + isempty
|
||||
e.bw.jumpToByteBoundary()
|
||||
return e.bw.dst
|
||||
}
|
||||
return dst
|
||||
}
|
||||
|
||||
var literalHisto [256]uint32
|
||||
var commandHisto [704]uint32
|
||||
var distanceHisto [64]uint32
|
||||
literalCount := 0
|
||||
commandCount := 0
|
||||
distanceCount := 0
|
||||
|
||||
if len(e.distCache) < len(matches) {
|
||||
e.distCache = make([]distanceCode, len(matches))
|
||||
}
|
||||
|
||||
// first pass: build the histograms
|
||||
pos := 0
|
||||
|
||||
// d is the ring buffer of the last 4 distances.
|
||||
d := [4]int{-10, -10, -10, -10}
|
||||
for i, m := range matches {
|
||||
if m.Unmatched > 0 {
|
||||
for _, c := range src[pos : pos+m.Unmatched] {
|
||||
literalHisto[c]++
|
||||
}
|
||||
literalCount += m.Unmatched
|
||||
}
|
||||
|
||||
insertCode := getInsertLengthCode(uint(m.Unmatched))
|
||||
copyCode := getCopyLengthCode(uint(m.Length))
|
||||
if m.Length == 0 {
|
||||
// If the stream ends with unmatched bytes, we need a dummy copy length.
|
||||
copyCode = 2
|
||||
}
|
||||
command := combineLengthCodes(insertCode, copyCode, false)
|
||||
commandHisto[command]++
|
||||
commandCount++
|
||||
|
||||
if command >= 128 && m.Length != 0 {
|
||||
var distCode distanceCode
|
||||
switch m.Distance {
|
||||
case d[3]:
|
||||
distCode.code = 0
|
||||
case d[2]:
|
||||
distCode.code = 1
|
||||
case d[1]:
|
||||
distCode.code = 2
|
||||
case d[0]:
|
||||
distCode.code = 3
|
||||
case d[3] - 1:
|
||||
distCode.code = 4
|
||||
case d[3] + 1:
|
||||
distCode.code = 5
|
||||
case d[3] - 2:
|
||||
distCode.code = 6
|
||||
case d[3] + 2:
|
||||
distCode.code = 7
|
||||
case d[3] - 3:
|
||||
distCode.code = 8
|
||||
case d[3] + 3:
|
||||
distCode.code = 9
|
||||
|
||||
// In my testing, codes 10–15 actually reduced the compression ratio.
|
||||
|
||||
default:
|
||||
distCode = getDistanceCode(m.Distance)
|
||||
}
|
||||
e.distCache[i] = distCode
|
||||
distanceHisto[distCode.code]++
|
||||
distanceCount++
|
||||
if distCode.code != 0 {
|
||||
d[0], d[1], d[2], d[3] = d[1], d[2], d[3], m.Distance
|
||||
}
|
||||
}
|
||||
|
||||
pos += m.Unmatched + m.Length
|
||||
}
|
||||
|
||||
storeMetaBlockHeaderBW(uint(len(src)), false, &e.bw)
|
||||
e.bw.writeBits(13, 0)
|
||||
|
||||
var literalDepths [256]byte
|
||||
var literalBits [256]uint16
|
||||
buildAndStoreHuffmanTreeFastBW(literalHisto[:], uint(literalCount), 8, literalDepths[:], literalBits[:], &e.bw)
|
||||
|
||||
var commandDepths [704]byte
|
||||
var commandBits [704]uint16
|
||||
buildAndStoreHuffmanTreeFastBW(commandHisto[:], uint(commandCount), 10, commandDepths[:], commandBits[:], &e.bw)
|
||||
|
||||
var distanceDepths [64]byte
|
||||
var distanceBits [64]uint16
|
||||
buildAndStoreHuffmanTreeFastBW(distanceHisto[:], uint(distanceCount), 6, distanceDepths[:], distanceBits[:], &e.bw)
|
||||
|
||||
pos = 0
|
||||
for i, m := range matches {
|
||||
insertCode := getInsertLengthCode(uint(m.Unmatched))
|
||||
copyCode := getCopyLengthCode(uint(m.Length))
|
||||
if m.Length == 0 {
|
||||
// If the stream ends with unmatched bytes, we need a dummy copy length.
|
||||
copyCode = 2
|
||||
}
|
||||
command := combineLengthCodes(insertCode, copyCode, false)
|
||||
e.bw.writeBits(uint(commandDepths[command]), uint64(commandBits[command]))
|
||||
if kInsExtra[insertCode] > 0 {
|
||||
e.bw.writeBits(uint(kInsExtra[insertCode]), uint64(m.Unmatched)-uint64(kInsBase[insertCode]))
|
||||
}
|
||||
if kCopyExtra[copyCode] > 0 {
|
||||
e.bw.writeBits(uint(kCopyExtra[copyCode]), uint64(m.Length)-uint64(kCopyBase[copyCode]))
|
||||
}
|
||||
|
||||
if m.Unmatched > 0 {
|
||||
for _, c := range src[pos : pos+m.Unmatched] {
|
||||
e.bw.writeBits(uint(literalDepths[c]), uint64(literalBits[c]))
|
||||
}
|
||||
}
|
||||
|
||||
if command >= 128 && m.Length != 0 {
|
||||
distCode := e.distCache[i]
|
||||
e.bw.writeBits(uint(distanceDepths[distCode.code]), uint64(distanceBits[distCode.code]))
|
||||
if distCode.nExtra > 0 {
|
||||
e.bw.writeBits(distCode.nExtra, distCode.extraBits)
|
||||
}
|
||||
}
|
||||
|
||||
pos += m.Unmatched + m.Length
|
||||
}
|
||||
|
||||
if lastBlock {
|
||||
e.bw.writeBits(2, 3) // islast + isempty
|
||||
e.bw.jumpToByteBoundary()
|
||||
}
|
||||
return e.bw.dst
|
||||
}
|
||||
|
||||
type distanceCode struct {
|
||||
code int
|
||||
nExtra uint
|
||||
extraBits uint64
|
||||
}
|
||||
|
||||
func getDistanceCode(distance int) distanceCode {
|
||||
d := distance + 3
|
||||
nbits := log2FloorNonZero(uint(d)) - 1
|
||||
prefix := (d >> nbits) & 1
|
||||
offset := (2 + prefix) << nbits
|
||||
distcode := int(2*(nbits-1)) + prefix + 16
|
||||
extra := d - offset
|
||||
return distanceCode{distcode, uint(nbits), uint64(extra)}
|
||||
}
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
package brotli
|
||||
|
||||
/* Dictionary data (words and transforms) for 1 possible context */
|
||||
type encoderDictionary struct {
|
||||
words *dictionary
|
||||
cutoffTransformsCount uint32
|
||||
cutoffTransforms uint64
|
||||
hash_table []uint16
|
||||
buckets []uint16
|
||||
dict_words []dictWord
|
||||
}
|
||||
|
||||
func initEncoderDictionary(dict *encoderDictionary) {
|
||||
dict.words = getDictionary()
|
||||
|
||||
dict.hash_table = kStaticDictionaryHash[:]
|
||||
dict.buckets = kStaticDictionaryBuckets[:]
|
||||
dict.dict_words = kStaticDictionaryWords[:]
|
||||
|
||||
dict.cutoffTransformsCount = kCutoffTransformsCount
|
||||
dict.cutoffTransforms = kCutoffTransforms
|
||||
}
|
||||
+592
@@ -0,0 +1,592 @@
|
||||
package brotli
|
||||
|
||||
import "math"
|
||||
|
||||
/* Copyright 2010 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Entropy encoding (Huffman) utilities. */
|
||||
|
||||
/* A node of a Huffman tree. */
|
||||
type huffmanTree struct {
|
||||
total_count_ uint32
|
||||
index_left_ int16
|
||||
index_right_or_value_ int16
|
||||
}
|
||||
|
||||
func initHuffmanTree(self *huffmanTree, count uint32, left int16, right int16) {
|
||||
self.total_count_ = count
|
||||
self.index_left_ = left
|
||||
self.index_right_or_value_ = right
|
||||
}
|
||||
|
||||
/* Input size optimized Shell sort. */
|
||||
type huffmanTreeComparator func(huffmanTree, huffmanTree) bool
|
||||
|
||||
var sortHuffmanTreeItems_gaps = []uint{132, 57, 23, 10, 4, 1}
|
||||
|
||||
func sortHuffmanTreeItems(items []huffmanTree, n uint, comparator huffmanTreeComparator) {
|
||||
if n < 13 {
|
||||
/* Insertion sort. */
|
||||
var i uint
|
||||
for i = 1; i < n; i++ {
|
||||
var tmp huffmanTree = items[i]
|
||||
var k uint = i
|
||||
var j uint = i - 1
|
||||
for comparator(tmp, items[j]) {
|
||||
items[k] = items[j]
|
||||
k = j
|
||||
if j == 0 {
|
||||
break
|
||||
}
|
||||
j--
|
||||
}
|
||||
|
||||
items[k] = tmp
|
||||
}
|
||||
|
||||
return
|
||||
} else {
|
||||
var g int
|
||||
if n < 57 {
|
||||
g = 2
|
||||
} else {
|
||||
g = 0
|
||||
}
|
||||
for ; g < 6; g++ {
|
||||
var gap uint = sortHuffmanTreeItems_gaps[g]
|
||||
var i uint
|
||||
for i = gap; i < n; i++ {
|
||||
var j uint = i
|
||||
var tmp huffmanTree = items[i]
|
||||
for ; j >= gap && comparator(tmp, items[j-gap]); j -= gap {
|
||||
items[j] = items[j-gap]
|
||||
}
|
||||
|
||||
items[j] = tmp
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Returns 1 if assignment of depths succeeded, otherwise 0. */
|
||||
func setDepth(p0 int, pool []huffmanTree, depth []byte, max_depth int) bool {
|
||||
var stack [16]int
|
||||
var level int = 0
|
||||
var p int = p0
|
||||
assert(max_depth <= 15)
|
||||
stack[0] = -1
|
||||
for {
|
||||
if pool[p].index_left_ >= 0 {
|
||||
level++
|
||||
if level > max_depth {
|
||||
return false
|
||||
}
|
||||
stack[level] = int(pool[p].index_right_or_value_)
|
||||
p = int(pool[p].index_left_)
|
||||
continue
|
||||
} else {
|
||||
depth[pool[p].index_right_or_value_] = byte(level)
|
||||
}
|
||||
|
||||
for level >= 0 && stack[level] == -1 {
|
||||
level--
|
||||
}
|
||||
if level < 0 {
|
||||
return true
|
||||
}
|
||||
p = stack[level]
|
||||
stack[level] = -1
|
||||
}
|
||||
}
|
||||
|
||||
/* Sort the root nodes, least popular first. */
|
||||
func sortHuffmanTree(v0 huffmanTree, v1 huffmanTree) bool {
|
||||
if v0.total_count_ != v1.total_count_ {
|
||||
return v0.total_count_ < v1.total_count_
|
||||
}
|
||||
|
||||
return v0.index_right_or_value_ > v1.index_right_or_value_
|
||||
}
|
||||
|
||||
/* This function will create a Huffman tree.
|
||||
|
||||
The catch here is that the tree cannot be arbitrarily deep.
|
||||
Brotli specifies a maximum depth of 15 bits for "code trees"
|
||||
and 7 bits for "code length code trees."
|
||||
|
||||
count_limit is the value that is to be faked as the minimum value
|
||||
and this minimum value is raised until the tree matches the
|
||||
maximum length requirement.
|
||||
|
||||
This algorithm is not of excellent performance for very long data blocks,
|
||||
especially when population counts are longer than 2**tree_limit, but
|
||||
we are not planning to use this with extremely long blocks.
|
||||
|
||||
See http://en.wikipedia.org/wiki/Huffman_coding */
|
||||
func createHuffmanTree(data []uint32, length uint, tree_limit int, tree []huffmanTree, depth []byte) {
|
||||
var count_limit uint32
|
||||
var sentinel huffmanTree
|
||||
initHuffmanTree(&sentinel, math.MaxUint32, -1, -1)
|
||||
|
||||
/* For block sizes below 64 kB, we never need to do a second iteration
|
||||
of this loop. Probably all of our block sizes will be smaller than
|
||||
that, so this loop is mostly of academic interest. If we actually
|
||||
would need this, we would be better off with the Katajainen algorithm. */
|
||||
for count_limit = 1; ; count_limit *= 2 {
|
||||
var n uint = 0
|
||||
var i uint
|
||||
var j uint
|
||||
var k uint
|
||||
for i = length; i != 0; {
|
||||
i--
|
||||
if data[i] != 0 {
|
||||
var count uint32 = brotli_max_uint32_t(data[i], count_limit)
|
||||
initHuffmanTree(&tree[n], count, -1, int16(i))
|
||||
n++
|
||||
}
|
||||
}
|
||||
|
||||
if n == 1 {
|
||||
depth[tree[0].index_right_or_value_] = 1 /* Only one element. */
|
||||
break
|
||||
}
|
||||
|
||||
sortHuffmanTreeItems(tree, n, huffmanTreeComparator(sortHuffmanTree))
|
||||
|
||||
/* The nodes are:
|
||||
[0, n): the sorted leaf nodes that we start with.
|
||||
[n]: we add a sentinel here.
|
||||
[n + 1, 2n): new parent nodes are added here, starting from
|
||||
(n+1). These are naturally in ascending order.
|
||||
[2n]: we add a sentinel at the end as well.
|
||||
There will be (2n+1) elements at the end. */
|
||||
tree[n] = sentinel
|
||||
|
||||
tree[n+1] = sentinel
|
||||
|
||||
i = 0 /* Points to the next leaf node. */
|
||||
j = n + 1 /* Points to the next non-leaf node. */
|
||||
for k = n - 1; k != 0; k-- {
|
||||
var left uint
|
||||
var right uint
|
||||
if tree[i].total_count_ <= tree[j].total_count_ {
|
||||
left = i
|
||||
i++
|
||||
} else {
|
||||
left = j
|
||||
j++
|
||||
}
|
||||
|
||||
if tree[i].total_count_ <= tree[j].total_count_ {
|
||||
right = i
|
||||
i++
|
||||
} else {
|
||||
right = j
|
||||
j++
|
||||
}
|
||||
{
|
||||
/* The sentinel node becomes the parent node. */
|
||||
var j_end uint = 2*n - k
|
||||
tree[j_end].total_count_ = tree[left].total_count_ + tree[right].total_count_
|
||||
tree[j_end].index_left_ = int16(left)
|
||||
tree[j_end].index_right_or_value_ = int16(right)
|
||||
|
||||
/* Add back the last sentinel node. */
|
||||
tree[j_end+1] = sentinel
|
||||
}
|
||||
}
|
||||
|
||||
if setDepth(int(2*n-1), tree[0:], depth, tree_limit) {
|
||||
/* We need to pack the Huffman tree in tree_limit bits. If this was not
|
||||
successful, add fake entities to the lowest values and retry. */
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func reverse(v []byte, start uint, end uint) {
|
||||
end--
|
||||
for start < end {
|
||||
var tmp byte = v[start]
|
||||
v[start] = v[end]
|
||||
v[end] = tmp
|
||||
start++
|
||||
end--
|
||||
}
|
||||
}
|
||||
|
||||
func writeHuffmanTreeRepetitions(previous_value byte, value byte, repetitions uint, tree_size *uint, tree []byte, extra_bits_data []byte) {
|
||||
assert(repetitions > 0)
|
||||
if previous_value != value {
|
||||
tree[*tree_size] = value
|
||||
extra_bits_data[*tree_size] = 0
|
||||
(*tree_size)++
|
||||
repetitions--
|
||||
}
|
||||
|
||||
if repetitions == 7 {
|
||||
tree[*tree_size] = value
|
||||
extra_bits_data[*tree_size] = 0
|
||||
(*tree_size)++
|
||||
repetitions--
|
||||
}
|
||||
|
||||
if repetitions < 3 {
|
||||
var i uint
|
||||
for i = 0; i < repetitions; i++ {
|
||||
tree[*tree_size] = value
|
||||
extra_bits_data[*tree_size] = 0
|
||||
(*tree_size)++
|
||||
}
|
||||
} else {
|
||||
var start uint = *tree_size
|
||||
repetitions -= 3
|
||||
for {
|
||||
tree[*tree_size] = repeatPreviousCodeLength
|
||||
extra_bits_data[*tree_size] = byte(repetitions & 0x3)
|
||||
(*tree_size)++
|
||||
repetitions >>= 2
|
||||
if repetitions == 0 {
|
||||
break
|
||||
}
|
||||
|
||||
repetitions--
|
||||
}
|
||||
|
||||
reverse(tree, start, *tree_size)
|
||||
reverse(extra_bits_data, start, *tree_size)
|
||||
}
|
||||
}
|
||||
|
||||
func writeHuffmanTreeRepetitionsZeros(repetitions uint, tree_size *uint, tree []byte, extra_bits_data []byte) {
|
||||
if repetitions == 11 {
|
||||
tree[*tree_size] = 0
|
||||
extra_bits_data[*tree_size] = 0
|
||||
(*tree_size)++
|
||||
repetitions--
|
||||
}
|
||||
|
||||
if repetitions < 3 {
|
||||
var i uint
|
||||
for i = 0; i < repetitions; i++ {
|
||||
tree[*tree_size] = 0
|
||||
extra_bits_data[*tree_size] = 0
|
||||
(*tree_size)++
|
||||
}
|
||||
} else {
|
||||
var start uint = *tree_size
|
||||
repetitions -= 3
|
||||
for {
|
||||
tree[*tree_size] = repeatZeroCodeLength
|
||||
extra_bits_data[*tree_size] = byte(repetitions & 0x7)
|
||||
(*tree_size)++
|
||||
repetitions >>= 3
|
||||
if repetitions == 0 {
|
||||
break
|
||||
}
|
||||
|
||||
repetitions--
|
||||
}
|
||||
|
||||
reverse(tree, start, *tree_size)
|
||||
reverse(extra_bits_data, start, *tree_size)
|
||||
}
|
||||
}
|
||||
|
||||
/* Change the population counts in a way that the consequent
|
||||
Huffman tree compression, especially its RLE-part will be more
|
||||
likely to compress this data more efficiently.
|
||||
|
||||
length contains the size of the histogram.
|
||||
counts contains the population counts.
|
||||
good_for_rle is a buffer of at least length size */
|
||||
func optimizeHuffmanCountsForRLE(length uint, counts []uint32, good_for_rle []byte) {
|
||||
var nonzero_count uint = 0
|
||||
var stride uint
|
||||
var limit uint
|
||||
var sum uint
|
||||
var streak_limit uint = 1240
|
||||
var i uint
|
||||
/* Let's make the Huffman code more compatible with RLE encoding. */
|
||||
for i = 0; i < length; i++ {
|
||||
if counts[i] != 0 {
|
||||
nonzero_count++
|
||||
}
|
||||
}
|
||||
|
||||
if nonzero_count < 16 {
|
||||
return
|
||||
}
|
||||
|
||||
for length != 0 && counts[length-1] == 0 {
|
||||
length--
|
||||
}
|
||||
|
||||
if length == 0 {
|
||||
return /* All zeros. */
|
||||
}
|
||||
|
||||
/* Now counts[0..length - 1] does not have trailing zeros. */
|
||||
{
|
||||
var nonzeros uint = 0
|
||||
var smallest_nonzero uint32 = 1 << 30
|
||||
for i = 0; i < length; i++ {
|
||||
if counts[i] != 0 {
|
||||
nonzeros++
|
||||
if smallest_nonzero > counts[i] {
|
||||
smallest_nonzero = counts[i]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if nonzeros < 5 {
|
||||
/* Small histogram will model it well. */
|
||||
return
|
||||
}
|
||||
|
||||
if smallest_nonzero < 4 {
|
||||
var zeros uint = length - nonzeros
|
||||
if zeros < 6 {
|
||||
for i = 1; i < length-1; i++ {
|
||||
if counts[i-1] != 0 && counts[i] == 0 && counts[i+1] != 0 {
|
||||
counts[i] = 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if nonzeros < 28 {
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
/* 2) Let's mark all population counts that already can be encoded
|
||||
with an RLE code. */
|
||||
for i := 0; i < int(length); i++ {
|
||||
good_for_rle[i] = 0
|
||||
}
|
||||
{
|
||||
var symbol uint32 = counts[0]
|
||||
/* Let's not spoil any of the existing good RLE codes.
|
||||
Mark any seq of 0's that is longer as 5 as a good_for_rle.
|
||||
Mark any seq of non-0's that is longer as 7 as a good_for_rle. */
|
||||
|
||||
var step uint = 0
|
||||
for i = 0; i <= length; i++ {
|
||||
if i == length || counts[i] != symbol {
|
||||
if (symbol == 0 && step >= 5) || (symbol != 0 && step >= 7) {
|
||||
var k uint
|
||||
for k = 0; k < step; k++ {
|
||||
good_for_rle[i-k-1] = 1
|
||||
}
|
||||
}
|
||||
|
||||
step = 1
|
||||
if i != length {
|
||||
symbol = counts[i]
|
||||
}
|
||||
} else {
|
||||
step++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* 3) Let's replace those population counts that lead to more RLE codes.
|
||||
Math here is in 24.8 fixed point representation. */
|
||||
stride = 0
|
||||
|
||||
limit = uint(256*(counts[0]+counts[1]+counts[2])/3 + 420)
|
||||
sum = 0
|
||||
for i = 0; i <= length; i++ {
|
||||
if i == length || good_for_rle[i] != 0 || (i != 0 && good_for_rle[i-1] != 0) || (256*counts[i]-uint32(limit)+uint32(streak_limit)) >= uint32(2*streak_limit) {
|
||||
if stride >= 4 || (stride >= 3 && sum == 0) {
|
||||
var k uint
|
||||
var count uint = (sum + stride/2) / stride
|
||||
/* The stride must end, collapse what we have, if we have enough (4). */
|
||||
if count == 0 {
|
||||
count = 1
|
||||
}
|
||||
|
||||
if sum == 0 {
|
||||
/* Don't make an all zeros stride to be upgraded to ones. */
|
||||
count = 0
|
||||
}
|
||||
|
||||
for k = 0; k < stride; k++ {
|
||||
/* We don't want to change value at counts[i],
|
||||
that is already belonging to the next stride. Thus - 1. */
|
||||
counts[i-k-1] = uint32(count)
|
||||
}
|
||||
}
|
||||
|
||||
stride = 0
|
||||
sum = 0
|
||||
if i < length-2 {
|
||||
/* All interesting strides have a count of at least 4, */
|
||||
/* at least when non-zeros. */
|
||||
limit = uint(256*(counts[i]+counts[i+1]+counts[i+2])/3 + 420)
|
||||
} else if i < length {
|
||||
limit = uint(256 * counts[i])
|
||||
} else {
|
||||
limit = 0
|
||||
}
|
||||
}
|
||||
|
||||
stride++
|
||||
if i != length {
|
||||
sum += uint(counts[i])
|
||||
if stride >= 4 {
|
||||
limit = (256*sum + stride/2) / stride
|
||||
}
|
||||
|
||||
if stride == 4 {
|
||||
limit += 120
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func decideOverRLEUse(depth []byte, length uint, use_rle_for_non_zero *bool, use_rle_for_zero *bool) {
|
||||
var total_reps_zero uint = 0
|
||||
var total_reps_non_zero uint = 0
|
||||
var count_reps_zero uint = 1
|
||||
var count_reps_non_zero uint = 1
|
||||
var i uint
|
||||
for i = 0; i < length; {
|
||||
var value byte = depth[i]
|
||||
var reps uint = 1
|
||||
var k uint
|
||||
for k = i + 1; k < length && depth[k] == value; k++ {
|
||||
reps++
|
||||
}
|
||||
|
||||
if reps >= 3 && value == 0 {
|
||||
total_reps_zero += reps
|
||||
count_reps_zero++
|
||||
}
|
||||
|
||||
if reps >= 4 && value != 0 {
|
||||
total_reps_non_zero += reps
|
||||
count_reps_non_zero++
|
||||
}
|
||||
|
||||
i += reps
|
||||
}
|
||||
|
||||
*use_rle_for_non_zero = total_reps_non_zero > count_reps_non_zero*2
|
||||
*use_rle_for_zero = total_reps_zero > count_reps_zero*2
|
||||
}
|
||||
|
||||
/* Write a Huffman tree from bit depths into the bit-stream representation
|
||||
of a Huffman tree. The generated Huffman tree is to be compressed once
|
||||
more using a Huffman tree */
|
||||
func writeHuffmanTree(depth []byte, length uint, tree_size *uint, tree []byte, extra_bits_data []byte) {
|
||||
var previous_value byte = initialRepeatedCodeLength
|
||||
var i uint
|
||||
var use_rle_for_non_zero bool = false
|
||||
var use_rle_for_zero bool = false
|
||||
var new_length uint = length
|
||||
/* Throw away trailing zeros. */
|
||||
for i = 0; i < length; i++ {
|
||||
if depth[length-i-1] == 0 {
|
||||
new_length--
|
||||
} else {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
/* First gather statistics on if it is a good idea to do RLE. */
|
||||
if length > 50 {
|
||||
/* Find RLE coding for longer codes.
|
||||
Shorter codes seem not to benefit from RLE. */
|
||||
decideOverRLEUse(depth, new_length, &use_rle_for_non_zero, &use_rle_for_zero)
|
||||
}
|
||||
|
||||
/* Actual RLE coding. */
|
||||
for i = 0; i < new_length; {
|
||||
var value byte = depth[i]
|
||||
var reps uint = 1
|
||||
if (value != 0 && use_rle_for_non_zero) || (value == 0 && use_rle_for_zero) {
|
||||
var k uint
|
||||
for k = i + 1; k < new_length && depth[k] == value; k++ {
|
||||
reps++
|
||||
}
|
||||
}
|
||||
|
||||
if value == 0 {
|
||||
writeHuffmanTreeRepetitionsZeros(reps, tree_size, tree, extra_bits_data)
|
||||
} else {
|
||||
writeHuffmanTreeRepetitions(previous_value, value, reps, tree_size, tree, extra_bits_data)
|
||||
previous_value = value
|
||||
}
|
||||
|
||||
i += reps
|
||||
}
|
||||
}
|
||||
|
||||
var reverseBits_kLut = [16]uint{
|
||||
0x00,
|
||||
0x08,
|
||||
0x04,
|
||||
0x0C,
|
||||
0x02,
|
||||
0x0A,
|
||||
0x06,
|
||||
0x0E,
|
||||
0x01,
|
||||
0x09,
|
||||
0x05,
|
||||
0x0D,
|
||||
0x03,
|
||||
0x0B,
|
||||
0x07,
|
||||
0x0F,
|
||||
}
|
||||
|
||||
func reverseBits(num_bits uint, bits uint16) uint16 {
|
||||
var retval uint = reverseBits_kLut[bits&0x0F]
|
||||
var i uint
|
||||
for i = 4; i < num_bits; i += 4 {
|
||||
retval <<= 4
|
||||
bits = uint16(bits >> 4)
|
||||
retval |= reverseBits_kLut[bits&0x0F]
|
||||
}
|
||||
|
||||
retval >>= ((0 - num_bits) & 0x03)
|
||||
return uint16(retval)
|
||||
}
|
||||
|
||||
/* 0..15 are values for bits */
|
||||
const maxHuffmanBits = 16
|
||||
|
||||
/* Get the actual bit values for a tree of bit depths. */
|
||||
func convertBitDepthsToSymbols(depth []byte, len uint, bits []uint16) {
|
||||
var bl_count = [maxHuffmanBits]uint16{0}
|
||||
var next_code [maxHuffmanBits]uint16
|
||||
var i uint
|
||||
/* In Brotli, all bit depths are [1..15]
|
||||
0 bit depth means that the symbol does not exist. */
|
||||
|
||||
var code int = 0
|
||||
for i = 0; i < len; i++ {
|
||||
bl_count[depth[i]]++
|
||||
}
|
||||
|
||||
bl_count[0] = 0
|
||||
next_code[0] = 0
|
||||
for i = 1; i < maxHuffmanBits; i++ {
|
||||
code = (code + int(bl_count[i-1])) << 1
|
||||
next_code[i] = uint16(code)
|
||||
}
|
||||
|
||||
for i = 0; i < len; i++ {
|
||||
if depth[i] != 0 {
|
||||
bits[i] = reverseBits(uint(depth[i]), next_code[depth[i]])
|
||||
next_code[depth[i]]++
|
||||
}
|
||||
}
|
||||
}
|
||||
+4399
File diff suppressed because it is too large.
Load diff
+290
@@ -0,0 +1,290 @@
|
||||
package brotli
|
||||
|
||||
import (
|
||||
"math"
|
||||
"math/bits"
|
||||
)
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Utilities for fast computation of logarithms. */
|
||||
|
||||
func log2FloorNonZero(n uint) uint32 {
|
||||
return uint32(bits.Len(n)) - 1
|
||||
}
|
||||
|
||||
/* A lookup table for small values of log2(int) to be used in entropy
|
||||
computation.
|
||||
|
||||
", ".join(["%.16ff" % x for x in [0.0]+[log2(x) for x in range(1, 256)]]) */
|
||||
var kLog2Table = []float32{
|
||||
0.0000000000000000,
|
||||
0.0000000000000000,
|
||||
1.0000000000000000,
|
||||
1.5849625007211563,
|
||||
2.0000000000000000,
|
||||
2.3219280948873622,
|
||||
2.5849625007211561,
|
||||
2.8073549220576042,
|
||||
3.0000000000000000,
|
||||
3.1699250014423126,
|
||||
3.3219280948873626,
|
||||
3.4594316186372978,
|
||||
3.5849625007211565,
|
||||
3.7004397181410922,
|
||||
3.8073549220576037,
|
||||
3.9068905956085187,
|
||||
4.0000000000000000,
|
||||
4.0874628412503400,
|
||||
4.1699250014423122,
|
||||
4.2479275134435852,
|
||||
4.3219280948873626,
|
||||
4.3923174227787607,
|
||||
4.4594316186372973,
|
||||
4.5235619560570131,
|
||||
4.5849625007211570,
|
||||
4.6438561897747244,
|
||||
4.7004397181410926,
|
||||
4.7548875021634691,
|
||||
4.8073549220576037,
|
||||
4.8579809951275728,
|
||||
4.9068905956085187,
|
||||
4.9541963103868758,
|
||||
5.0000000000000000,
|
||||
5.0443941193584534,
|
||||
5.0874628412503400,
|
||||
5.1292830169449664,
|
||||
5.1699250014423122,
|
||||
5.2094533656289501,
|
||||
5.2479275134435852,
|
||||
5.2854022188622487,
|
||||
5.3219280948873626,
|
||||
5.3575520046180838,
|
||||
5.3923174227787607,
|
||||
5.4262647547020979,
|
||||
5.4594316186372973,
|
||||
5.4918530963296748,
|
||||
5.5235619560570131,
|
||||
5.5545888516776376,
|
||||
5.5849625007211570,
|
||||
5.6147098441152083,
|
||||
5.6438561897747244,
|
||||
5.6724253419714961,
|
||||
5.7004397181410926,
|
||||
5.7279204545631996,
|
||||
5.7548875021634691,
|
||||
5.7813597135246599,
|
||||
5.8073549220576046,
|
||||
5.8328900141647422,
|
||||
5.8579809951275719,
|
||||
5.8826430493618416,
|
||||
5.9068905956085187,
|
||||
5.9307373375628867,
|
||||
5.9541963103868758,
|
||||
5.9772799234999168,
|
||||
6.0000000000000000,
|
||||
6.0223678130284544,
|
||||
6.0443941193584534,
|
||||
6.0660891904577721,
|
||||
6.0874628412503400,
|
||||
6.1085244567781700,
|
||||
6.1292830169449672,
|
||||
6.1497471195046822,
|
||||
6.1699250014423122,
|
||||
6.1898245588800176,
|
||||
6.2094533656289510,
|
||||
6.2288186904958804,
|
||||
6.2479275134435861,
|
||||
6.2667865406949019,
|
||||
6.2854022188622487,
|
||||
6.3037807481771031,
|
||||
6.3219280948873617,
|
||||
6.3398500028846252,
|
||||
6.3575520046180847,
|
||||
6.3750394313469254,
|
||||
6.3923174227787598,
|
||||
6.4093909361377026,
|
||||
6.4262647547020979,
|
||||
6.4429434958487288,
|
||||
6.4594316186372982,
|
||||
6.4757334309663976,
|
||||
6.4918530963296748,
|
||||
6.5077946401986964,
|
||||
6.5235619560570131,
|
||||
6.5391588111080319,
|
||||
6.5545888516776376,
|
||||
6.5698556083309478,
|
||||
6.5849625007211561,
|
||||
6.5999128421871278,
|
||||
6.6147098441152092,
|
||||
6.6293566200796095,
|
||||
6.6438561897747253,
|
||||
6.6582114827517955,
|
||||
6.6724253419714952,
|
||||
6.6865005271832185,
|
||||
6.7004397181410917,
|
||||
6.7142455176661224,
|
||||
6.7279204545631988,
|
||||
6.7414669864011465,
|
||||
6.7548875021634691,
|
||||
6.7681843247769260,
|
||||
6.7813597135246599,
|
||||
6.7944158663501062,
|
||||
6.8073549220576037,
|
||||
6.8201789624151887,
|
||||
6.8328900141647422,
|
||||
6.8454900509443757,
|
||||
6.8579809951275719,
|
||||
6.8703647195834048,
|
||||
6.8826430493618416,
|
||||
6.8948177633079437,
|
||||
6.9068905956085187,
|
||||
6.9188632372745955,
|
||||
6.9307373375628867,
|
||||
6.9425145053392399,
|
||||
6.9541963103868758,
|
||||
6.9657842846620879,
|
||||
6.9772799234999168,
|
||||
6.9886846867721664,
|
||||
7.0000000000000000,
|
||||
7.0112272554232540,
|
||||
7.0223678130284544,
|
||||
7.0334230015374501,
|
||||
7.0443941193584534,
|
||||
7.0552824355011898,
|
||||
7.0660891904577721,
|
||||
7.0768155970508317,
|
||||
7.0874628412503400,
|
||||
7.0980320829605272,
|
||||
7.1085244567781700,
|
||||
7.1189410727235076,
|
||||
7.1292830169449664,
|
||||
7.1395513523987937,
|
||||
7.1497471195046822,
|
||||
7.1598713367783891,
|
||||
7.1699250014423130,
|
||||
7.1799090900149345,
|
||||
7.1898245588800176,
|
||||
7.1996723448363644,
|
||||
7.2094533656289492,
|
||||
7.2191685204621621,
|
||||
7.2288186904958804,
|
||||
7.2384047393250794,
|
||||
7.2479275134435861,
|
||||
7.2573878426926521,
|
||||
7.2667865406949019,
|
||||
7.2761244052742384,
|
||||
7.2854022188622487,
|
||||
7.2946207488916270,
|
||||
7.3037807481771031,
|
||||
7.3128829552843557,
|
||||
7.3219280948873617,
|
||||
7.3309168781146177,
|
||||
7.3398500028846243,
|
||||
7.3487281542310781,
|
||||
7.3575520046180847,
|
||||
7.3663222142458151,
|
||||
7.3750394313469254,
|
||||
7.3837042924740528,
|
||||
7.3923174227787607,
|
||||
7.4008794362821844,
|
||||
7.4093909361377026,
|
||||
7.4178525148858991,
|
||||
7.4262647547020979,
|
||||
7.4346282276367255,
|
||||
7.4429434958487288,
|
||||
7.4512111118323299,
|
||||
7.4594316186372973,
|
||||
7.4676055500829976,
|
||||
7.4757334309663976,
|
||||
7.4838157772642564,
|
||||
7.4918530963296748,
|
||||
7.4998458870832057,
|
||||
7.5077946401986964,
|
||||
7.5156998382840436,
|
||||
7.5235619560570131,
|
||||
7.5313814605163119,
|
||||
7.5391588111080319,
|
||||
7.5468944598876373,
|
||||
7.5545888516776376,
|
||||
7.5622424242210728,
|
||||
7.5698556083309478,
|
||||
7.5774288280357487,
|
||||
7.5849625007211561,
|
||||
7.5924570372680806,
|
||||
7.5999128421871278,
|
||||
7.6073303137496113,
|
||||
7.6147098441152075,
|
||||
7.6220518194563764,
|
||||
7.6293566200796095,
|
||||
7.6366246205436488,
|
||||
7.6438561897747244,
|
||||
7.6510516911789290,
|
||||
7.6582114827517955,
|
||||
7.6653359171851765,
|
||||
7.6724253419714952,
|
||||
7.6794800995054464,
|
||||
7.6865005271832185,
|
||||
7.6934869574993252,
|
||||
7.7004397181410926,
|
||||
7.7073591320808825,
|
||||
7.7142455176661224,
|
||||
7.7210991887071856,
|
||||
7.7279204545631996,
|
||||
7.7347096202258392,
|
||||
7.7414669864011465,
|
||||
7.7481928495894596,
|
||||
7.7548875021634691,
|
||||
7.7615512324444795,
|
||||
7.7681843247769260,
|
||||
7.7747870596011737,
|
||||
7.7813597135246608,
|
||||
7.7879025593914317,
|
||||
7.7944158663501062,
|
||||
7.8008998999203047,
|
||||
7.8073549220576037,
|
||||
7.8137811912170374,
|
||||
7.8201789624151887,
|
||||
7.8265484872909159,
|
||||
7.8328900141647422,
|
||||
7.8392037880969445,
|
||||
7.8454900509443757,
|
||||
7.8517490414160571,
|
||||
7.8579809951275719,
|
||||
7.8641861446542798,
|
||||
7.8703647195834048,
|
||||
7.8765169465650002,
|
||||
7.8826430493618425,
|
||||
7.8887432488982601,
|
||||
7.8948177633079446,
|
||||
7.9008668079807496,
|
||||
7.9068905956085187,
|
||||
7.9128893362299619,
|
||||
7.9188632372745955,
|
||||
7.9248125036057813,
|
||||
7.9307373375628867,
|
||||
7.9366379390025719,
|
||||
7.9425145053392399,
|
||||
7.9483672315846778,
|
||||
7.9541963103868758,
|
||||
7.9600019320680806,
|
||||
7.9657842846620870,
|
||||
7.9715435539507720,
|
||||
7.9772799234999168,
|
||||
7.9829935746943104,
|
||||
7.9886846867721664,
|
||||
7.9943534368588578,
|
||||
}
|
||||
|
||||
/* Faster logarithm for small integers, with the property of log2(0) == 0. */
|
||||
func fastLog2(v uint) float64 {
|
||||
if v < uint(len(kLog2Table)) {
|
||||
return float64(kLog2Table[v])
|
||||
}
|
||||
|
||||
return math.Log2(float64(v))
|
||||
}
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
package brotli
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"math/bits"
|
||||
"runtime"
|
||||
)
|
||||
|
||||
/* Copyright 2010 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Function to find maximal matching prefixes of strings. */
|
||||
func findMatchLengthWithLimit(s1 []byte, s2 []byte, limit uint) uint {
|
||||
var matched uint = 0
|
||||
_, _ = s1[limit-1], s2[limit-1] // bounds check
|
||||
switch runtime.GOARCH {
|
||||
case "amd64":
|
||||
// Compare 8 bytes at at time.
|
||||
for matched+8 <= limit {
|
||||
w1 := binary.LittleEndian.Uint64(s1[matched:])
|
||||
w2 := binary.LittleEndian.Uint64(s2[matched:])
|
||||
if w1 != w2 {
|
||||
return matched + uint(bits.TrailingZeros64(w1^w2)>>3)
|
||||
}
|
||||
matched += 8
|
||||
}
|
||||
case "386":
|
||||
// Compare 4 bytes at at time.
|
||||
for matched+4 <= limit {
|
||||
w1 := binary.LittleEndian.Uint32(s1[matched:])
|
||||
w2 := binary.LittleEndian.Uint32(s2[matched:])
|
||||
if w1 != w2 {
|
||||
return matched + uint(bits.TrailingZeros32(w1^w2)>>3)
|
||||
}
|
||||
matched += 4
|
||||
}
|
||||
}
|
||||
for matched < limit && s1[matched] == s2[matched] {
|
||||
matched++
|
||||
}
|
||||
return matched
|
||||
}
|
||||
+287
@@ -0,0 +1,287 @@
|
||||
package brotli
|
||||
|
||||
import "encoding/binary"
|
||||
|
||||
/* Copyright 2016 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
func (*h10) HashTypeLength() uint {
|
||||
return 4
|
||||
}
|
||||
|
||||
func (*h10) StoreLookahead() uint {
|
||||
return 128
|
||||
}
|
||||
|
||||
func hashBytesH10(data []byte) uint32 {
|
||||
var h uint32 = binary.LittleEndian.Uint32(data) * kHashMul32
|
||||
|
||||
/* The higher bits contain more mixture from the multiplication,
|
||||
so we take our results from there. */
|
||||
return h >> (32 - 17)
|
||||
}
|
||||
|
||||
/* A (forgetful) hash table where each hash bucket contains a binary tree of
|
||||
sequences whose first 4 bytes share the same hash code.
|
||||
Each sequence is 128 long and is identified by its starting
|
||||
position in the input data. The binary tree is sorted by the lexicographic
|
||||
order of the sequences, and it is also a max-heap with respect to the
|
||||
starting positions. */
|
||||
type h10 struct {
|
||||
hasherCommon
|
||||
window_mask_ uint
|
||||
buckets_ [1 << 17]uint32
|
||||
invalid_pos_ uint32
|
||||
forest []uint32
|
||||
}
|
||||
|
||||
func (h *h10) Initialize(params *encoderParams) {
|
||||
h.window_mask_ = (1 << params.lgwin) - 1
|
||||
h.invalid_pos_ = uint32(0 - h.window_mask_)
|
||||
var num_nodes uint = uint(1) << params.lgwin
|
||||
h.forest = make([]uint32, 2*num_nodes)
|
||||
}
|
||||
|
||||
func (h *h10) Prepare(one_shot bool, input_size uint, data []byte) {
|
||||
var invalid_pos uint32 = h.invalid_pos_
|
||||
var i uint32
|
||||
for i = 0; i < 1<<17; i++ {
|
||||
h.buckets_[i] = invalid_pos
|
||||
}
|
||||
}
|
||||
|
||||
func leftChildIndexH10(self *h10, pos uint) uint {
|
||||
return 2 * (pos & self.window_mask_)
|
||||
}
|
||||
|
||||
func rightChildIndexH10(self *h10, pos uint) uint {
|
||||
return 2*(pos&self.window_mask_) + 1
|
||||
}
|
||||
|
||||
/* Stores the hash of the next 4 bytes and in a single tree-traversal, the
|
||||
hash bucket's binary tree is searched for matches and is re-rooted at the
|
||||
current position.
|
||||
|
||||
If less than 128 data is available, the hash bucket of the
|
||||
current position is searched for matches, but the state of the hash table
|
||||
is not changed, since we can not know the final sorting order of the
|
||||
current (incomplete) sequence.
|
||||
|
||||
This function must be called with increasing cur_ix positions. */
|
||||
func storeAndFindMatchesH10(self *h10, data []byte, cur_ix uint, ring_buffer_mask uint, max_length uint, max_backward uint, best_len *uint, matches []backwardMatch) []backwardMatch {
|
||||
var cur_ix_masked uint = cur_ix & ring_buffer_mask
|
||||
var max_comp_len uint = brotli_min_size_t(max_length, 128)
|
||||
var should_reroot_tree bool = (max_length >= 128)
|
||||
var key uint32 = hashBytesH10(data[cur_ix_masked:])
|
||||
var forest []uint32 = self.forest
|
||||
var prev_ix uint = uint(self.buckets_[key])
|
||||
var node_left uint = leftChildIndexH10(self, cur_ix)
|
||||
var node_right uint = rightChildIndexH10(self, cur_ix)
|
||||
var best_len_left uint = 0
|
||||
var best_len_right uint = 0
|
||||
var depth_remaining uint
|
||||
/* The forest index of the rightmost node of the left subtree of the new
|
||||
root, updated as we traverse and re-root the tree of the hash bucket. */
|
||||
|
||||
/* The forest index of the leftmost node of the right subtree of the new
|
||||
root, updated as we traverse and re-root the tree of the hash bucket. */
|
||||
|
||||
/* The match length of the rightmost node of the left subtree of the new
|
||||
root, updated as we traverse and re-root the tree of the hash bucket. */
|
||||
|
||||
/* The match length of the leftmost node of the right subtree of the new
|
||||
root, updated as we traverse and re-root the tree of the hash bucket. */
|
||||
if should_reroot_tree {
|
||||
self.buckets_[key] = uint32(cur_ix)
|
||||
}
|
||||
|
||||
for depth_remaining = 64; ; depth_remaining-- {
|
||||
var backward uint = cur_ix - prev_ix
|
||||
var prev_ix_masked uint = prev_ix & ring_buffer_mask
|
||||
if backward == 0 || backward > max_backward || depth_remaining == 0 {
|
||||
if should_reroot_tree {
|
||||
forest[node_left] = self.invalid_pos_
|
||||
forest[node_right] = self.invalid_pos_
|
||||
}
|
||||
|
||||
break
|
||||
}
|
||||
{
|
||||
var cur_len uint = brotli_min_size_t(best_len_left, best_len_right)
|
||||
var len uint
|
||||
assert(cur_len <= 128)
|
||||
len = cur_len + findMatchLengthWithLimit(data[cur_ix_masked+cur_len:], data[prev_ix_masked+cur_len:], max_length-cur_len)
|
||||
if matches != nil && len > *best_len {
|
||||
*best_len = uint(len)
|
||||
initBackwardMatch(&matches[0], backward, uint(len))
|
||||
matches = matches[1:]
|
||||
}
|
||||
|
||||
if len >= max_comp_len {
|
||||
if should_reroot_tree {
|
||||
forest[node_left] = forest[leftChildIndexH10(self, prev_ix)]
|
||||
forest[node_right] = forest[rightChildIndexH10(self, prev_ix)]
|
||||
}
|
||||
|
||||
break
|
||||
}
|
||||
|
||||
if data[cur_ix_masked+len] > data[prev_ix_masked+len] {
|
||||
best_len_left = uint(len)
|
||||
if should_reroot_tree {
|
||||
forest[node_left] = uint32(prev_ix)
|
||||
}
|
||||
|
||||
node_left = rightChildIndexH10(self, prev_ix)
|
||||
prev_ix = uint(forest[node_left])
|
||||
} else {
|
||||
best_len_right = uint(len)
|
||||
if should_reroot_tree {
|
||||
forest[node_right] = uint32(prev_ix)
|
||||
}
|
||||
|
||||
node_right = leftChildIndexH10(self, prev_ix)
|
||||
prev_ix = uint(forest[node_right])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return matches
|
||||
}
|
||||
|
||||
/* Finds all backward matches of &data[cur_ix & ring_buffer_mask] up to the
|
||||
length of max_length and stores the position cur_ix in the hash table.
|
||||
|
||||
Sets *num_matches to the number of matches found, and stores the found
|
||||
matches in matches[0] to matches[*num_matches - 1]. The matches will be
|
||||
sorted by strictly increasing length and (non-strictly) increasing
|
||||
distance. */
|
||||
func findAllMatchesH10(handle *h10, dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, cur_ix uint, max_length uint, max_backward uint, gap uint, params *encoderParams, matches []backwardMatch) uint {
|
||||
var orig_matches []backwardMatch = matches
|
||||
var cur_ix_masked uint = cur_ix & ring_buffer_mask
|
||||
var best_len uint = 1
|
||||
var short_match_max_backward uint
|
||||
if params.quality != hqZopflificationQuality {
|
||||
short_match_max_backward = 16
|
||||
} else {
|
||||
short_match_max_backward = 64
|
||||
}
|
||||
var stop uint = cur_ix - short_match_max_backward
|
||||
var dict_matches [maxStaticDictionaryMatchLen + 1]uint32
|
||||
var i uint
|
||||
if cur_ix < short_match_max_backward {
|
||||
stop = 0
|
||||
}
|
||||
for i = cur_ix - 1; i > stop && best_len <= 2; i-- {
|
||||
var prev_ix uint = i
|
||||
var backward uint = cur_ix - prev_ix
|
||||
if backward > max_backward {
|
||||
break
|
||||
}
|
||||
|
||||
prev_ix &= ring_buffer_mask
|
||||
if data[cur_ix_masked] != data[prev_ix] || data[cur_ix_masked+1] != data[prev_ix+1] {
|
||||
continue
|
||||
}
|
||||
{
|
||||
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
|
||||
if len > best_len {
|
||||
best_len = uint(len)
|
||||
initBackwardMatch(&matches[0], backward, uint(len))
|
||||
matches = matches[1:]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if best_len < max_length {
|
||||
matches = storeAndFindMatchesH10(handle, data, cur_ix, ring_buffer_mask, max_length, max_backward, &best_len, matches)
|
||||
}
|
||||
|
||||
for i = 0; i <= maxStaticDictionaryMatchLen; i++ {
|
||||
dict_matches[i] = kInvalidMatch
|
||||
}
|
||||
{
|
||||
var minlen uint = brotli_max_size_t(4, best_len+1)
|
||||
if findAllStaticDictionaryMatches(dictionary, data[cur_ix_masked:], minlen, max_length, dict_matches[0:]) {
|
||||
var maxlen uint = brotli_min_size_t(maxStaticDictionaryMatchLen, max_length)
|
||||
var l uint
|
||||
for l = minlen; l <= maxlen; l++ {
|
||||
var dict_id uint32 = dict_matches[l]
|
||||
if dict_id < kInvalidMatch {
|
||||
var distance uint = max_backward + gap + uint(dict_id>>5) + 1
|
||||
if distance <= params.dist.max_distance {
|
||||
initDictionaryBackwardMatch(&matches[0], distance, l, uint(dict_id&31))
|
||||
matches = matches[1:]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return uint(-cap(matches) + cap(orig_matches))
|
||||
}
|
||||
|
||||
/* Stores the hash of the next 4 bytes and re-roots the binary tree at the
|
||||
current sequence, without returning any matches.
|
||||
REQUIRES: ix + 128 <= end-of-current-block */
|
||||
func (h *h10) Store(data []byte, mask uint, ix uint) {
|
||||
var max_backward uint = h.window_mask_ - windowGap + 1
|
||||
/* Maximum distance is window size - 16, see section 9.1. of the spec. */
|
||||
storeAndFindMatchesH10(h, data, ix, mask, 128, max_backward, nil, nil)
|
||||
}
|
||||
|
||||
func (h *h10) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
|
||||
var i uint = ix_start
|
||||
var j uint = ix_start
|
||||
if ix_start+63 <= ix_end {
|
||||
i = ix_end - 63
|
||||
}
|
||||
|
||||
if ix_start+512 <= i {
|
||||
for ; j < i; j += 8 {
|
||||
h.Store(data, mask, j)
|
||||
}
|
||||
}
|
||||
|
||||
for ; i < ix_end; i++ {
|
||||
h.Store(data, mask, i)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *h10) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint) {
|
||||
if num_bytes >= h.HashTypeLength()-1 && position >= 128 {
|
||||
var i_start uint = position - 128 + 1
|
||||
var i_end uint = brotli_min_size_t(position, i_start+num_bytes)
|
||||
/* Store the last `128 - 1` positions in the hasher.
|
||||
These could not be calculated before, since they require knowledge
|
||||
of both the previous and the current block. */
|
||||
|
||||
var i uint
|
||||
for i = i_start; i < i_end; i++ {
|
||||
/* Maximum distance is window size - 16, see section 9.1. of the spec.
|
||||
Furthermore, we have to make sure that we don't look further back
|
||||
from the start of the next block than the window size, otherwise we
|
||||
could access already overwritten areas of the ring-buffer. */
|
||||
var max_backward uint = h.window_mask_ - brotli_max_size_t(windowGap-1, position-i)
|
||||
|
||||
/* We know that i + 128 <= position + num_bytes, i.e. the
|
||||
end of the current block and that we have at least
|
||||
128 tail in the ring-buffer. */
|
||||
storeAndFindMatchesH10(h, ringbuffer, i, ringbuffer_mask, 128, max_backward, nil, nil)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* MAX_NUM_MATCHES == 64 + MAX_TREE_SEARCH_DEPTH */
|
||||
const maxNumMatchesH10 = 128
|
||||
|
||||
func (*h10) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
|
||||
panic("unimplemented")
|
||||
}
|
||||
|
||||
func (*h10) PrepareDistanceCache(distance_cache []int) {
|
||||
panic("unimplemented")
|
||||
}
|
||||
+214
@@ -0,0 +1,214 @@
|
||||
package brotli
|
||||
|
||||
import "encoding/binary"
|
||||
|
||||
/* Copyright 2010 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* A (forgetful) hash table to the data seen by the compressor, to
|
||||
help create backward references to previous data.
|
||||
|
||||
This is a hash map of fixed size (bucket_size_) to a ring buffer of
|
||||
fixed size (block_size_). The ring buffer contains the last block_size_
|
||||
index positions of the given hash key in the compressed data. */
|
||||
func (*h5) HashTypeLength() uint {
|
||||
return 4
|
||||
}
|
||||
|
||||
func (*h5) StoreLookahead() uint {
|
||||
return 4
|
||||
}
|
||||
|
||||
/* HashBytes is the function that chooses the bucket to place the address in. */
|
||||
func hashBytesH5(data []byte, shift int) uint32 {
|
||||
var h uint32 = binary.LittleEndian.Uint32(data) * kHashMul32
|
||||
|
||||
/* The higher bits contain more mixture from the multiplication,
|
||||
so we take our results from there. */
|
||||
return uint32(h >> uint(shift))
|
||||
}
|
||||
|
||||
type h5 struct {
|
||||
hasherCommon
|
||||
bucket_size_ uint
|
||||
block_size_ uint
|
||||
hash_shift_ int
|
||||
block_mask_ uint32
|
||||
num []uint16
|
||||
buckets []uint32
|
||||
}
|
||||
|
||||
func (h *h5) Initialize(params *encoderParams) {
|
||||
h.hash_shift_ = 32 - h.params.bucket_bits
|
||||
h.bucket_size_ = uint(1) << uint(h.params.bucket_bits)
|
||||
h.block_size_ = uint(1) << uint(h.params.block_bits)
|
||||
h.block_mask_ = uint32(h.block_size_ - 1)
|
||||
h.num = make([]uint16, h.bucket_size_)
|
||||
h.buckets = make([]uint32, h.block_size_*h.bucket_size_)
|
||||
}
|
||||
|
||||
func (h *h5) Prepare(one_shot bool, input_size uint, data []byte) {
|
||||
var num []uint16 = h.num
|
||||
var partial_prepare_threshold uint = h.bucket_size_ >> 6
|
||||
/* Partial preparation is 100 times slower (per socket). */
|
||||
if one_shot && input_size <= partial_prepare_threshold {
|
||||
var i uint
|
||||
for i = 0; i < input_size; i++ {
|
||||
var key uint32 = hashBytesH5(data[i:], h.hash_shift_)
|
||||
num[key] = 0
|
||||
}
|
||||
} else {
|
||||
for i := 0; i < int(h.bucket_size_); i++ {
|
||||
num[i] = 0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Look at 4 bytes at &data[ix & mask].
|
||||
Compute a hash from these, and store the value of ix at that position. */
|
||||
func (h *h5) Store(data []byte, mask uint, ix uint) {
|
||||
var num []uint16 = h.num
|
||||
var key uint32 = hashBytesH5(data[ix&mask:], h.hash_shift_)
|
||||
var minor_ix uint = uint(num[key]) & uint(h.block_mask_)
|
||||
var offset uint = minor_ix + uint(key<<uint(h.params.block_bits))
|
||||
h.buckets[offset] = uint32(ix)
|
||||
num[key]++
|
||||
}
|
||||
|
||||
func (h *h5) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
|
||||
var i uint
|
||||
for i = ix_start; i < ix_end; i++ {
|
||||
h.Store(data, mask, i)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *h5) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint) {
|
||||
if num_bytes >= h.HashTypeLength()-1 && position >= 3 {
|
||||
/* Prepare the hashes for three last bytes of the last write.
|
||||
These could not be calculated before, since they require knowledge
|
||||
of both the previous and the current block. */
|
||||
h.Store(ringbuffer, ringbuffer_mask, position-3)
|
||||
h.Store(ringbuffer, ringbuffer_mask, position-2)
|
||||
h.Store(ringbuffer, ringbuffer_mask, position-1)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *h5) PrepareDistanceCache(distance_cache []int) {
|
||||
prepareDistanceCache(distance_cache, h.params.num_last_distances_to_check)
|
||||
}
|
||||
|
||||
/* Find a longest backward match of &data[cur_ix] up to the length of
|
||||
max_length and stores the position cur_ix in the hash table.
|
||||
|
||||
REQUIRES: PrepareDistanceCacheH5 must be invoked for current distance cache
|
||||
values; if this method is invoked repeatedly with the same distance
|
||||
cache values, it is enough to invoke PrepareDistanceCacheH5 once.
|
||||
|
||||
Does not look for matches longer than max_length.
|
||||
Does not look for matches further away than max_backward.
|
||||
Writes the best match into |out|.
|
||||
|out|->score is updated only if a better match is found. */
|
||||
func (h *h5) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
|
||||
var num []uint16 = h.num
|
||||
var buckets []uint32 = h.buckets
|
||||
var cur_ix_masked uint = cur_ix & ring_buffer_mask
|
||||
var min_score uint = out.score
|
||||
var best_score uint = out.score
|
||||
var best_len uint = out.len
|
||||
var i uint
|
||||
var bucket []uint32
|
||||
/* Don't accept a short copy from far away. */
|
||||
out.len = 0
|
||||
|
||||
out.len_code_delta = 0
|
||||
|
||||
/* Try last distance first. */
|
||||
for i = 0; i < uint(h.params.num_last_distances_to_check); i++ {
|
||||
var backward uint = uint(distance_cache[i])
|
||||
var prev_ix uint = uint(cur_ix - backward)
|
||||
if prev_ix >= cur_ix {
|
||||
continue
|
||||
}
|
||||
|
||||
if backward > max_backward {
|
||||
continue
|
||||
}
|
||||
|
||||
prev_ix &= ring_buffer_mask
|
||||
|
||||
if cur_ix_masked+best_len > ring_buffer_mask || prev_ix+best_len > ring_buffer_mask || data[cur_ix_masked+best_len] != data[prev_ix+best_len] {
|
||||
continue
|
||||
}
|
||||
{
|
||||
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
|
||||
if len >= 3 || (len == 2 && i < 2) {
|
||||
/* Comparing for >= 2 does not change the semantics, but just saves for
|
||||
a few unnecessary binary logarithms in backward reference score,
|
||||
since we are not interested in such short matches. */
|
||||
var score uint = backwardReferenceScoreUsingLastDistance(uint(len))
|
||||
if best_score < score {
|
||||
if i != 0 {
|
||||
score -= backwardReferencePenaltyUsingLastDistance(i)
|
||||
}
|
||||
if best_score < score {
|
||||
best_score = score
|
||||
best_len = uint(len)
|
||||
out.len = best_len
|
||||
out.distance = backward
|
||||
out.score = best_score
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
var key uint32 = hashBytesH5(data[cur_ix_masked:], h.hash_shift_)
|
||||
bucket = buckets[key<<uint(h.params.block_bits):]
|
||||
var down uint
|
||||
if uint(num[key]) > h.block_size_ {
|
||||
down = uint(num[key]) - h.block_size_
|
||||
} else {
|
||||
down = 0
|
||||
}
|
||||
for i = uint(num[key]); i > down; {
|
||||
var prev_ix uint
|
||||
i--
|
||||
prev_ix = uint(bucket[uint32(i)&h.block_mask_])
|
||||
var backward uint = cur_ix - prev_ix
|
||||
if backward > max_backward {
|
||||
break
|
||||
}
|
||||
|
||||
prev_ix &= ring_buffer_mask
|
||||
if cur_ix_masked+best_len > ring_buffer_mask || prev_ix+best_len > ring_buffer_mask || data[cur_ix_masked+best_len] != data[prev_ix+best_len] {
|
||||
continue
|
||||
}
|
||||
{
|
||||
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
|
||||
if len >= 4 {
|
||||
/* Comparing for >= 3 does not change the semantics, but just saves
|
||||
for a few unnecessary binary logarithms in backward reference
|
||||
score, since we are not interested in such short matches. */
|
||||
var score uint = backwardReferenceScore(uint(len), backward)
|
||||
if best_score < score {
|
||||
best_score = score
|
||||
best_len = uint(len)
|
||||
out.len = best_len
|
||||
out.distance = backward
|
||||
out.score = best_score
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
bucket[uint32(num[key])&h.block_mask_] = uint32(cur_ix)
|
||||
num[key]++
|
||||
}
|
||||
|
||||
if min_score == out.score {
|
||||
searchInStaticDictionary(dictionary, h, data[cur_ix_masked:], max_length, max_backward+gap, max_distance, out, false)
|
||||
}
|
||||
}
|
||||
+216
@@ -0,0 +1,216 @@
|
||||
package brotli
|
||||
|
||||
import "encoding/binary"
|
||||
|
||||
/* Copyright 2010 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* A (forgetful) hash table to the data seen by the compressor, to
|
||||
help create backward references to previous data.
|
||||
|
||||
This is a hash map of fixed size (bucket_size_) to a ring buffer of
|
||||
fixed size (block_size_). The ring buffer contains the last block_size_
|
||||
index positions of the given hash key in the compressed data. */
|
||||
func (*h6) HashTypeLength() uint {
|
||||
return 8
|
||||
}
|
||||
|
||||
func (*h6) StoreLookahead() uint {
|
||||
return 8
|
||||
}
|
||||
|
||||
/* HashBytes is the function that chooses the bucket to place the address in. */
|
||||
func hashBytesH6(data []byte, mask uint64, shift int) uint32 {
|
||||
var h uint64 = (binary.LittleEndian.Uint64(data) & mask) * kHashMul64Long
|
||||
|
||||
/* The higher bits contain more mixture from the multiplication,
|
||||
so we take our results from there. */
|
||||
return uint32(h >> uint(shift))
|
||||
}
|
||||
|
||||
type h6 struct {
|
||||
hasherCommon
|
||||
bucket_size_ uint
|
||||
block_size_ uint
|
||||
hash_shift_ int
|
||||
hash_mask_ uint64
|
||||
block_mask_ uint32
|
||||
num []uint16
|
||||
buckets []uint32
|
||||
}
|
||||
|
||||
func (h *h6) Initialize(params *encoderParams) {
|
||||
h.hash_shift_ = 64 - h.params.bucket_bits
|
||||
h.hash_mask_ = (^(uint64(0))) >> uint(64-8*h.params.hash_len)
|
||||
h.bucket_size_ = uint(1) << uint(h.params.bucket_bits)
|
||||
h.block_size_ = uint(1) << uint(h.params.block_bits)
|
||||
h.block_mask_ = uint32(h.block_size_ - 1)
|
||||
h.num = make([]uint16, h.bucket_size_)
|
||||
h.buckets = make([]uint32, h.block_size_*h.bucket_size_)
|
||||
}
|
||||
|
||||
func (h *h6) Prepare(one_shot bool, input_size uint, data []byte) {
|
||||
var num []uint16 = h.num
|
||||
var partial_prepare_threshold uint = h.bucket_size_ >> 6
|
||||
/* Partial preparation is 100 times slower (per socket). */
|
||||
if one_shot && input_size <= partial_prepare_threshold {
|
||||
var i uint
|
||||
for i = 0; i < input_size; i++ {
|
||||
var key uint32 = hashBytesH6(data[i:], h.hash_mask_, h.hash_shift_)
|
||||
num[key] = 0
|
||||
}
|
||||
} else {
|
||||
for i := 0; i < int(h.bucket_size_); i++ {
|
||||
num[i] = 0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Look at 4 bytes at &data[ix & mask].
|
||||
Compute a hash from these, and store the value of ix at that position. */
|
||||
func (h *h6) Store(data []byte, mask uint, ix uint) {
|
||||
var num []uint16 = h.num
|
||||
var key uint32 = hashBytesH6(data[ix&mask:], h.hash_mask_, h.hash_shift_)
|
||||
var minor_ix uint = uint(num[key]) & uint(h.block_mask_)
|
||||
var offset uint = minor_ix + uint(key<<uint(h.params.block_bits))
|
||||
h.buckets[offset] = uint32(ix)
|
||||
num[key]++
|
||||
}
|
||||
|
||||
func (h *h6) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
|
||||
var i uint
|
||||
for i = ix_start; i < ix_end; i++ {
|
||||
h.Store(data, mask, i)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *h6) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint) {
|
||||
if num_bytes >= h.HashTypeLength()-1 && position >= 3 {
|
||||
/* Prepare the hashes for three last bytes of the last write.
|
||||
These could not be calculated before, since they require knowledge
|
||||
of both the previous and the current block. */
|
||||
h.Store(ringbuffer, ringbuffer_mask, position-3)
|
||||
h.Store(ringbuffer, ringbuffer_mask, position-2)
|
||||
h.Store(ringbuffer, ringbuffer_mask, position-1)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *h6) PrepareDistanceCache(distance_cache []int) {
|
||||
prepareDistanceCache(distance_cache, h.params.num_last_distances_to_check)
|
||||
}
|
||||
|
||||
/* Find a longest backward match of &data[cur_ix] up to the length of
|
||||
max_length and stores the position cur_ix in the hash table.
|
||||
|
||||
REQUIRES: PrepareDistanceCacheH6 must be invoked for current distance cache
|
||||
values; if this method is invoked repeatedly with the same distance
|
||||
cache values, it is enough to invoke PrepareDistanceCacheH6 once.
|
||||
|
||||
Does not look for matches longer than max_length.
|
||||
Does not look for matches further away than max_backward.
|
||||
Writes the best match into |out|.
|
||||
|out|->score is updated only if a better match is found. */
|
||||
func (h *h6) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
|
||||
var num []uint16 = h.num
|
||||
var buckets []uint32 = h.buckets
|
||||
var cur_ix_masked uint = cur_ix & ring_buffer_mask
|
||||
var min_score uint = out.score
|
||||
var best_score uint = out.score
|
||||
var best_len uint = out.len
|
||||
var i uint
|
||||
var bucket []uint32
|
||||
/* Don't accept a short copy from far away. */
|
||||
out.len = 0
|
||||
|
||||
out.len_code_delta = 0
|
||||
|
||||
/* Try last distance first. */
|
||||
for i = 0; i < uint(h.params.num_last_distances_to_check); i++ {
|
||||
var backward uint = uint(distance_cache[i])
|
||||
var prev_ix uint = uint(cur_ix - backward)
|
||||
if prev_ix >= cur_ix {
|
||||
continue
|
||||
}
|
||||
|
||||
if backward > max_backward {
|
||||
continue
|
||||
}
|
||||
|
||||
prev_ix &= ring_buffer_mask
|
||||
|
||||
if cur_ix_masked+best_len > ring_buffer_mask || prev_ix+best_len > ring_buffer_mask || data[cur_ix_masked+best_len] != data[prev_ix+best_len] {
|
||||
continue
|
||||
}
|
||||
{
|
||||
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
|
||||
if len >= 3 || (len == 2 && i < 2) {
|
||||
/* Comparing for >= 2 does not change the semantics, but just saves for
|
||||
a few unnecessary binary logarithms in backward reference score,
|
||||
since we are not interested in such short matches. */
|
||||
var score uint = backwardReferenceScoreUsingLastDistance(uint(len))
|
||||
if best_score < score {
|
||||
if i != 0 {
|
||||
score -= backwardReferencePenaltyUsingLastDistance(i)
|
||||
}
|
||||
if best_score < score {
|
||||
best_score = score
|
||||
best_len = uint(len)
|
||||
out.len = best_len
|
||||
out.distance = backward
|
||||
out.score = best_score
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
var key uint32 = hashBytesH6(data[cur_ix_masked:], h.hash_mask_, h.hash_shift_)
|
||||
bucket = buckets[key<<uint(h.params.block_bits):]
|
||||
var down uint
|
||||
if uint(num[key]) > h.block_size_ {
|
||||
down = uint(num[key]) - h.block_size_
|
||||
} else {
|
||||
down = 0
|
||||
}
|
||||
for i = uint(num[key]); i > down; {
|
||||
var prev_ix uint
|
||||
i--
|
||||
prev_ix = uint(bucket[uint32(i)&h.block_mask_])
|
||||
var backward uint = cur_ix - prev_ix
|
||||
if backward > max_backward {
|
||||
break
|
||||
}
|
||||
|
||||
prev_ix &= ring_buffer_mask
|
||||
if cur_ix_masked+best_len > ring_buffer_mask || prev_ix+best_len > ring_buffer_mask || data[cur_ix_masked+best_len] != data[prev_ix+best_len] {
|
||||
continue
|
||||
}
|
||||
{
|
||||
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
|
||||
if len >= 4 {
|
||||
/* Comparing for >= 3 does not change the semantics, but just saves
|
||||
for a few unnecessary binary logarithms in backward reference
|
||||
score, since we are not interested in such short matches. */
|
||||
var score uint = backwardReferenceScore(uint(len), backward)
|
||||
if best_score < score {
|
||||
best_score = score
|
||||
best_len = uint(len)
|
||||
out.len = best_len
|
||||
out.distance = backward
|
||||
out.score = best_score
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
bucket[uint32(num[key])&h.block_mask_] = uint32(cur_ix)
|
||||
num[key]++
|
||||
}
|
||||
|
||||
if min_score == out.score {
|
||||
searchInStaticDictionary(dictionary, h, data[cur_ix_masked:], max_length, max_backward+gap, max_distance, out, false)
|
||||
}
|
||||
}
|
||||
+342
@@ -0,0 +1,342 @@
|
||||
package brotli
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
type hasherCommon struct {
|
||||
params hasherParams
|
||||
is_prepared_ bool
|
||||
dict_num_lookups uint
|
||||
dict_num_matches uint
|
||||
}
|
||||
|
||||
func (h *hasherCommon) Common() *hasherCommon {
|
||||
return h
|
||||
}
|
||||
|
||||
type hasherHandle interface {
|
||||
Common() *hasherCommon
|
||||
Initialize(params *encoderParams)
|
||||
Prepare(one_shot bool, input_size uint, data []byte)
|
||||
StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint)
|
||||
HashTypeLength() uint
|
||||
StoreLookahead() uint
|
||||
PrepareDistanceCache(distance_cache []int)
|
||||
FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult)
|
||||
StoreRange(data []byte, mask uint, ix_start uint, ix_end uint)
|
||||
Store(data []byte, mask uint, ix uint)
|
||||
}
|
||||
|
||||
const kCutoffTransformsCount uint32 = 10
|
||||
|
||||
/* 0, 12, 27, 23, 42, 63, 56, 48, 59, 64 */
|
||||
/* 0+0, 4+8, 8+19, 12+11, 16+26, 20+43, 24+32, 28+20, 32+27, 36+28 */
|
||||
const kCutoffTransforms uint64 = 0x071B520ADA2D3200
|
||||
|
||||
type hasherSearchResult struct {
|
||||
len uint
|
||||
distance uint
|
||||
score uint
|
||||
len_code_delta int
|
||||
}
|
||||
|
||||
/* kHashMul32 multiplier has these properties:
|
||||
* The multiplier must be odd. Otherwise we may lose the highest bit.
|
||||
* No long streaks of ones or zeros.
|
||||
* There is no effort to ensure that it is a prime, the oddity is enough
|
||||
for this use.
|
||||
* The number has been tuned heuristically against compression benchmarks. */
|
||||
const kHashMul32 uint32 = 0x1E35A7BD
|
||||
|
||||
const kHashMul64 uint64 = 0x1E35A7BD1E35A7BD
|
||||
|
||||
const kHashMul64Long uint64 = 0x1FE35A7BD3579BD3
|
||||
|
||||
func hash14(data []byte) uint32 {
|
||||
var h uint32 = binary.LittleEndian.Uint32(data) * kHashMul32
|
||||
|
||||
/* The higher bits contain more mixture from the multiplication,
|
||||
so we take our results from there. */
|
||||
return h >> (32 - 14)
|
||||
}
|
||||
|
||||
func prepareDistanceCache(distance_cache []int, num_distances int) {
|
||||
if num_distances > 4 {
|
||||
var last_distance int = distance_cache[0]
|
||||
distance_cache[4] = last_distance - 1
|
||||
distance_cache[5] = last_distance + 1
|
||||
distance_cache[6] = last_distance - 2
|
||||
distance_cache[7] = last_distance + 2
|
||||
distance_cache[8] = last_distance - 3
|
||||
distance_cache[9] = last_distance + 3
|
||||
if num_distances > 10 {
|
||||
var next_last_distance int = distance_cache[1]
|
||||
distance_cache[10] = next_last_distance - 1
|
||||
distance_cache[11] = next_last_distance + 1
|
||||
distance_cache[12] = next_last_distance - 2
|
||||
distance_cache[13] = next_last_distance + 2
|
||||
distance_cache[14] = next_last_distance - 3
|
||||
distance_cache[15] = next_last_distance + 3
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const literalByteScore = 135
|
||||
|
||||
const distanceBitPenalty = 30
|
||||
|
||||
/* Score must be positive after applying maximal penalty. */
|
||||
const scoreBase = (distanceBitPenalty * 8 * 8)
|
||||
|
||||
/* Usually, we always choose the longest backward reference. This function
|
||||
allows for the exception of that rule.
|
||||
|
||||
If we choose a backward reference that is further away, it will
|
||||
usually be coded with more bits. We approximate this by assuming
|
||||
log2(distance). If the distance can be expressed in terms of the
|
||||
last four distances, we use some heuristic constants to estimate
|
||||
the bits cost. For the first up to four literals we use the bit
|
||||
cost of the literals from the literal cost model, after that we
|
||||
use the average bit cost of the cost model.
|
||||
|
||||
This function is used to sometimes discard a longer backward reference
|
||||
when it is not much longer and the bit cost for encoding it is more
|
||||
than the saved literals.
|
||||
|
||||
backward_reference_offset MUST be positive. */
|
||||
func backwardReferenceScore(copy_length uint, backward_reference_offset uint) uint {
|
||||
return scoreBase + literalByteScore*uint(copy_length) - distanceBitPenalty*uint(log2FloorNonZero(backward_reference_offset))
|
||||
}
|
||||
|
||||
func backwardReferenceScoreUsingLastDistance(copy_length uint) uint {
|
||||
return literalByteScore*uint(copy_length) + scoreBase + 15
|
||||
}
|
||||
|
||||
func backwardReferencePenaltyUsingLastDistance(distance_short_code uint) uint {
|
||||
return uint(39) + ((0x1CA10 >> (distance_short_code & 0xE)) & 0xE)
|
||||
}
|
||||
|
||||
func testStaticDictionaryItem(dictionary *encoderDictionary, item uint, data []byte, max_length uint, max_backward uint, max_distance uint, out *hasherSearchResult) bool {
|
||||
var len uint
|
||||
var word_idx uint
|
||||
var offset uint
|
||||
var matchlen uint
|
||||
var backward uint
|
||||
var score uint
|
||||
len = item & 0x1F
|
||||
word_idx = item >> 5
|
||||
offset = uint(dictionary.words.offsets_by_length[len]) + len*word_idx
|
||||
if len > max_length {
|
||||
return false
|
||||
}
|
||||
|
||||
matchlen = findMatchLengthWithLimit(data, dictionary.words.data[offset:], uint(len))
|
||||
if matchlen+uint(dictionary.cutoffTransformsCount) <= len || matchlen == 0 {
|
||||
return false
|
||||
}
|
||||
{
|
||||
var cut uint = len - matchlen
|
||||
var transform_id uint = (cut << 2) + uint((dictionary.cutoffTransforms>>(cut*6))&0x3F)
|
||||
backward = max_backward + 1 + word_idx + (transform_id << dictionary.words.size_bits_by_length[len])
|
||||
}
|
||||
|
||||
if backward > max_distance {
|
||||
return false
|
||||
}
|
||||
|
||||
score = backwardReferenceScore(matchlen, backward)
|
||||
if score < out.score {
|
||||
return false
|
||||
}
|
||||
|
||||
out.len = matchlen
|
||||
out.len_code_delta = int(len) - int(matchlen)
|
||||
out.distance = backward
|
||||
out.score = score
|
||||
return true
|
||||
}
|
||||
|
||||
func searchInStaticDictionary(dictionary *encoderDictionary, handle hasherHandle, data []byte, max_length uint, max_backward uint, max_distance uint, out *hasherSearchResult, shallow bool) {
|
||||
var key uint
|
||||
var i uint
|
||||
var self *hasherCommon = handle.Common()
|
||||
if self.dict_num_matches < self.dict_num_lookups>>7 {
|
||||
return
|
||||
}
|
||||
|
||||
key = uint(hash14(data) << 1)
|
||||
for i = 0; ; (func() { i++; key++ })() {
|
||||
var tmp uint
|
||||
if shallow {
|
||||
tmp = 1
|
||||
} else {
|
||||
tmp = 2
|
||||
}
|
||||
if i >= tmp {
|
||||
break
|
||||
}
|
||||
var item uint = uint(dictionary.hash_table[key])
|
||||
self.dict_num_lookups++
|
||||
if item != 0 {
|
||||
var item_matches bool = testStaticDictionaryItem(dictionary, item, data, max_length, max_backward, max_distance, out)
|
||||
if item_matches {
|
||||
self.dict_num_matches++
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type backwardMatch struct {
|
||||
distance uint32
|
||||
length_and_code uint32
|
||||
}
|
||||
|
||||
func initBackwardMatch(self *backwardMatch, dist uint, len uint) {
|
||||
self.distance = uint32(dist)
|
||||
self.length_and_code = uint32(len << 5)
|
||||
}
|
||||
|
||||
func initDictionaryBackwardMatch(self *backwardMatch, dist uint, len uint, len_code uint) {
|
||||
self.distance = uint32(dist)
|
||||
var tmp uint
|
||||
if len == len_code {
|
||||
tmp = 0
|
||||
} else {
|
||||
tmp = len_code
|
||||
}
|
||||
self.length_and_code = uint32(len<<5 | tmp)
|
||||
}
|
||||
|
||||
func backwardMatchLength(self *backwardMatch) uint {
|
||||
return uint(self.length_and_code >> 5)
|
||||
}
|
||||
|
||||
func backwardMatchLengthCode(self *backwardMatch) uint {
|
||||
var code uint = uint(self.length_and_code) & 31
|
||||
if code != 0 {
|
||||
return code
|
||||
} else {
|
||||
return backwardMatchLength(self)
|
||||
}
|
||||
}
|
||||
|
||||
func hasherReset(handle hasherHandle) {
|
||||
if handle == nil {
|
||||
return
|
||||
}
|
||||
handle.Common().is_prepared_ = false
|
||||
}
|
||||
|
||||
func newHasher(typ int) hasherHandle {
|
||||
switch typ {
|
||||
case 2:
|
||||
return &hashLongestMatchQuickly{
|
||||
bucketBits: 16,
|
||||
bucketSweep: 1,
|
||||
hashLen: 5,
|
||||
useDictionary: true,
|
||||
}
|
||||
case 3:
|
||||
return &hashLongestMatchQuickly{
|
||||
bucketBits: 16,
|
||||
bucketSweep: 2,
|
||||
hashLen: 5,
|
||||
useDictionary: false,
|
||||
}
|
||||
case 4:
|
||||
return &hashLongestMatchQuickly{
|
||||
bucketBits: 17,
|
||||
bucketSweep: 4,
|
||||
hashLen: 5,
|
||||
useDictionary: true,
|
||||
}
|
||||
case 5:
|
||||
return new(h5)
|
||||
case 6:
|
||||
return new(h6)
|
||||
case 10:
|
||||
return new(h10)
|
||||
case 35:
|
||||
return &hashComposite{
|
||||
ha: newHasher(3),
|
||||
hb: &hashRolling{jump: 4},
|
||||
}
|
||||
case 40:
|
||||
return &hashForgetfulChain{
|
||||
bucketBits: 15,
|
||||
numBanks: 1,
|
||||
bankBits: 16,
|
||||
numLastDistancesToCheck: 4,
|
||||
}
|
||||
case 41:
|
||||
return &hashForgetfulChain{
|
||||
bucketBits: 15,
|
||||
numBanks: 1,
|
||||
bankBits: 16,
|
||||
numLastDistancesToCheck: 10,
|
||||
}
|
||||
case 42:
|
||||
return &hashForgetfulChain{
|
||||
bucketBits: 15,
|
||||
numBanks: 512,
|
||||
bankBits: 9,
|
||||
numLastDistancesToCheck: 16,
|
||||
}
|
||||
case 54:
|
||||
return &hashLongestMatchQuickly{
|
||||
bucketBits: 20,
|
||||
bucketSweep: 4,
|
||||
hashLen: 7,
|
||||
useDictionary: false,
|
||||
}
|
||||
case 55:
|
||||
return &hashComposite{
|
||||
ha: newHasher(54),
|
||||
hb: &hashRolling{jump: 4},
|
||||
}
|
||||
case 65:
|
||||
return &hashComposite{
|
||||
ha: newHasher(6),
|
||||
hb: &hashRolling{jump: 1},
|
||||
}
|
||||
}
|
||||
|
||||
panic(fmt.Sprintf("unknown hasher type: %d", typ))
|
||||
}
|
||||
|
||||
func hasherSetup(handle *hasherHandle, params *encoderParams, data []byte, position uint, input_size uint, is_last bool) {
|
||||
var self hasherHandle = nil
|
||||
var common *hasherCommon = nil
|
||||
var one_shot bool = (position == 0 && is_last)
|
||||
if *handle == nil {
|
||||
chooseHasher(params, ¶ms.hasher)
|
||||
self = newHasher(params.hasher.type_)
|
||||
|
||||
*handle = self
|
||||
common = self.Common()
|
||||
common.params = params.hasher
|
||||
self.Initialize(params)
|
||||
}
|
||||
|
||||
self = *handle
|
||||
common = self.Common()
|
||||
if !common.is_prepared_ {
|
||||
self.Prepare(one_shot, input_size, data)
|
||||
|
||||
if position == 0 {
|
||||
common.dict_num_lookups = 0
|
||||
common.dict_num_matches = 0
|
||||
}
|
||||
|
||||
common.is_prepared_ = true
|
||||
}
|
||||
}
|
||||
|
||||
func initOrStitchToPreviousBlock(handle *hasherHandle, data []byte, mask uint, params *encoderParams, position uint, input_size uint, is_last bool) {
|
||||
var self hasherHandle
|
||||
hasherSetup(handle, params, data, position, input_size, is_last)
|
||||
self = *handle
|
||||
self.StitchToPreviousBlock(input_size, position, data, mask)
|
||||
}
|
||||
+93
@@ -0,0 +1,93 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2018 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
func (h *hashComposite) HashTypeLength() uint {
|
||||
var a uint = h.ha.HashTypeLength()
|
||||
var b uint = h.hb.HashTypeLength()
|
||||
if a > b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func (h *hashComposite) StoreLookahead() uint {
|
||||
var a uint = h.ha.StoreLookahead()
|
||||
var b uint = h.hb.StoreLookahead()
|
||||
if a > b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
/* Composite hasher: This hasher allows to combine two other hashers, HASHER_A
|
||||
and HASHER_B. */
|
||||
type hashComposite struct {
|
||||
hasherCommon
|
||||
ha hasherHandle
|
||||
hb hasherHandle
|
||||
params *encoderParams
|
||||
}
|
||||
|
||||
func (h *hashComposite) Initialize(params *encoderParams) {
|
||||
h.params = params
|
||||
}
|
||||
|
||||
/* TODO: Initialize of the hashers is defered to Prepare (and params
|
||||
remembered here) because we don't get the one_shot and input_size params
|
||||
here that are needed to know the memory size of them. Instead provide
|
||||
those params to all hashers InitializehashComposite */
|
||||
func (h *hashComposite) Prepare(one_shot bool, input_size uint, data []byte) {
|
||||
if h.ha == nil {
|
||||
var common_a *hasherCommon
|
||||
var common_b *hasherCommon
|
||||
|
||||
common_a = h.ha.Common()
|
||||
common_a.params = h.params.hasher
|
||||
common_a.is_prepared_ = false
|
||||
common_a.dict_num_lookups = 0
|
||||
common_a.dict_num_matches = 0
|
||||
h.ha.Initialize(h.params)
|
||||
|
||||
common_b = h.hb.Common()
|
||||
common_b.params = h.params.hasher
|
||||
common_b.is_prepared_ = false
|
||||
common_b.dict_num_lookups = 0
|
||||
common_b.dict_num_matches = 0
|
||||
h.hb.Initialize(h.params)
|
||||
}
|
||||
|
||||
h.ha.Prepare(one_shot, input_size, data)
|
||||
h.hb.Prepare(one_shot, input_size, data)
|
||||
}
|
||||
|
||||
func (h *hashComposite) Store(data []byte, mask uint, ix uint) {
|
||||
h.ha.Store(data, mask, ix)
|
||||
h.hb.Store(data, mask, ix)
|
||||
}
|
||||
|
||||
func (h *hashComposite) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
|
||||
h.ha.StoreRange(data, mask, ix_start, ix_end)
|
||||
h.hb.StoreRange(data, mask, ix_start, ix_end)
|
||||
}
|
||||
|
||||
func (h *hashComposite) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ring_buffer_mask uint) {
|
||||
h.ha.StitchToPreviousBlock(num_bytes, position, ringbuffer, ring_buffer_mask)
|
||||
h.hb.StitchToPreviousBlock(num_bytes, position, ringbuffer, ring_buffer_mask)
|
||||
}
|
||||
|
||||
func (h *hashComposite) PrepareDistanceCache(distance_cache []int) {
|
||||
h.ha.PrepareDistanceCache(distance_cache)
|
||||
h.hb.PrepareDistanceCache(distance_cache)
|
||||
}
|
||||
|
||||
func (h *hashComposite) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
|
||||
h.ha.FindLongestMatch(dictionary, data, ring_buffer_mask, distance_cache, cur_ix, max_length, max_backward, gap, max_distance, out)
|
||||
h.hb.FindLongestMatch(dictionary, data, ring_buffer_mask, distance_cache, cur_ix, max_length, max_backward, gap, max_distance, out)
|
||||
}
|
||||
+252
@@ -0,0 +1,252 @@
|
||||
package brotli
|
||||
|
||||
import "encoding/binary"
|
||||
|
||||
/* Copyright 2016 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
func (*hashForgetfulChain) HashTypeLength() uint {
|
||||
return 4
|
||||
}
|
||||
|
||||
func (*hashForgetfulChain) StoreLookahead() uint {
|
||||
return 4
|
||||
}
|
||||
|
||||
/* HashBytes is the function that chooses the bucket to place the address in.*/
|
||||
func (h *hashForgetfulChain) HashBytes(data []byte) uint {
|
||||
var hash uint32 = binary.LittleEndian.Uint32(data) * kHashMul32
|
||||
|
||||
/* The higher bits contain more mixture from the multiplication,
|
||||
so we take our results from there. */
|
||||
return uint(hash >> (32 - h.bucketBits))
|
||||
}
|
||||
|
||||
type slot struct {
|
||||
delta uint16
|
||||
next uint16
|
||||
}
|
||||
|
||||
/* A (forgetful) hash table to the data seen by the compressor, to
|
||||
help create backward references to previous data.
|
||||
|
||||
Hashes are stored in chains which are bucketed to groups. Group of chains
|
||||
share a storage "bank". When more than "bank size" chain nodes are added,
|
||||
oldest nodes are replaced; this way several chains may share a tail. */
|
||||
type hashForgetfulChain struct {
|
||||
hasherCommon
|
||||
|
||||
bucketBits uint
|
||||
numBanks uint
|
||||
bankBits uint
|
||||
numLastDistancesToCheck int
|
||||
|
||||
addr []uint32
|
||||
head []uint16
|
||||
tiny_hash [65536]byte
|
||||
banks [][]slot
|
||||
free_slot_idx []uint16
|
||||
max_hops uint
|
||||
}
|
||||
|
||||
func (h *hashForgetfulChain) Initialize(params *encoderParams) {
|
||||
var q uint
|
||||
if params.quality > 6 {
|
||||
q = 7
|
||||
} else {
|
||||
q = 8
|
||||
}
|
||||
h.max_hops = q << uint(params.quality-4)
|
||||
|
||||
bankSize := 1 << h.bankBits
|
||||
bucketSize := 1 << h.bucketBits
|
||||
|
||||
h.addr = make([]uint32, bucketSize)
|
||||
h.head = make([]uint16, bucketSize)
|
||||
h.banks = make([][]slot, h.numBanks)
|
||||
for i := range h.banks {
|
||||
h.banks[i] = make([]slot, bankSize)
|
||||
}
|
||||
h.free_slot_idx = make([]uint16, h.numBanks)
|
||||
}
|
||||
|
||||
func (h *hashForgetfulChain) Prepare(one_shot bool, input_size uint, data []byte) {
|
||||
var partial_prepare_threshold uint = (1 << h.bucketBits) >> 6
|
||||
/* Partial preparation is 100 times slower (per socket). */
|
||||
if one_shot && input_size <= partial_prepare_threshold {
|
||||
var i uint
|
||||
for i = 0; i < input_size; i++ {
|
||||
var bucket uint = h.HashBytes(data[i:])
|
||||
|
||||
/* See InitEmpty comment. */
|
||||
h.addr[bucket] = 0xCCCCCCCC
|
||||
|
||||
h.head[bucket] = 0xCCCC
|
||||
}
|
||||
} else {
|
||||
/* Fill |addr| array with 0xCCCCCCCC value. Because of wrapping, position
|
||||
processed by hasher never reaches 3GB + 64M; this makes all new chains
|
||||
to be terminated after the first node. */
|
||||
for i := range h.addr {
|
||||
h.addr[i] = 0xCCCCCCCC
|
||||
}
|
||||
|
||||
for i := range h.head {
|
||||
h.head[i] = 0
|
||||
}
|
||||
}
|
||||
|
||||
h.tiny_hash = [65536]byte{}
|
||||
for i := range h.free_slot_idx {
|
||||
h.free_slot_idx[i] = 0
|
||||
}
|
||||
}
|
||||
|
||||
/* Look at 4 bytes at &data[ix & mask]. Compute a hash from these, and prepend
|
||||
node to corresponding chain; also update tiny_hash for current position. */
|
||||
func (h *hashForgetfulChain) Store(data []byte, mask uint, ix uint) {
|
||||
var key uint = h.HashBytes(data[ix&mask:])
|
||||
var bank uint = key & (h.numBanks - 1)
|
||||
idx := uint(h.free_slot_idx[bank]) & ((1 << h.bankBits) - 1)
|
||||
h.free_slot_idx[bank]++
|
||||
var delta uint = ix - uint(h.addr[key])
|
||||
h.tiny_hash[uint16(ix)] = byte(key)
|
||||
if delta > 0xFFFF {
|
||||
delta = 0xFFFF
|
||||
}
|
||||
h.banks[bank][idx].delta = uint16(delta)
|
||||
h.banks[bank][idx].next = h.head[key]
|
||||
h.addr[key] = uint32(ix)
|
||||
h.head[key] = uint16(idx)
|
||||
}
|
||||
|
||||
func (h *hashForgetfulChain) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
|
||||
var i uint
|
||||
for i = ix_start; i < ix_end; i++ {
|
||||
h.Store(data, mask, i)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *hashForgetfulChain) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ring_buffer_mask uint) {
|
||||
if num_bytes >= h.HashTypeLength()-1 && position >= 3 {
|
||||
/* Prepare the hashes for three last bytes of the last write.
|
||||
These could not be calculated before, since they require knowledge
|
||||
of both the previous and the current block. */
|
||||
h.Store(ringbuffer, ring_buffer_mask, position-3)
|
||||
h.Store(ringbuffer, ring_buffer_mask, position-2)
|
||||
h.Store(ringbuffer, ring_buffer_mask, position-1)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *hashForgetfulChain) PrepareDistanceCache(distance_cache []int) {
|
||||
prepareDistanceCache(distance_cache, h.numLastDistancesToCheck)
|
||||
}
|
||||
|
||||
/* Find a longest backward match of &data[cur_ix] up to the length of
|
||||
max_length and stores the position cur_ix in the hash table.
|
||||
|
||||
REQUIRES: PrepareDistanceCachehashForgetfulChain must be invoked for current distance cache
|
||||
values; if this method is invoked repeatedly with the same distance
|
||||
cache values, it is enough to invoke PrepareDistanceCachehashForgetfulChain once.
|
||||
|
||||
Does not look for matches longer than max_length.
|
||||
Does not look for matches further away than max_backward.
|
||||
Writes the best match into |out|.
|
||||
|out|->score is updated only if a better match is found. */
|
||||
func (h *hashForgetfulChain) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
|
||||
var cur_ix_masked uint = cur_ix & ring_buffer_mask
|
||||
var min_score uint = out.score
|
||||
var best_score uint = out.score
|
||||
var best_len uint = out.len
|
||||
var key uint = h.HashBytes(data[cur_ix_masked:])
|
||||
var tiny_hash byte = byte(key)
|
||||
/* Don't accept a short copy from far away. */
|
||||
out.len = 0
|
||||
|
||||
out.len_code_delta = 0
|
||||
|
||||
/* Try last distance first. */
|
||||
for i := 0; i < h.numLastDistancesToCheck; i++ {
|
||||
var backward uint = uint(distance_cache[i])
|
||||
var prev_ix uint = (cur_ix - backward)
|
||||
|
||||
/* For distance code 0 we want to consider 2-byte matches. */
|
||||
if i > 0 && h.tiny_hash[uint16(prev_ix)] != tiny_hash {
|
||||
continue
|
||||
}
|
||||
if prev_ix >= cur_ix || backward > max_backward {
|
||||
continue
|
||||
}
|
||||
|
||||
prev_ix &= ring_buffer_mask
|
||||
{
|
||||
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
|
||||
if len >= 2 {
|
||||
var score uint = backwardReferenceScoreUsingLastDistance(uint(len))
|
||||
if best_score < score {
|
||||
if i != 0 {
|
||||
score -= backwardReferencePenaltyUsingLastDistance(uint(i))
|
||||
}
|
||||
if best_score < score {
|
||||
best_score = score
|
||||
best_len = uint(len)
|
||||
out.len = best_len
|
||||
out.distance = backward
|
||||
out.score = best_score
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
{
|
||||
var bank uint = key & (h.numBanks - 1)
|
||||
var backward uint = 0
|
||||
var hops uint = h.max_hops
|
||||
var delta uint = cur_ix - uint(h.addr[key])
|
||||
var slot uint = uint(h.head[key])
|
||||
for {
|
||||
tmp6 := hops
|
||||
hops--
|
||||
if tmp6 == 0 {
|
||||
break
|
||||
}
|
||||
var prev_ix uint
|
||||
var last uint = slot
|
||||
backward += delta
|
||||
if backward > max_backward {
|
||||
break
|
||||
}
|
||||
prev_ix = (cur_ix - backward) & ring_buffer_mask
|
||||
slot = uint(h.banks[bank][last].next)
|
||||
delta = uint(h.banks[bank][last].delta)
|
||||
if cur_ix_masked+best_len > ring_buffer_mask || prev_ix+best_len > ring_buffer_mask || data[cur_ix_masked+best_len] != data[prev_ix+best_len] {
|
||||
continue
|
||||
}
|
||||
{
|
||||
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
|
||||
if len >= 4 {
|
||||
/* Comparing for >= 3 does not change the semantics, but just saves
|
||||
for a few unnecessary binary logarithms in backward reference
|
||||
score, since we are not interested in such short matches. */
|
||||
var score uint = backwardReferenceScore(uint(len), backward)
|
||||
if best_score < score {
|
||||
best_score = score
|
||||
best_len = uint(len)
|
||||
out.len = best_len
|
||||
out.distance = backward
|
||||
out.score = best_score
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
h.Store(data, ring_buffer_mask, cur_ix)
|
||||
}
|
||||
|
||||
if out.score == min_score {
|
||||
searchInStaticDictionary(dictionary, h, data[cur_ix_masked:], max_length, max_backward+gap, max_distance, out, false)
|
||||
}
|
||||
}
|
||||
+214
@@ -0,0 +1,214 @@
|
||||
package brotli
|
||||
|
||||
import "encoding/binary"
|
||||
|
||||
/* Copyright 2010 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* For BUCKET_SWEEP == 1, enabling the dictionary lookup makes compression
|
||||
a little faster (0.5% - 1%) and it compresses 0.15% better on small text
|
||||
and HTML inputs. */
|
||||
|
||||
func (*hashLongestMatchQuickly) HashTypeLength() uint {
|
||||
return 8
|
||||
}
|
||||
|
||||
func (*hashLongestMatchQuickly) StoreLookahead() uint {
|
||||
return 8
|
||||
}
|
||||
|
||||
/* HashBytes is the function that chooses the bucket to place
|
||||
the address in. The HashLongestMatch and hashLongestMatchQuickly
|
||||
classes have separate, different implementations of hashing. */
|
||||
func (h *hashLongestMatchQuickly) HashBytes(data []byte) uint32 {
|
||||
var hash uint64 = ((binary.LittleEndian.Uint64(data) << (64 - 8*h.hashLen)) * kHashMul64)
|
||||
|
||||
/* The higher bits contain more mixture from the multiplication,
|
||||
so we take our results from there. */
|
||||
return uint32(hash >> (64 - h.bucketBits))
|
||||
}
|
||||
|
||||
/* A (forgetful) hash table to the data seen by the compressor, to
|
||||
help create backward references to previous data.
|
||||
|
||||
This is a hash map of fixed size (1 << 16). Starting from the
|
||||
given index, 1 buckets are used to store values of a key. */
|
||||
type hashLongestMatchQuickly struct {
|
||||
hasherCommon
|
||||
|
||||
bucketBits uint
|
||||
bucketSweep int
|
||||
hashLen uint
|
||||
useDictionary bool
|
||||
|
||||
buckets []uint32
|
||||
}
|
||||
|
||||
func (h *hashLongestMatchQuickly) Initialize(params *encoderParams) {
|
||||
h.buckets = make([]uint32, 1<<h.bucketBits+h.bucketSweep)
|
||||
}
|
||||
|
||||
func (h *hashLongestMatchQuickly) Prepare(one_shot bool, input_size uint, data []byte) {
|
||||
var partial_prepare_threshold uint = (4 << h.bucketBits) >> 7
|
||||
/* Partial preparation is 100 times slower (per socket). */
|
||||
if one_shot && input_size <= partial_prepare_threshold {
|
||||
var i uint
|
||||
for i = 0; i < input_size; i++ {
|
||||
var key uint32 = h.HashBytes(data[i:])
|
||||
for j := 0; j < h.bucketSweep; j++ {
|
||||
h.buckets[key+uint32(j)] = 0
|
||||
}
|
||||
}
|
||||
} else {
|
||||
/* It is not strictly necessary to fill this buffer here, but
|
||||
not filling will make the results of the compression stochastic
|
||||
(but correct). This is because random data would cause the
|
||||
system to find accidentally good backward references here and there. */
|
||||
for i := range h.buckets {
|
||||
h.buckets[i] = 0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Look at 5 bytes at &data[ix & mask].
|
||||
Compute a hash from these, and store the value somewhere within
|
||||
[ix .. ix+3]. */
|
||||
func (h *hashLongestMatchQuickly) Store(data []byte, mask uint, ix uint) {
|
||||
var key uint32 = h.HashBytes(data[ix&mask:])
|
||||
var off uint32 = uint32(ix>>3) % uint32(h.bucketSweep)
|
||||
/* Wiggle the value with the bucket sweep range. */
|
||||
h.buckets[key+off] = uint32(ix)
|
||||
}
|
||||
|
||||
func (h *hashLongestMatchQuickly) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
|
||||
var i uint
|
||||
for i = ix_start; i < ix_end; i++ {
|
||||
h.Store(data, mask, i)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *hashLongestMatchQuickly) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ringbuffer_mask uint) {
|
||||
if num_bytes >= h.HashTypeLength()-1 && position >= 3 {
|
||||
/* Prepare the hashes for three last bytes of the last write.
|
||||
These could not be calculated before, since they require knowledge
|
||||
of both the previous and the current block. */
|
||||
h.Store(ringbuffer, ringbuffer_mask, position-3)
|
||||
h.Store(ringbuffer, ringbuffer_mask, position-2)
|
||||
h.Store(ringbuffer, ringbuffer_mask, position-1)
|
||||
}
|
||||
}
|
||||
|
||||
func (*hashLongestMatchQuickly) PrepareDistanceCache(distance_cache []int) {
|
||||
}
|
||||
|
||||
/* Find a longest backward match of &data[cur_ix & ring_buffer_mask]
|
||||
up to the length of max_length and stores the position cur_ix in the
|
||||
hash table.
|
||||
|
||||
Does not look for matches longer than max_length.
|
||||
Does not look for matches further away than max_backward.
|
||||
Writes the best match into |out|.
|
||||
|out|->score is updated only if a better match is found. */
|
||||
func (h *hashLongestMatchQuickly) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
|
||||
var best_len_in uint = out.len
|
||||
var cur_ix_masked uint = cur_ix & ring_buffer_mask
|
||||
var key uint32 = h.HashBytes(data[cur_ix_masked:])
|
||||
var compare_char int = int(data[cur_ix_masked+best_len_in])
|
||||
var min_score uint = out.score
|
||||
var best_score uint = out.score
|
||||
var best_len uint = best_len_in
|
||||
var cached_backward uint = uint(distance_cache[0])
|
||||
var prev_ix uint = cur_ix - cached_backward
|
||||
var bucket []uint32
|
||||
out.len_code_delta = 0
|
||||
if prev_ix < cur_ix {
|
||||
prev_ix &= uint(uint32(ring_buffer_mask))
|
||||
if compare_char == int(data[prev_ix+best_len]) {
|
||||
var len uint = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
|
||||
if len >= 4 {
|
||||
var score uint = backwardReferenceScoreUsingLastDistance(uint(len))
|
||||
if best_score < score {
|
||||
best_score = score
|
||||
best_len = uint(len)
|
||||
out.len = uint(len)
|
||||
out.distance = cached_backward
|
||||
out.score = best_score
|
||||
compare_char = int(data[cur_ix_masked+best_len])
|
||||
if h.bucketSweep == 1 {
|
||||
h.buckets[key] = uint32(cur_ix)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if h.bucketSweep == 1 {
|
||||
var backward uint
|
||||
var len uint
|
||||
|
||||
/* Only one to look for, don't bother to prepare for a loop. */
|
||||
prev_ix = uint(h.buckets[key])
|
||||
|
||||
h.buckets[key] = uint32(cur_ix)
|
||||
backward = cur_ix - prev_ix
|
||||
prev_ix &= uint(uint32(ring_buffer_mask))
|
||||
if compare_char != int(data[prev_ix+best_len_in]) {
|
||||
return
|
||||
}
|
||||
|
||||
if backward == 0 || backward > max_backward {
|
||||
return
|
||||
}
|
||||
|
||||
len = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
|
||||
if len >= 4 {
|
||||
var score uint = backwardReferenceScore(uint(len), backward)
|
||||
if best_score < score {
|
||||
out.len = uint(len)
|
||||
out.distance = backward
|
||||
out.score = score
|
||||
return
|
||||
}
|
||||
}
|
||||
} else {
|
||||
bucket = h.buckets[key:]
|
||||
var i int
|
||||
prev_ix = uint(bucket[0])
|
||||
bucket = bucket[1:]
|
||||
for i = 0; i < h.bucketSweep; (func() { i++; tmp3 := bucket; bucket = bucket[1:]; prev_ix = uint(tmp3[0]) })() {
|
||||
var backward uint = cur_ix - prev_ix
|
||||
var len uint
|
||||
prev_ix &= uint(uint32(ring_buffer_mask))
|
||||
if compare_char != int(data[prev_ix+best_len]) {
|
||||
continue
|
||||
}
|
||||
|
||||
if backward == 0 || backward > max_backward {
|
||||
continue
|
||||
}
|
||||
|
||||
len = findMatchLengthWithLimit(data[prev_ix:], data[cur_ix_masked:], max_length)
|
||||
if len >= 4 {
|
||||
var score uint = backwardReferenceScore(uint(len), backward)
|
||||
if best_score < score {
|
||||
best_score = score
|
||||
best_len = uint(len)
|
||||
out.len = best_len
|
||||
out.distance = backward
|
||||
out.score = score
|
||||
compare_char = int(data[cur_ix_masked+best_len])
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if h.useDictionary && min_score == out.score {
|
||||
searchInStaticDictionary(dictionary, h, data[cur_ix_masked:], max_length, max_backward+gap, max_distance, out, true)
|
||||
}
|
||||
|
||||
h.buckets[key+uint32((cur_ix>>3)%uint(h.bucketSweep))] = uint32(cur_ix)
|
||||
}
|
||||
+168
@@ -0,0 +1,168 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2018 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* NOTE: this hasher does not search in the dictionary. It is used as
|
||||
backup-hasher, the main hasher already searches in it. */
|
||||
|
||||
const kRollingHashMul32 uint32 = 69069
|
||||
|
||||
const kInvalidPosHashRolling uint32 = 0xffffffff
|
||||
|
||||
/* This hasher uses a longer forward length, but returning a higher value here
|
||||
will hurt compression by the main hasher when combined with a composite
|
||||
hasher. The hasher tests for forward itself instead. */
|
||||
func (*hashRolling) HashTypeLength() uint {
|
||||
return 4
|
||||
}
|
||||
|
||||
func (*hashRolling) StoreLookahead() uint {
|
||||
return 4
|
||||
}
|
||||
|
||||
/* Computes a code from a single byte. A lookup table of 256 values could be
|
||||
used, but simply adding 1 works about as good. */
|
||||
func (*hashRolling) HashByte(b byte) uint32 {
|
||||
return uint32(b) + 1
|
||||
}
|
||||
|
||||
func (h *hashRolling) HashRollingFunctionInitial(state uint32, add byte, factor uint32) uint32 {
|
||||
return uint32(factor*state + h.HashByte(add))
|
||||
}
|
||||
|
||||
func (h *hashRolling) HashRollingFunction(state uint32, add byte, rem byte, factor uint32, factor_remove uint32) uint32 {
|
||||
return uint32(factor*state + h.HashByte(add) - factor_remove*h.HashByte(rem))
|
||||
}
|
||||
|
||||
/* Rolling hash for long distance long string matches. Stores one position
|
||||
per bucket, bucket key is computed over a long region. */
|
||||
type hashRolling struct {
|
||||
hasherCommon
|
||||
|
||||
jump int
|
||||
|
||||
state uint32
|
||||
table []uint32
|
||||
next_ix uint
|
||||
factor uint32
|
||||
factor_remove uint32
|
||||
}
|
||||
|
||||
func (h *hashRolling) Initialize(params *encoderParams) {
|
||||
h.state = 0
|
||||
h.next_ix = 0
|
||||
|
||||
h.factor = kRollingHashMul32
|
||||
|
||||
/* Compute the factor of the oldest byte to remove: factor**steps modulo
|
||||
0xffffffff (the multiplications rely on 32-bit overflow) */
|
||||
h.factor_remove = 1
|
||||
|
||||
for i := 0; i < 32; i += h.jump {
|
||||
h.factor_remove *= h.factor
|
||||
}
|
||||
|
||||
h.table = make([]uint32, 16777216)
|
||||
for i := 0; i < 16777216; i++ {
|
||||
h.table[i] = kInvalidPosHashRolling
|
||||
}
|
||||
}
|
||||
|
||||
func (h *hashRolling) Prepare(one_shot bool, input_size uint, data []byte) {
|
||||
/* Too small size, cannot use this hasher. */
|
||||
if input_size < 32 {
|
||||
return
|
||||
}
|
||||
h.state = 0
|
||||
for i := 0; i < 32; i += h.jump {
|
||||
h.state = h.HashRollingFunctionInitial(h.state, data[i], h.factor)
|
||||
}
|
||||
}
|
||||
|
||||
func (*hashRolling) Store(data []byte, mask uint, ix uint) {
|
||||
}
|
||||
|
||||
func (*hashRolling) StoreRange(data []byte, mask uint, ix_start uint, ix_end uint) {
|
||||
}
|
||||
|
||||
func (h *hashRolling) StitchToPreviousBlock(num_bytes uint, position uint, ringbuffer []byte, ring_buffer_mask uint) {
|
||||
var position_masked uint
|
||||
/* In this case we must re-initialize the hasher from scratch from the
|
||||
current position. */
|
||||
|
||||
var available uint = num_bytes
|
||||
if position&uint(h.jump-1) != 0 {
|
||||
var diff uint = uint(h.jump) - (position & uint(h.jump-1))
|
||||
if diff > available {
|
||||
available = 0
|
||||
} else {
|
||||
available = available - diff
|
||||
}
|
||||
position += diff
|
||||
}
|
||||
|
||||
position_masked = position & ring_buffer_mask
|
||||
|
||||
/* wrapping around ringbuffer not handled. */
|
||||
if available > ring_buffer_mask-position_masked {
|
||||
available = ring_buffer_mask - position_masked
|
||||
}
|
||||
|
||||
h.Prepare(false, available, ringbuffer[position&ring_buffer_mask:])
|
||||
h.next_ix = position
|
||||
}
|
||||
|
||||
func (*hashRolling) PrepareDistanceCache(distance_cache []int) {
|
||||
}
|
||||
|
||||
func (h *hashRolling) FindLongestMatch(dictionary *encoderDictionary, data []byte, ring_buffer_mask uint, distance_cache []int, cur_ix uint, max_length uint, max_backward uint, gap uint, max_distance uint, out *hasherSearchResult) {
|
||||
var cur_ix_masked uint = cur_ix & ring_buffer_mask
|
||||
var pos uint = h.next_ix
|
||||
|
||||
if cur_ix&uint(h.jump-1) != 0 {
|
||||
return
|
||||
}
|
||||
|
||||
/* Not enough lookahead */
|
||||
if max_length < 32 {
|
||||
return
|
||||
}
|
||||
|
||||
for pos = h.next_ix; pos <= cur_ix; pos += uint(h.jump) {
|
||||
var code uint32 = h.state & ((16777216 * 64) - 1)
|
||||
var rem byte = data[pos&ring_buffer_mask]
|
||||
var add byte = data[(pos+32)&ring_buffer_mask]
|
||||
var found_ix uint = uint(kInvalidPosHashRolling)
|
||||
|
||||
h.state = h.HashRollingFunction(h.state, add, rem, h.factor, h.factor_remove)
|
||||
|
||||
if code < 16777216 {
|
||||
found_ix = uint(h.table[code])
|
||||
h.table[code] = uint32(pos)
|
||||
if pos == cur_ix && uint32(found_ix) != kInvalidPosHashRolling {
|
||||
/* The cast to 32-bit makes backward distances up to 4GB work even
|
||||
if cur_ix is above 4GB, despite using 32-bit values in the table. */
|
||||
var backward uint = uint(uint32(cur_ix - found_ix))
|
||||
if backward <= max_backward {
|
||||
var found_ix_masked uint = found_ix & ring_buffer_mask
|
||||
var len uint = findMatchLengthWithLimit(data[found_ix_masked:], data[cur_ix_masked:], max_length)
|
||||
if len >= 4 && len > out.len {
|
||||
var score uint = backwardReferenceScore(uint(len), backward)
|
||||
if score > out.score {
|
||||
out.len = uint(len)
|
||||
out.distance = backward
|
||||
out.score = score
|
||||
out.len_code_delta = 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
h.next_ix = cur_ix + uint(h.jump)
|
||||
}
|
||||
+226
@@ -0,0 +1,226 @@
|
||||
package brotli
|
||||
|
||||
import "math"
|
||||
|
||||
/* The distance symbols effectively used by "Large Window Brotli" (32-bit). */
|
||||
const numHistogramDistanceSymbols = 544
|
||||
|
||||
type histogramLiteral struct {
|
||||
data_ [numLiteralSymbols]uint32
|
||||
total_count_ uint
|
||||
bit_cost_ float64
|
||||
}
|
||||
|
||||
func histogramClearLiteral(self *histogramLiteral) {
|
||||
self.data_ = [numLiteralSymbols]uint32{}
|
||||
self.total_count_ = 0
|
||||
self.bit_cost_ = math.MaxFloat64
|
||||
}
|
||||
|
||||
func clearHistogramsLiteral(array []histogramLiteral, length uint) {
|
||||
var i uint
|
||||
for i = 0; i < length; i++ {
|
||||
histogramClearLiteral(&array[i:][0])
|
||||
}
|
||||
}
|
||||
|
||||
func histogramAddLiteral(self *histogramLiteral, val uint) {
|
||||
self.data_[val]++
|
||||
self.total_count_++
|
||||
}
|
||||
|
||||
func histogramAddVectorLiteral(self *histogramLiteral, p []byte, n uint) {
|
||||
self.total_count_ += n
|
||||
n += 1
|
||||
for {
|
||||
n--
|
||||
if n == 0 {
|
||||
break
|
||||
}
|
||||
self.data_[p[0]]++
|
||||
p = p[1:]
|
||||
}
|
||||
}
|
||||
|
||||
func histogramAddHistogramLiteral(self *histogramLiteral, v *histogramLiteral) {
|
||||
var i uint
|
||||
self.total_count_ += v.total_count_
|
||||
for i = 0; i < numLiteralSymbols; i++ {
|
||||
self.data_[i] += v.data_[i]
|
||||
}
|
||||
}
|
||||
|
||||
func histogramDataSizeLiteral() uint {
|
||||
return numLiteralSymbols
|
||||
}
|
||||
|
||||
type histogramCommand struct {
|
||||
data_ [numCommandSymbols]uint32
|
||||
total_count_ uint
|
||||
bit_cost_ float64
|
||||
}
|
||||
|
||||
func histogramClearCommand(self *histogramCommand) {
|
||||
self.data_ = [numCommandSymbols]uint32{}
|
||||
self.total_count_ = 0
|
||||
self.bit_cost_ = math.MaxFloat64
|
||||
}
|
||||
|
||||
func clearHistogramsCommand(array []histogramCommand, length uint) {
|
||||
var i uint
|
||||
for i = 0; i < length; i++ {
|
||||
histogramClearCommand(&array[i:][0])
|
||||
}
|
||||
}
|
||||
|
||||
func histogramAddCommand(self *histogramCommand, val uint) {
|
||||
self.data_[val]++
|
||||
self.total_count_++
|
||||
}
|
||||
|
||||
func histogramAddVectorCommand(self *histogramCommand, p []uint16, n uint) {
|
||||
self.total_count_ += n
|
||||
n += 1
|
||||
for {
|
||||
n--
|
||||
if n == 0 {
|
||||
break
|
||||
}
|
||||
self.data_[p[0]]++
|
||||
p = p[1:]
|
||||
}
|
||||
}
|
||||
|
||||
func histogramAddHistogramCommand(self *histogramCommand, v *histogramCommand) {
|
||||
var i uint
|
||||
self.total_count_ += v.total_count_
|
||||
for i = 0; i < numCommandSymbols; i++ {
|
||||
self.data_[i] += v.data_[i]
|
||||
}
|
||||
}
|
||||
|
||||
func histogramDataSizeCommand() uint {
|
||||
return numCommandSymbols
|
||||
}
|
||||
|
||||
type histogramDistance struct {
|
||||
data_ [numDistanceSymbols]uint32
|
||||
total_count_ uint
|
||||
bit_cost_ float64
|
||||
}
|
||||
|
||||
func histogramClearDistance(self *histogramDistance) {
|
||||
self.data_ = [numDistanceSymbols]uint32{}
|
||||
self.total_count_ = 0
|
||||
self.bit_cost_ = math.MaxFloat64
|
||||
}
|
||||
|
||||
func clearHistogramsDistance(array []histogramDistance, length uint) {
|
||||
var i uint
|
||||
for i = 0; i < length; i++ {
|
||||
histogramClearDistance(&array[i:][0])
|
||||
}
|
||||
}
|
||||
|
||||
func histogramAddDistance(self *histogramDistance, val uint) {
|
||||
self.data_[val]++
|
||||
self.total_count_++
|
||||
}
|
||||
|
||||
func histogramAddVectorDistance(self *histogramDistance, p []uint16, n uint) {
|
||||
self.total_count_ += n
|
||||
n += 1
|
||||
for {
|
||||
n--
|
||||
if n == 0 {
|
||||
break
|
||||
}
|
||||
self.data_[p[0]]++
|
||||
p = p[1:]
|
||||
}
|
||||
}
|
||||
|
||||
func histogramAddHistogramDistance(self *histogramDistance, v *histogramDistance) {
|
||||
var i uint
|
||||
self.total_count_ += v.total_count_
|
||||
for i = 0; i < numDistanceSymbols; i++ {
|
||||
self.data_[i] += v.data_[i]
|
||||
}
|
||||
}
|
||||
|
||||
func histogramDataSizeDistance() uint {
|
||||
return numDistanceSymbols
|
||||
}
|
||||
|
||||
type blockSplitIterator struct {
|
||||
split_ *blockSplit
|
||||
idx_ uint
|
||||
type_ uint
|
||||
length_ uint
|
||||
}
|
||||
|
||||
func initBlockSplitIterator(self *blockSplitIterator, split *blockSplit) {
|
||||
self.split_ = split
|
||||
self.idx_ = 0
|
||||
self.type_ = 0
|
||||
if len(split.lengths) > 0 {
|
||||
self.length_ = uint(split.lengths[0])
|
||||
} else {
|
||||
self.length_ = 0
|
||||
}
|
||||
}
|
||||
|
||||
func blockSplitIteratorNext(self *blockSplitIterator) {
|
||||
if self.length_ == 0 {
|
||||
self.idx_++
|
||||
self.type_ = uint(self.split_.types[self.idx_])
|
||||
self.length_ = uint(self.split_.lengths[self.idx_])
|
||||
}
|
||||
|
||||
self.length_--
|
||||
}
|
||||
|
||||
func buildHistogramsWithContext(cmds []command, literal_split *blockSplit, insert_and_copy_split *blockSplit, dist_split *blockSplit, ringbuffer []byte, start_pos uint, mask uint, prev_byte byte, prev_byte2 byte, context_modes []int, literal_histograms []histogramLiteral, insert_and_copy_histograms []histogramCommand, copy_dist_histograms []histogramDistance) {
|
||||
var pos uint = start_pos
|
||||
var literal_it blockSplitIterator
|
||||
var insert_and_copy_it blockSplitIterator
|
||||
var dist_it blockSplitIterator
|
||||
|
||||
initBlockSplitIterator(&literal_it, literal_split)
|
||||
initBlockSplitIterator(&insert_and_copy_it, insert_and_copy_split)
|
||||
initBlockSplitIterator(&dist_it, dist_split)
|
||||
for i := range cmds {
|
||||
var cmd *command = &cmds[i]
|
||||
var j uint
|
||||
blockSplitIteratorNext(&insert_and_copy_it)
|
||||
histogramAddCommand(&insert_and_copy_histograms[insert_and_copy_it.type_], uint(cmd.cmd_prefix_))
|
||||
|
||||
/* TODO: unwrap iterator blocks. */
|
||||
for j = uint(cmd.insert_len_); j != 0; j-- {
|
||||
var context uint
|
||||
blockSplitIteratorNext(&literal_it)
|
||||
context = literal_it.type_
|
||||
if context_modes != nil {
|
||||
var lut contextLUT = getContextLUT(context_modes[context])
|
||||
context = (context << literalContextBits) + uint(getContext(prev_byte, prev_byte2, lut))
|
||||
}
|
||||
|
||||
histogramAddLiteral(&literal_histograms[context], uint(ringbuffer[pos&mask]))
|
||||
prev_byte2 = prev_byte
|
||||
prev_byte = ringbuffer[pos&mask]
|
||||
pos++
|
||||
}
|
||||
|
||||
pos += uint(commandCopyLen(cmd))
|
||||
if commandCopyLen(cmd) != 0 {
|
||||
prev_byte2 = ringbuffer[(pos-2)&mask]
|
||||
prev_byte = ringbuffer[(pos-1)&mask]
|
||||
if cmd.cmd_prefix_ >= 128 {
|
||||
var context uint
|
||||
blockSplitIteratorNext(&dist_it)
|
||||
context = uint(uint32(dist_it.type_<<distanceContextBits) + commandDistanceContext(cmd))
|
||||
histogramAddDistance(©_dist_histograms[context], uint(cmd.dist_prefix_)&0x3FF)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+184
@@ -0,0 +1,184 @@
|
||||
package brotli
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// HTTPCompressor chooses a compression method (brotli, gzip, or none) based on
|
||||
// the Accept-Encoding header, sets the Content-Encoding header, and returns a
|
||||
// WriteCloser that implements that compression. The Close method must be called
|
||||
// before the current HTTP handler returns.
|
||||
func HTTPCompressor(w http.ResponseWriter, r *http.Request) io.WriteCloser {
|
||||
if w.Header().Get("Vary") == "" {
|
||||
w.Header().Set("Vary", "Accept-Encoding")
|
||||
}
|
||||
|
||||
encoding := negotiateContentEncoding(r, []string{"br", "gzip"})
|
||||
switch encoding {
|
||||
case "br":
|
||||
w.Header().Set("Content-Encoding", "br")
|
||||
return NewWriterV2(w, DefaultCompression)
|
||||
case "gzip":
|
||||
w.Header().Set("Content-Encoding", "gzip")
|
||||
return gzip.NewWriter(w)
|
||||
}
|
||||
return nopCloser{w}
|
||||
}
|
||||
|
||||
// negotiateContentEncoding returns the best offered content encoding for the
|
||||
// request's Accept-Encoding header. If two offers match with equal weight and
|
||||
// then the offer earlier in the list is preferred. If no offers are
|
||||
// acceptable, then "" is returned.
|
||||
func negotiateContentEncoding(r *http.Request, offers []string) string {
|
||||
bestOffer := "identity"
|
||||
bestQ := -1.0
|
||||
specs := parseAccept(r.Header, "Accept-Encoding")
|
||||
for _, offer := range offers {
|
||||
for _, spec := range specs {
|
||||
if spec.Q > bestQ &&
|
||||
(spec.Value == "*" || spec.Value == offer) {
|
||||
bestQ = spec.Q
|
||||
bestOffer = offer
|
||||
}
|
||||
}
|
||||
}
|
||||
if bestQ == 0 {
|
||||
bestOffer = ""
|
||||
}
|
||||
return bestOffer
|
||||
}
|
||||
|
||||
// acceptSpec describes an Accept* header.
|
||||
type acceptSpec struct {
|
||||
Value string
|
||||
Q float64
|
||||
}
|
||||
|
||||
// parseAccept parses Accept* headers.
|
||||
func parseAccept(header http.Header, key string) (specs []acceptSpec) {
|
||||
loop:
|
||||
for _, s := range header[key] {
|
||||
for {
|
||||
var spec acceptSpec
|
||||
spec.Value, s = expectTokenSlash(s)
|
||||
if spec.Value == "" {
|
||||
continue loop
|
||||
}
|
||||
spec.Q = 1.0
|
||||
s = skipSpace(s)
|
||||
if strings.HasPrefix(s, ";") {
|
||||
s = skipSpace(s[1:])
|
||||
if !strings.HasPrefix(s, "q=") {
|
||||
continue loop
|
||||
}
|
||||
spec.Q, s = expectQuality(s[2:])
|
||||
if spec.Q < 0.0 {
|
||||
continue loop
|
||||
}
|
||||
}
|
||||
specs = append(specs, spec)
|
||||
s = skipSpace(s)
|
||||
if !strings.HasPrefix(s, ",") {
|
||||
continue loop
|
||||
}
|
||||
s = skipSpace(s[1:])
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func skipSpace(s string) (rest string) {
|
||||
i := 0
|
||||
for ; i < len(s); i++ {
|
||||
if octetTypes[s[i]]&isSpace == 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
return s[i:]
|
||||
}
|
||||
|
||||
func expectTokenSlash(s string) (token, rest string) {
|
||||
i := 0
|
||||
for ; i < len(s); i++ {
|
||||
b := s[i]
|
||||
if (octetTypes[b]&isToken == 0) && b != '/' {
|
||||
break
|
||||
}
|
||||
}
|
||||
return s[:i], s[i:]
|
||||
}
|
||||
|
||||
func expectQuality(s string) (q float64, rest string) {
|
||||
switch {
|
||||
case len(s) == 0:
|
||||
return -1, ""
|
||||
case s[0] == '0':
|
||||
q = 0
|
||||
case s[0] == '1':
|
||||
q = 1
|
||||
default:
|
||||
return -1, ""
|
||||
}
|
||||
s = s[1:]
|
||||
if !strings.HasPrefix(s, ".") {
|
||||
return q, s
|
||||
}
|
||||
s = s[1:]
|
||||
i := 0
|
||||
n := 0
|
||||
d := 1
|
||||
for ; i < len(s); i++ {
|
||||
b := s[i]
|
||||
if b < '0' || b > '9' {
|
||||
break
|
||||
}
|
||||
n = n*10 + int(b) - '0'
|
||||
d *= 10
|
||||
}
|
||||
return q + float64(n)/float64(d), s[i:]
|
||||
}
|
||||
|
||||
// Octet types from RFC 2616.
|
||||
var octetTypes [256]octetType
|
||||
|
||||
type octetType byte
|
||||
|
||||
const (
|
||||
isToken octetType = 1 << iota
|
||||
isSpace
|
||||
)
|
||||
|
||||
func init() {
|
||||
// OCTET = <any 8-bit sequence of data>
|
||||
// CHAR = <any US-ASCII character (octets 0 - 127)>
|
||||
// CTL = <any US-ASCII control character (octets 0 - 31) and DEL (127)>
|
||||
// CR = <US-ASCII CR, carriage return (13)>
|
||||
// LF = <US-ASCII LF, linefeed (10)>
|
||||
// SP = <US-ASCII SP, space (32)>
|
||||
// HT = <US-ASCII HT, horizontal-tab (9)>
|
||||
// <"> = <US-ASCII double-quote mark (34)>
|
||||
// CRLF = CR LF
|
||||
// LWS = [CRLF] 1*( SP | HT )
|
||||
// TEXT = <any OCTET except CTLs, but including LWS>
|
||||
// separators = "(" | ")" | "<" | ">" | "@" | "," | ";" | ":" | "\" | <">
|
||||
// | "/" | "[" | "]" | "?" | "=" | "{" | "}" | SP | HT
|
||||
// token = 1*<any CHAR except CTLs or separators>
|
||||
// qdtext = <any TEXT except <">>
|
||||
|
||||
for c := 0; c < 256; c++ {
|
||||
var t octetType
|
||||
isCtl := c <= 31 || c == 127
|
||||
isChar := 0 <= c && c <= 127
|
||||
isSeparator := strings.ContainsRune(" \t\"(),/:;<=>?@[]\\{}", rune(c))
|
||||
if strings.ContainsRune(" \t\r\n", rune(c)) {
|
||||
t |= isSpace
|
||||
}
|
||||
if isChar && !isCtl && !isSeparator {
|
||||
t |= isToken
|
||||
}
|
||||
octetTypes[c] = t
|
||||
}
|
||||
}
|
||||
+653
@@ -0,0 +1,653 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Utilities for building Huffman decoding tables. */
|
||||
|
||||
const huffmanMaxCodeLength = 15
|
||||
|
||||
/* Maximum possible Huffman table size for an alphabet size of (index * 32),
|
||||
max code length 15 and root table bits 8. */
|
||||
var kMaxHuffmanTableSize = []uint16{
|
||||
256,
|
||||
402,
|
||||
436,
|
||||
468,
|
||||
500,
|
||||
534,
|
||||
566,
|
||||
598,
|
||||
630,
|
||||
662,
|
||||
694,
|
||||
726,
|
||||
758,
|
||||
790,
|
||||
822,
|
||||
854,
|
||||
886,
|
||||
920,
|
||||
952,
|
||||
984,
|
||||
1016,
|
||||
1048,
|
||||
1080,
|
||||
1112,
|
||||
1144,
|
||||
1176,
|
||||
1208,
|
||||
1240,
|
||||
1272,
|
||||
1304,
|
||||
1336,
|
||||
1368,
|
||||
1400,
|
||||
1432,
|
||||
1464,
|
||||
1496,
|
||||
1528,
|
||||
}
|
||||
|
||||
/* BROTLI_NUM_BLOCK_LEN_SYMBOLS == 26 */
|
||||
const huffmanMaxSize26 = 396
|
||||
|
||||
/* BROTLI_MAX_BLOCK_TYPE_SYMBOLS == 258 */
|
||||
const huffmanMaxSize258 = 632
|
||||
|
||||
/* BROTLI_MAX_CONTEXT_MAP_SYMBOLS == 272 */
|
||||
const huffmanMaxSize272 = 646
|
||||
|
||||
const huffmanMaxCodeLengthCodeLength = 5
|
||||
|
||||
/* Do not create this struct directly - use the ConstructHuffmanCode
|
||||
* constructor below! */
|
||||
type huffmanCode struct {
|
||||
bits byte
|
||||
value uint16
|
||||
}
|
||||
|
||||
func constructHuffmanCode(bits byte, value uint16) huffmanCode {
|
||||
var h huffmanCode
|
||||
h.bits = bits
|
||||
h.value = value
|
||||
return h
|
||||
}
|
||||
|
||||
/* Builds Huffman lookup table assuming code lengths are in symbol order. */
|
||||
|
||||
/* Builds Huffman lookup table assuming code lengths are in symbol order.
|
||||
Returns size of resulting table. */
|
||||
|
||||
/* Builds a simple Huffman table. The |num_symbols| parameter is to be
|
||||
interpreted as follows: 0 means 1 symbol, 1 means 2 symbols,
|
||||
2 means 3 symbols, 3 means 4 symbols with lengths [2, 2, 2, 2],
|
||||
4 means 4 symbols with lengths [1, 2, 3, 3]. */
|
||||
|
||||
/* Contains a collection of Huffman trees with the same alphabet size. */
|
||||
/* max_symbol is needed due to simple codes since log2(alphabet_size) could be
|
||||
greater than log2(max_symbol). */
|
||||
type huffmanTreeGroup struct {
|
||||
htrees [][]huffmanCode
|
||||
codes []huffmanCode
|
||||
alphabet_size uint16
|
||||
max_symbol uint16
|
||||
num_htrees uint16
|
||||
}
|
||||
|
||||
const reverseBitsMax = 8
|
||||
|
||||
const reverseBitsBase = 0
|
||||
|
||||
var kReverseBits = [1 << reverseBitsMax]byte{
|
||||
0x00,
|
||||
0x80,
|
||||
0x40,
|
||||
0xC0,
|
||||
0x20,
|
||||
0xA0,
|
||||
0x60,
|
||||
0xE0,
|
||||
0x10,
|
||||
0x90,
|
||||
0x50,
|
||||
0xD0,
|
||||
0x30,
|
||||
0xB0,
|
||||
0x70,
|
||||
0xF0,
|
||||
0x08,
|
||||
0x88,
|
||||
0x48,
|
||||
0xC8,
|
||||
0x28,
|
||||
0xA8,
|
||||
0x68,
|
||||
0xE8,
|
||||
0x18,
|
||||
0x98,
|
||||
0x58,
|
||||
0xD8,
|
||||
0x38,
|
||||
0xB8,
|
||||
0x78,
|
||||
0xF8,
|
||||
0x04,
|
||||
0x84,
|
||||
0x44,
|
||||
0xC4,
|
||||
0x24,
|
||||
0xA4,
|
||||
0x64,
|
||||
0xE4,
|
||||
0x14,
|
||||
0x94,
|
||||
0x54,
|
||||
0xD4,
|
||||
0x34,
|
||||
0xB4,
|
||||
0x74,
|
||||
0xF4,
|
||||
0x0C,
|
||||
0x8C,
|
||||
0x4C,
|
||||
0xCC,
|
||||
0x2C,
|
||||
0xAC,
|
||||
0x6C,
|
||||
0xEC,
|
||||
0x1C,
|
||||
0x9C,
|
||||
0x5C,
|
||||
0xDC,
|
||||
0x3C,
|
||||
0xBC,
|
||||
0x7C,
|
||||
0xFC,
|
||||
0x02,
|
||||
0x82,
|
||||
0x42,
|
||||
0xC2,
|
||||
0x22,
|
||||
0xA2,
|
||||
0x62,
|
||||
0xE2,
|
||||
0x12,
|
||||
0x92,
|
||||
0x52,
|
||||
0xD2,
|
||||
0x32,
|
||||
0xB2,
|
||||
0x72,
|
||||
0xF2,
|
||||
0x0A,
|
||||
0x8A,
|
||||
0x4A,
|
||||
0xCA,
|
||||
0x2A,
|
||||
0xAA,
|
||||
0x6A,
|
||||
0xEA,
|
||||
0x1A,
|
||||
0x9A,
|
||||
0x5A,
|
||||
0xDA,
|
||||
0x3A,
|
||||
0xBA,
|
||||
0x7A,
|
||||
0xFA,
|
||||
0x06,
|
||||
0x86,
|
||||
0x46,
|
||||
0xC6,
|
||||
0x26,
|
||||
0xA6,
|
||||
0x66,
|
||||
0xE6,
|
||||
0x16,
|
||||
0x96,
|
||||
0x56,
|
||||
0xD6,
|
||||
0x36,
|
||||
0xB6,
|
||||
0x76,
|
||||
0xF6,
|
||||
0x0E,
|
||||
0x8E,
|
||||
0x4E,
|
||||
0xCE,
|
||||
0x2E,
|
||||
0xAE,
|
||||
0x6E,
|
||||
0xEE,
|
||||
0x1E,
|
||||
0x9E,
|
||||
0x5E,
|
||||
0xDE,
|
||||
0x3E,
|
||||
0xBE,
|
||||
0x7E,
|
||||
0xFE,
|
||||
0x01,
|
||||
0x81,
|
||||
0x41,
|
||||
0xC1,
|
||||
0x21,
|
||||
0xA1,
|
||||
0x61,
|
||||
0xE1,
|
||||
0x11,
|
||||
0x91,
|
||||
0x51,
|
||||
0xD1,
|
||||
0x31,
|
||||
0xB1,
|
||||
0x71,
|
||||
0xF1,
|
||||
0x09,
|
||||
0x89,
|
||||
0x49,
|
||||
0xC9,
|
||||
0x29,
|
||||
0xA9,
|
||||
0x69,
|
||||
0xE9,
|
||||
0x19,
|
||||
0x99,
|
||||
0x59,
|
||||
0xD9,
|
||||
0x39,
|
||||
0xB9,
|
||||
0x79,
|
||||
0xF9,
|
||||
0x05,
|
||||
0x85,
|
||||
0x45,
|
||||
0xC5,
|
||||
0x25,
|
||||
0xA5,
|
||||
0x65,
|
||||
0xE5,
|
||||
0x15,
|
||||
0x95,
|
||||
0x55,
|
||||
0xD5,
|
||||
0x35,
|
||||
0xB5,
|
||||
0x75,
|
||||
0xF5,
|
||||
0x0D,
|
||||
0x8D,
|
||||
0x4D,
|
||||
0xCD,
|
||||
0x2D,
|
||||
0xAD,
|
||||
0x6D,
|
||||
0xED,
|
||||
0x1D,
|
||||
0x9D,
|
||||
0x5D,
|
||||
0xDD,
|
||||
0x3D,
|
||||
0xBD,
|
||||
0x7D,
|
||||
0xFD,
|
||||
0x03,
|
||||
0x83,
|
||||
0x43,
|
||||
0xC3,
|
||||
0x23,
|
||||
0xA3,
|
||||
0x63,
|
||||
0xE3,
|
||||
0x13,
|
||||
0x93,
|
||||
0x53,
|
||||
0xD3,
|
||||
0x33,
|
||||
0xB3,
|
||||
0x73,
|
||||
0xF3,
|
||||
0x0B,
|
||||
0x8B,
|
||||
0x4B,
|
||||
0xCB,
|
||||
0x2B,
|
||||
0xAB,
|
||||
0x6B,
|
||||
0xEB,
|
||||
0x1B,
|
||||
0x9B,
|
||||
0x5B,
|
||||
0xDB,
|
||||
0x3B,
|
||||
0xBB,
|
||||
0x7B,
|
||||
0xFB,
|
||||
0x07,
|
||||
0x87,
|
||||
0x47,
|
||||
0xC7,
|
||||
0x27,
|
||||
0xA7,
|
||||
0x67,
|
||||
0xE7,
|
||||
0x17,
|
||||
0x97,
|
||||
0x57,
|
||||
0xD7,
|
||||
0x37,
|
||||
0xB7,
|
||||
0x77,
|
||||
0xF7,
|
||||
0x0F,
|
||||
0x8F,
|
||||
0x4F,
|
||||
0xCF,
|
||||
0x2F,
|
||||
0xAF,
|
||||
0x6F,
|
||||
0xEF,
|
||||
0x1F,
|
||||
0x9F,
|
||||
0x5F,
|
||||
0xDF,
|
||||
0x3F,
|
||||
0xBF,
|
||||
0x7F,
|
||||
0xFF,
|
||||
}
|
||||
|
||||
const reverseBitsLowest = (uint64(1) << (reverseBitsMax - 1 + reverseBitsBase))
|
||||
|
||||
/* Returns reverse(num >> BROTLI_REVERSE_BITS_BASE, BROTLI_REVERSE_BITS_MAX),
|
||||
where reverse(value, len) is the bit-wise reversal of the len least
|
||||
significant bits of value. */
|
||||
func reverseBits8(num uint64) uint64 {
|
||||
return uint64(kReverseBits[num])
|
||||
}
|
||||
|
||||
/* Stores code in table[0], table[step], table[2*step], ..., table[end] */
|
||||
/* Assumes that end is an integer multiple of step */
|
||||
func replicateValue(table []huffmanCode, step int, end int, code huffmanCode) {
|
||||
for {
|
||||
end -= step
|
||||
table[end] = code
|
||||
if end <= 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Returns the table width of the next 2nd level table. |count| is the histogram
|
||||
of bit lengths for the remaining symbols, |len| is the code length of the
|
||||
next processed symbol. */
|
||||
func nextTableBitSize(count []uint16, len int, root_bits int) int {
|
||||
var left int = 1 << uint(len-root_bits)
|
||||
for len < huffmanMaxCodeLength {
|
||||
left -= int(count[len])
|
||||
if left <= 0 {
|
||||
break
|
||||
}
|
||||
len++
|
||||
left <<= 1
|
||||
}
|
||||
|
||||
return len - root_bits
|
||||
}
|
||||
|
||||
func buildCodeLengthsHuffmanTable(table []huffmanCode, code_lengths []byte, count []uint16) {
|
||||
var code huffmanCode /* current table entry */ /* symbol index in original or sorted table */ /* prefix code */ /* prefix code addend */ /* step size to replicate values in current table */ /* size of current table */ /* symbols sorted by code length */
|
||||
var symbol int
|
||||
var key uint64
|
||||
var key_step uint64
|
||||
var step int
|
||||
var table_size int
|
||||
var sorted [codeLengthCodes]int
|
||||
var offset [huffmanMaxCodeLengthCodeLength + 1]int
|
||||
var bits int
|
||||
var bits_count int
|
||||
/* offsets in sorted table for each length */
|
||||
assert(huffmanMaxCodeLengthCodeLength <= reverseBitsMax)
|
||||
|
||||
/* Generate offsets into sorted symbol table by code length. */
|
||||
symbol = -1
|
||||
|
||||
bits = 1
|
||||
var i int
|
||||
for i = 0; i < huffmanMaxCodeLengthCodeLength; i++ {
|
||||
symbol += int(count[bits])
|
||||
offset[bits] = symbol
|
||||
bits++
|
||||
}
|
||||
|
||||
/* Symbols with code length 0 are placed after all other symbols. */
|
||||
offset[0] = codeLengthCodes - 1
|
||||
|
||||
/* Sort symbols by length, by symbol order within each length. */
|
||||
symbol = codeLengthCodes
|
||||
|
||||
for {
|
||||
var i int
|
||||
for i = 0; i < 6; i++ {
|
||||
symbol--
|
||||
sorted[offset[code_lengths[symbol]]] = symbol
|
||||
offset[code_lengths[symbol]]--
|
||||
}
|
||||
if symbol == 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
table_size = 1 << huffmanMaxCodeLengthCodeLength
|
||||
|
||||
/* Special case: all symbols but one have 0 code length. */
|
||||
if offset[0] == 0 {
|
||||
code = constructHuffmanCode(0, uint16(sorted[0]))
|
||||
for key = 0; key < uint64(table_size); key++ {
|
||||
table[key] = code
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
/* Fill in table. */
|
||||
key = 0
|
||||
|
||||
key_step = reverseBitsLowest
|
||||
symbol = 0
|
||||
bits = 1
|
||||
step = 2
|
||||
for {
|
||||
for bits_count = int(count[bits]); bits_count != 0; bits_count-- {
|
||||
code = constructHuffmanCode(byte(bits), uint16(sorted[symbol]))
|
||||
symbol++
|
||||
replicateValue(table[reverseBits8(key):], step, table_size, code)
|
||||
key += key_step
|
||||
}
|
||||
|
||||
step <<= 1
|
||||
key_step >>= 1
|
||||
bits++
|
||||
if bits > huffmanMaxCodeLengthCodeLength {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func buildHuffmanTable(root_table []huffmanCode, root_bits int, symbol_lists symbolList, count []uint16) uint32 {
|
||||
var code huffmanCode /* current table entry */ /* next available space in table */ /* current code length */ /* symbol index in original or sorted table */ /* prefix code */ /* prefix code addend */ /* 2nd level table prefix code */ /* 2nd level table prefix code addend */ /* step size to replicate values in current table */ /* key length of current table */ /* size of current table */ /* sum of root table size and 2nd level table sizes */
|
||||
var table []huffmanCode
|
||||
var len int
|
||||
var symbol int
|
||||
var key uint64
|
||||
var key_step uint64
|
||||
var sub_key uint64
|
||||
var sub_key_step uint64
|
||||
var step int
|
||||
var table_bits int
|
||||
var table_size int
|
||||
var total_size int
|
||||
var max_length int = -1
|
||||
var bits int
|
||||
var bits_count int
|
||||
|
||||
assert(root_bits <= reverseBitsMax)
|
||||
assert(huffmanMaxCodeLength-root_bits <= reverseBitsMax)
|
||||
|
||||
for symbolListGet(symbol_lists, max_length) == 0xFFFF {
|
||||
max_length--
|
||||
}
|
||||
max_length += huffmanMaxCodeLength + 1
|
||||
|
||||
table = root_table
|
||||
table_bits = root_bits
|
||||
table_size = 1 << uint(table_bits)
|
||||
total_size = table_size
|
||||
|
||||
/* Fill in the root table. Reduce the table size to if possible,
|
||||
and create the repetitions by memcpy. */
|
||||
if table_bits > max_length {
|
||||
table_bits = max_length
|
||||
table_size = 1 << uint(table_bits)
|
||||
}
|
||||
|
||||
key = 0
|
||||
key_step = reverseBitsLowest
|
||||
bits = 1
|
||||
step = 2
|
||||
for {
|
||||
symbol = bits - (huffmanMaxCodeLength + 1)
|
||||
for bits_count = int(count[bits]); bits_count != 0; bits_count-- {
|
||||
symbol = int(symbolListGet(symbol_lists, symbol))
|
||||
code = constructHuffmanCode(byte(bits), uint16(symbol))
|
||||
replicateValue(table[reverseBits8(key):], step, table_size, code)
|
||||
key += key_step
|
||||
}
|
||||
|
||||
step <<= 1
|
||||
key_step >>= 1
|
||||
bits++
|
||||
if bits > table_bits {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
/* If root_bits != table_bits then replicate to fill the remaining slots. */
|
||||
for total_size != table_size {
|
||||
copy(table[table_size:], table[:uint(table_size)])
|
||||
table_size <<= 1
|
||||
}
|
||||
|
||||
/* Fill in 2nd level tables and add pointers to root table. */
|
||||
key_step = reverseBitsLowest >> uint(root_bits-1)
|
||||
|
||||
sub_key = reverseBitsLowest << 1
|
||||
sub_key_step = reverseBitsLowest
|
||||
len = root_bits + 1
|
||||
step = 2
|
||||
for ; len <= max_length; len++ {
|
||||
symbol = len - (huffmanMaxCodeLength + 1)
|
||||
for ; count[len] != 0; count[len]-- {
|
||||
if sub_key == reverseBitsLowest<<1 {
|
||||
table = table[table_size:]
|
||||
table_bits = nextTableBitSize(count, int(len), root_bits)
|
||||
table_size = 1 << uint(table_bits)
|
||||
total_size += table_size
|
||||
sub_key = reverseBits8(key)
|
||||
key += key_step
|
||||
root_table[sub_key] = constructHuffmanCode(byte(table_bits+root_bits), uint16(uint64(uint(-cap(table)+cap(root_table)))-sub_key))
|
||||
sub_key = 0
|
||||
}
|
||||
|
||||
symbol = int(symbolListGet(symbol_lists, symbol))
|
||||
code = constructHuffmanCode(byte(len-root_bits), uint16(symbol))
|
||||
replicateValue(table[reverseBits8(sub_key):], step, table_size, code)
|
||||
sub_key += sub_key_step
|
||||
}
|
||||
|
||||
step <<= 1
|
||||
sub_key_step >>= 1
|
||||
}
|
||||
|
||||
return uint32(total_size)
|
||||
}
|
||||
|
||||
func buildSimpleHuffmanTable(table []huffmanCode, root_bits int, val []uint16, num_symbols uint32) uint32 {
|
||||
var table_size uint32 = 1
|
||||
var goal_size uint32 = 1 << uint(root_bits)
|
||||
switch num_symbols {
|
||||
case 0:
|
||||
table[0] = constructHuffmanCode(0, val[0])
|
||||
|
||||
case 1:
|
||||
if val[1] > val[0] {
|
||||
table[0] = constructHuffmanCode(1, val[0])
|
||||
table[1] = constructHuffmanCode(1, val[1])
|
||||
} else {
|
||||
table[0] = constructHuffmanCode(1, val[1])
|
||||
table[1] = constructHuffmanCode(1, val[0])
|
||||
}
|
||||
|
||||
table_size = 2
|
||||
|
||||
case 2:
|
||||
table[0] = constructHuffmanCode(1, val[0])
|
||||
table[2] = constructHuffmanCode(1, val[0])
|
||||
if val[2] > val[1] {
|
||||
table[1] = constructHuffmanCode(2, val[1])
|
||||
table[3] = constructHuffmanCode(2, val[2])
|
||||
} else {
|
||||
table[1] = constructHuffmanCode(2, val[2])
|
||||
table[3] = constructHuffmanCode(2, val[1])
|
||||
}
|
||||
|
||||
table_size = 4
|
||||
|
||||
case 3:
|
||||
var i int
|
||||
var k int
|
||||
for i = 0; i < 3; i++ {
|
||||
for k = i + 1; k < 4; k++ {
|
||||
if val[k] < val[i] {
|
||||
var t uint16 = val[k]
|
||||
val[k] = val[i]
|
||||
val[i] = t
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
table[0] = constructHuffmanCode(2, val[0])
|
||||
table[2] = constructHuffmanCode(2, val[1])
|
||||
table[1] = constructHuffmanCode(2, val[2])
|
||||
table[3] = constructHuffmanCode(2, val[3])
|
||||
table_size = 4
|
||||
|
||||
case 4:
|
||||
if val[3] < val[2] {
|
||||
var t uint16 = val[3]
|
||||
val[3] = val[2]
|
||||
val[2] = t
|
||||
}
|
||||
|
||||
table[0] = constructHuffmanCode(1, val[0])
|
||||
table[1] = constructHuffmanCode(2, val[1])
|
||||
table[2] = constructHuffmanCode(1, val[0])
|
||||
table[3] = constructHuffmanCode(3, val[2])
|
||||
table[4] = constructHuffmanCode(1, val[0])
|
||||
table[5] = constructHuffmanCode(2, val[1])
|
||||
table[6] = constructHuffmanCode(1, val[0])
|
||||
table[7] = constructHuffmanCode(3, val[3])
|
||||
table_size = 8
|
||||
}
|
||||
|
||||
for table_size != goal_size {
|
||||
copy(table[table_size:], table[:uint(table_size)])
|
||||
table_size <<= 1
|
||||
}
|
||||
|
||||
return goal_size
|
||||
}
|
||||
+182
@@ -0,0 +1,182 @@
|
||||
package brotli
|
||||
|
||||
func utf8Position(last uint, c uint, clamp uint) uint {
|
||||
if c < 128 {
|
||||
return 0 /* Next one is the 'Byte 1' again. */
|
||||
} else if c >= 192 { /* Next one is the 'Byte 2' of utf-8 encoding. */
|
||||
return brotli_min_size_t(1, clamp)
|
||||
} else {
|
||||
/* Let's decide over the last byte if this ends the sequence. */
|
||||
if last < 0xE0 {
|
||||
return 0 /* Completed two or three byte coding. */ /* Next one is the 'Byte 3' of utf-8 encoding. */
|
||||
} else {
|
||||
return brotli_min_size_t(2, clamp)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func decideMultiByteStatsLevel(pos uint, len uint, mask uint, data []byte) uint {
|
||||
var counts = [3]uint{0} /* should be 2, but 1 compresses better. */
|
||||
var max_utf8 uint = 1
|
||||
var last_c uint = 0
|
||||
var i uint
|
||||
for i = 0; i < len; i++ {
|
||||
var c uint = uint(data[(pos+i)&mask])
|
||||
counts[utf8Position(last_c, c, 2)]++
|
||||
last_c = c
|
||||
}
|
||||
|
||||
if counts[2] < 500 {
|
||||
max_utf8 = 1
|
||||
}
|
||||
|
||||
if counts[1]+counts[2] < 25 {
|
||||
max_utf8 = 0
|
||||
}
|
||||
|
||||
return max_utf8
|
||||
}
|
||||
|
||||
func estimateBitCostsForLiteralsUTF8(pos uint, len uint, mask uint, data []byte, cost []float32) {
|
||||
var max_utf8 uint = decideMultiByteStatsLevel(pos, uint(len), mask, data)
|
||||
/* Bootstrap histograms. */
|
||||
var histogram = [3][256]uint{[256]uint{0}}
|
||||
var window_half uint = 495
|
||||
var in_window uint = brotli_min_size_t(window_half, uint(len))
|
||||
var in_window_utf8 = [3]uint{0}
|
||||
/* max_utf8 is 0 (normal ASCII single byte modeling),
|
||||
1 (for 2-byte UTF-8 modeling), or 2 (for 3-byte UTF-8 modeling). */
|
||||
|
||||
var i uint
|
||||
{
|
||||
var last_c uint = 0
|
||||
var utf8_pos uint = 0
|
||||
for i = 0; i < in_window; i++ {
|
||||
var c uint = uint(data[(pos+i)&mask])
|
||||
histogram[utf8_pos][c]++
|
||||
in_window_utf8[utf8_pos]++
|
||||
utf8_pos = utf8Position(last_c, c, max_utf8)
|
||||
last_c = c
|
||||
}
|
||||
}
|
||||
|
||||
/* Compute bit costs with sliding window. */
|
||||
for i = 0; i < len; i++ {
|
||||
if i >= window_half {
|
||||
var c uint
|
||||
var last_c uint
|
||||
if i < window_half+1 {
|
||||
c = 0
|
||||
} else {
|
||||
c = uint(data[(pos+i-window_half-1)&mask])
|
||||
}
|
||||
if i < window_half+2 {
|
||||
last_c = 0
|
||||
} else {
|
||||
last_c = uint(data[(pos+i-window_half-2)&mask])
|
||||
}
|
||||
/* Remove a byte in the past. */
|
||||
|
||||
var utf8_pos2 uint = utf8Position(last_c, c, max_utf8)
|
||||
histogram[utf8_pos2][data[(pos+i-window_half)&mask]]--
|
||||
in_window_utf8[utf8_pos2]--
|
||||
}
|
||||
|
||||
if i+window_half < len {
|
||||
var c uint = uint(data[(pos+i+window_half-1)&mask])
|
||||
var last_c uint = uint(data[(pos+i+window_half-2)&mask])
|
||||
/* Add a byte in the future. */
|
||||
|
||||
var utf8_pos2 uint = utf8Position(last_c, c, max_utf8)
|
||||
histogram[utf8_pos2][data[(pos+i+window_half)&mask]]++
|
||||
in_window_utf8[utf8_pos2]++
|
||||
}
|
||||
{
|
||||
var c uint
|
||||
var last_c uint
|
||||
if i < 1 {
|
||||
c = 0
|
||||
} else {
|
||||
c = uint(data[(pos+i-1)&mask])
|
||||
}
|
||||
if i < 2 {
|
||||
last_c = 0
|
||||
} else {
|
||||
last_c = uint(data[(pos+i-2)&mask])
|
||||
}
|
||||
var utf8_pos uint = utf8Position(last_c, c, max_utf8)
|
||||
var masked_pos uint = (pos + i) & mask
|
||||
var histo uint = histogram[utf8_pos][data[masked_pos]]
|
||||
var lit_cost float64
|
||||
if histo == 0 {
|
||||
histo = 1
|
||||
}
|
||||
|
||||
lit_cost = fastLog2(in_window_utf8[utf8_pos]) - fastLog2(histo)
|
||||
lit_cost += 0.02905
|
||||
if lit_cost < 1.0 {
|
||||
lit_cost *= 0.5
|
||||
lit_cost += 0.5
|
||||
}
|
||||
|
||||
/* Make the first bytes more expensive -- seems to help, not sure why.
|
||||
Perhaps because the entropy source is changing its properties
|
||||
rapidly in the beginning of the file, perhaps because the beginning
|
||||
of the data is a statistical "anomaly". */
|
||||
if i < 2000 {
|
||||
lit_cost += 0.7 - (float64(2000-i) / 2000.0 * 0.35)
|
||||
}
|
||||
|
||||
cost[i] = float32(lit_cost)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func estimateBitCostsForLiterals(pos uint, len uint, mask uint, data []byte, cost []float32) {
|
||||
if isMostlyUTF8(data, pos, mask, uint(len), kMinUTF8Ratio) {
|
||||
estimateBitCostsForLiteralsUTF8(pos, uint(len), mask, data, cost)
|
||||
return
|
||||
} else {
|
||||
var histogram = [256]uint{0}
|
||||
var window_half uint = 2000
|
||||
var in_window uint = brotli_min_size_t(window_half, uint(len))
|
||||
var i uint
|
||||
/* Bootstrap histogram. */
|
||||
for i = 0; i < in_window; i++ {
|
||||
histogram[data[(pos+i)&mask]]++
|
||||
}
|
||||
|
||||
/* Compute bit costs with sliding window. */
|
||||
for i = 0; i < len; i++ {
|
||||
var histo uint
|
||||
if i >= window_half {
|
||||
/* Remove a byte in the past. */
|
||||
histogram[data[(pos+i-window_half)&mask]]--
|
||||
|
||||
in_window--
|
||||
}
|
||||
|
||||
if i+window_half < len {
|
||||
/* Add a byte in the future. */
|
||||
histogram[data[(pos+i+window_half)&mask]]++
|
||||
|
||||
in_window++
|
||||
}
|
||||
|
||||
histo = histogram[data[(pos+i)&mask]]
|
||||
if histo == 0 {
|
||||
histo = 1
|
||||
}
|
||||
{
|
||||
var lit_cost float64 = fastLog2(in_window) - fastLog2(histo)
|
||||
lit_cost += 0.029
|
||||
if lit_cost < 1.0 {
|
||||
lit_cost *= 0.5
|
||||
lit_cost += 0.5
|
||||
}
|
||||
|
||||
cost[i] = float32(lit_cost)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
package matchfinder
|
||||
|
||||
// An absoluteMatch is like a Match, but it stores indexes into the byte
|
||||
// stream instead of lengths.
|
||||
type absoluteMatch struct {
|
||||
// Start is the index of the first byte.
|
||||
Start int
|
||||
|
||||
// End is the index of the byte after the last byte
|
||||
// (so that End - Start = Length).
|
||||
End int
|
||||
|
||||
// Match is the index of the previous data that matches
|
||||
// (Start - Match = Distance).
|
||||
Match int
|
||||
}
|
||||
|
||||
// A matchEmitter manages the output of matches for a MatchFinder.
|
||||
type matchEmitter struct {
|
||||
// Dst is the destination slice that Matches are added to.
|
||||
Dst []Match
|
||||
|
||||
// NextEmit is the index of the next byte to emit.
|
||||
NextEmit int
|
||||
}
|
||||
|
||||
func (e *matchEmitter) emit(m absoluteMatch) {
|
||||
e.Dst = append(e.Dst, Match{
|
||||
Unmatched: m.Start - e.NextEmit,
|
||||
Length: m.End - m.Start,
|
||||
Distance: m.Start - m.Match,
|
||||
})
|
||||
e.NextEmit = m.End
|
||||
}
|
||||
+169
@@ -0,0 +1,169 @@
|
||||
package matchfinder
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
)
|
||||
|
||||
// M0 is an implementation of the MatchFinder interface based
|
||||
// on the algorithm used by snappy, but modified to be more like the algorithm
|
||||
// used by compression level 0 of the brotli reference implementation.
|
||||
//
|
||||
// It has a maximum block size of 65536 bytes.
|
||||
type M0 struct {
|
||||
// Lazy turns on "lazy matching," for higher compression but less speed.
|
||||
Lazy bool
|
||||
|
||||
MaxDistance int
|
||||
MaxLength int
|
||||
}
|
||||
|
||||
func (M0) Reset() {}
|
||||
|
||||
const (
|
||||
m0HashLen = 5
|
||||
|
||||
m0TableBits = 14
|
||||
m0TableSize = 1 << m0TableBits
|
||||
m0Shift = 32 - m0TableBits
|
||||
// m0TableMask is redundant, but helps the compiler eliminate bounds
|
||||
// checks.
|
||||
m0TableMask = m0TableSize - 1
|
||||
)
|
||||
|
||||
func (m M0) hash(data uint64) uint64 {
|
||||
hash := (data << (64 - 8*m0HashLen)) * hashMul64
|
||||
return hash >> (64 - m0TableBits)
|
||||
}
|
||||
|
||||
// FindMatches looks for matches in src, appends them to dst, and returns dst.
|
||||
// src must not be longer than 65536 bytes.
|
||||
func (m M0) FindMatches(dst []Match, src []byte) []Match {
|
||||
const inputMargin = 16 - 1
|
||||
const minNonLiteralBlockSize = 1 + 1 + inputMargin
|
||||
|
||||
if len(src) < minNonLiteralBlockSize {
|
||||
dst = append(dst, Match{
|
||||
Unmatched: len(src),
|
||||
})
|
||||
return dst
|
||||
}
|
||||
if len(src) > 65536 {
|
||||
panic("block too long")
|
||||
}
|
||||
|
||||
var table [m0TableSize]uint16
|
||||
|
||||
// sLimit is when to stop looking for offset/length copies. The inputMargin
|
||||
// lets us use a fast path for emitLiteral in the main loop, while we are
|
||||
// looking for copies.
|
||||
sLimit := len(src) - inputMargin
|
||||
|
||||
// nextEmit is where in src the next emitLiteral should start from.
|
||||
nextEmit := 0
|
||||
|
||||
// The encoded form must start with a literal, as there are no previous
|
||||
// bytes to copy, so we start looking for hash matches at s == 1.
|
||||
s := 1
|
||||
nextHash := m.hash(binary.LittleEndian.Uint64(src[s:]))
|
||||
|
||||
for {
|
||||
// Copied from the C++ snappy implementation:
|
||||
//
|
||||
// Heuristic match skipping: If 32 bytes are scanned with no matches
|
||||
// found, start looking only at every other byte. If 32 more bytes are
|
||||
// scanned (or skipped), look at every third byte, etc.. When a match
|
||||
// is found, immediately go back to looking at every byte. This is a
|
||||
// small loss (~5% performance, ~0.1% density) for compressible data
|
||||
// due to more bookkeeping, but for non-compressible data (such as
|
||||
// JPEG) it's a huge win since the compressor quickly "realizes" the
|
||||
// data is incompressible and doesn't bother looking for matches
|
||||
// everywhere.
|
||||
//
|
||||
// The "skip" variable keeps track of how many bytes there are since
|
||||
// the last match; dividing it by 32 (ie. right-shifting by five) gives
|
||||
// the number of bytes to move ahead for each iteration.
|
||||
skip := 32
|
||||
|
||||
nextS := s
|
||||
candidate := 0
|
||||
for {
|
||||
s = nextS
|
||||
bytesBetweenHashLookups := skip >> 5
|
||||
nextS = s + bytesBetweenHashLookups
|
||||
skip += bytesBetweenHashLookups
|
||||
if nextS > sLimit {
|
||||
goto emitRemainder
|
||||
}
|
||||
candidate = int(table[nextHash&m0TableMask])
|
||||
table[nextHash&m0TableMask] = uint16(s)
|
||||
nextHash = m.hash(binary.LittleEndian.Uint64(src[nextS:]))
|
||||
if m.MaxDistance != 0 && s-candidate > m.MaxDistance {
|
||||
continue
|
||||
}
|
||||
if binary.LittleEndian.Uint32(src[s:]) == binary.LittleEndian.Uint32(src[candidate:]) {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Invariant: we have a 4-byte match at s.
|
||||
base := s
|
||||
s = extendMatch(src, candidate+4, s+4)
|
||||
|
||||
origBase := base
|
||||
if m.Lazy && base+1 < sLimit {
|
||||
newBase := base + 1
|
||||
h := m.hash(binary.LittleEndian.Uint64(src[newBase:]))
|
||||
newCandidate := int(table[h&m0TableMask])
|
||||
table[h&m0TableMask] = uint16(newBase)
|
||||
okDistance := true
|
||||
if m.MaxDistance != 0 && newBase-newCandidate > m.MaxDistance {
|
||||
okDistance = false
|
||||
}
|
||||
if okDistance && binary.LittleEndian.Uint32(src[newBase:]) == binary.LittleEndian.Uint32(src[newCandidate:]) {
|
||||
newS := extendMatch(src, newCandidate+4, newBase+4)
|
||||
if newS-newBase > s-base+1 {
|
||||
s = newS
|
||||
base = newBase
|
||||
candidate = newCandidate
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if m.MaxLength != 0 && s-base > m.MaxLength {
|
||||
s = base + m.MaxLength
|
||||
}
|
||||
dst = append(dst, Match{
|
||||
Unmatched: base - nextEmit,
|
||||
Length: s - base,
|
||||
Distance: base - candidate,
|
||||
})
|
||||
nextEmit = s
|
||||
if s >= sLimit {
|
||||
goto emitRemainder
|
||||
}
|
||||
|
||||
if m.Lazy {
|
||||
// If lazy matching is enabled, we update the hash table for
|
||||
// every byte in the match.
|
||||
for i := origBase + 2; i < s-1; i++ {
|
||||
x := binary.LittleEndian.Uint64(src[i:])
|
||||
table[m.hash(x)&m0TableMask] = uint16(i)
|
||||
}
|
||||
}
|
||||
|
||||
// We could immediately start working at s now, but to improve
|
||||
// compression we first update the hash table at s-1 and at s.
|
||||
x := binary.LittleEndian.Uint64(src[s-1:])
|
||||
prevHash := m.hash(x >> 0)
|
||||
table[prevHash&m0TableMask] = uint16(s - 1)
|
||||
nextHash = m.hash(x >> 8)
|
||||
}
|
||||
|
||||
emitRemainder:
|
||||
if nextEmit < len(src) {
|
||||
dst = append(dst, Match{
|
||||
Unmatched: len(src) - nextEmit,
|
||||
})
|
||||
}
|
||||
return dst
|
||||
}
|
||||
+308
@@ -0,0 +1,308 @@
|
||||
package matchfinder
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"math/bits"
|
||||
"runtime"
|
||||
)
|
||||
|
||||
// M4 is an implementation of the MatchFinder
|
||||
// interface that uses a hash table to find matches,
|
||||
// optional match chains,
|
||||
// and the advanced parsing technique from
|
||||
// https://fastcompression.blogspot.com/2011/12/advanced-parsing-strategies.html.
|
||||
type M4 struct {
|
||||
// MaxDistance is the maximum distance (in bytes) to look back for
|
||||
// a match. The default is 65535.
|
||||
MaxDistance int
|
||||
|
||||
// MinLength is the length of the shortest match to return.
|
||||
// The default is 4.
|
||||
MinLength int
|
||||
|
||||
// HashLen is the number of bytes to use to calculate the hashes.
|
||||
// The maximum is 8 and the default is 6.
|
||||
HashLen int
|
||||
|
||||
// TableBits is the number of bits in the hash table indexes.
|
||||
// The default is 17 (128K entries).
|
||||
TableBits int
|
||||
|
||||
// ChainLength is how many entries to search on the "match chain" of older
|
||||
// locations with the same hash as the current location.
|
||||
ChainLength int
|
||||
|
||||
// DistanceBitCost is used when comparing two matches to see
|
||||
// which is better. The comparison is primarily based on the length
|
||||
// of the matches, but it can also take the distance into account,
|
||||
// in terms of the number of bits needed to represent the distance.
|
||||
// One byte of length is given a score of 256, so 32 (256/8) would
|
||||
// be a reasonable first guess for the value of one bit.
|
||||
// (The default is 0, which bases the comparison solely on length.)
|
||||
DistanceBitCost int
|
||||
|
||||
table []uint32
|
||||
chain []uint16
|
||||
|
||||
history []byte
|
||||
}
|
||||
|
||||
func (q *M4) Reset() {
|
||||
for i := range q.table {
|
||||
q.table[i] = 0
|
||||
}
|
||||
q.history = q.history[:0]
|
||||
q.chain = q.chain[:0]
|
||||
}
|
||||
|
||||
func (q *M4) score(m absoluteMatch) int {
|
||||
return (m.End-m.Start)*256 + (bits.LeadingZeros32(uint32(m.Start-m.Match))-32)*q.DistanceBitCost
|
||||
}
|
||||
|
||||
func (q *M4) FindMatches(dst []Match, src []byte) []Match {
|
||||
if q.MaxDistance == 0 {
|
||||
q.MaxDistance = 65535
|
||||
}
|
||||
if q.MinLength == 0 {
|
||||
q.MinLength = 4
|
||||
}
|
||||
if q.HashLen == 0 {
|
||||
q.HashLen = 6
|
||||
}
|
||||
if q.TableBits == 0 {
|
||||
q.TableBits = 17
|
||||
}
|
||||
if len(q.table) < 1<<q.TableBits {
|
||||
q.table = make([]uint32, 1<<q.TableBits)
|
||||
}
|
||||
|
||||
e := matchEmitter{Dst: dst}
|
||||
|
||||
if len(q.history) > q.MaxDistance*2 {
|
||||
// Trim down the history buffer.
|
||||
delta := len(q.history) - q.MaxDistance
|
||||
copy(q.history, q.history[delta:])
|
||||
q.history = q.history[:q.MaxDistance]
|
||||
if q.ChainLength > 0 {
|
||||
q.chain = q.chain[:q.MaxDistance]
|
||||
}
|
||||
|
||||
for i, v := range q.table {
|
||||
newV := int(v) - delta
|
||||
if newV < 0 {
|
||||
newV = 0
|
||||
}
|
||||
q.table[i] = uint32(newV)
|
||||
}
|
||||
}
|
||||
|
||||
// Append src to the history buffer.
|
||||
e.NextEmit = len(q.history)
|
||||
q.history = append(q.history, src...)
|
||||
if q.ChainLength > 0 {
|
||||
q.chain = append(q.chain, make([]uint16, len(src))...)
|
||||
}
|
||||
src = q.history
|
||||
|
||||
// matches stores the matches that have been found but not emitted,
|
||||
// in reverse order. (matches[0] is the most recent one.)
|
||||
var matches [3]absoluteMatch
|
||||
for i := e.NextEmit; i < len(src)-7; i++ {
|
||||
if matches[0] != (absoluteMatch{}) && i >= matches[0].End {
|
||||
// We have found some matches, and we're far enough along that we probably
|
||||
// won't find overlapping matches, so we might as well emit them.
|
||||
if matches[1] != (absoluteMatch{}) {
|
||||
if matches[1].End > matches[0].Start {
|
||||
matches[1].End = matches[0].Start
|
||||
}
|
||||
if matches[1].End-matches[1].Start >= q.MinLength && q.score(matches[1]) > 0 {
|
||||
e.emit(matches[1])
|
||||
}
|
||||
}
|
||||
e.emit(matches[0])
|
||||
matches = [3]absoluteMatch{}
|
||||
}
|
||||
|
||||
// Calculate and store the hash.
|
||||
h := ((binary.LittleEndian.Uint64(src[i:]) & (1<<(8*q.HashLen) - 1)) * hashMul64) >> (64 - q.TableBits)
|
||||
candidate := int(q.table[h])
|
||||
q.table[h] = uint32(i)
|
||||
if q.ChainLength > 0 && candidate != 0 {
|
||||
delta := i - candidate
|
||||
if delta < 1<<16 {
|
||||
q.chain[i] = uint16(delta)
|
||||
}
|
||||
}
|
||||
|
||||
if i < matches[0].End && i != matches[0].End+2-q.HashLen {
|
||||
continue
|
||||
}
|
||||
if candidate == 0 || i-candidate > q.MaxDistance {
|
||||
continue
|
||||
}
|
||||
|
||||
// Look for a match.
|
||||
var currentMatch absoluteMatch
|
||||
|
||||
if binary.LittleEndian.Uint32(src[candidate:]) == binary.LittleEndian.Uint32(src[i:]) {
|
||||
m := extendMatch2(src, i, candidate, e.NextEmit)
|
||||
if m.End-m.Start > q.MinLength && q.score(m) > 0 {
|
||||
currentMatch = m
|
||||
}
|
||||
}
|
||||
|
||||
for j := 0; j < q.ChainLength; j++ {
|
||||
delta := q.chain[candidate]
|
||||
if delta == 0 {
|
||||
break
|
||||
}
|
||||
candidate -= int(delta)
|
||||
if candidate <= 0 || i-candidate > q.MaxDistance {
|
||||
break
|
||||
}
|
||||
if binary.LittleEndian.Uint32(src[candidate:]) == binary.LittleEndian.Uint32(src[i:]) {
|
||||
m := extendMatch2(src, i, candidate, e.NextEmit)
|
||||
if m.End-m.Start > q.MinLength && q.score(m) > q.score(currentMatch) {
|
||||
currentMatch = m
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if currentMatch.End-currentMatch.Start < q.MinLength {
|
||||
continue
|
||||
}
|
||||
|
||||
overlapPenalty := 0
|
||||
if matches[0] != (absoluteMatch{}) {
|
||||
overlapPenalty = 275
|
||||
if currentMatch.Start <= matches[1].End {
|
||||
// This match would completely replace the previous match,
|
||||
// so there is no penalty for overlap.
|
||||
overlapPenalty = 0
|
||||
}
|
||||
}
|
||||
|
||||
if q.score(currentMatch) <= q.score(matches[0])+overlapPenalty {
|
||||
continue
|
||||
}
|
||||
|
||||
matches = [3]absoluteMatch{
|
||||
currentMatch,
|
||||
matches[0],
|
||||
matches[1],
|
||||
}
|
||||
|
||||
if matches[2] == (absoluteMatch{}) {
|
||||
continue
|
||||
}
|
||||
|
||||
// We have three matches, so it's time to emit one and/or eliminate one.
|
||||
switch {
|
||||
case matches[0].Start < matches[2].End:
|
||||
// The first and third matches overlap; discard the one in between.
|
||||
matches = [3]absoluteMatch{
|
||||
matches[0],
|
||||
matches[2],
|
||||
absoluteMatch{},
|
||||
}
|
||||
|
||||
case matches[0].Start < matches[2].End+q.MinLength:
|
||||
// The first and third matches don't overlap, but there's no room for
|
||||
// another match between them. Emit the first match and discard the second.
|
||||
e.emit(matches[2])
|
||||
matches = [3]absoluteMatch{
|
||||
matches[0],
|
||||
absoluteMatch{},
|
||||
absoluteMatch{},
|
||||
}
|
||||
|
||||
default:
|
||||
// Emit the first match, shortening it if necessary to avoid overlap with the second.
|
||||
if matches[2].End > matches[1].Start {
|
||||
matches[2].End = matches[1].Start
|
||||
}
|
||||
if matches[2].End-matches[2].Start >= q.MinLength && q.score(matches[2]) > 0 {
|
||||
e.emit(matches[2])
|
||||
}
|
||||
matches[2] = absoluteMatch{}
|
||||
}
|
||||
}
|
||||
|
||||
// We've found all the matches now; emit the remaining ones.
|
||||
if matches[1] != (absoluteMatch{}) {
|
||||
if matches[1].End > matches[0].Start {
|
||||
matches[1].End = matches[0].Start
|
||||
}
|
||||
if matches[1].End-matches[1].Start >= q.MinLength && q.score(matches[1]) > 0 {
|
||||
e.emit(matches[1])
|
||||
}
|
||||
}
|
||||
if matches[0] != (absoluteMatch{}) {
|
||||
e.emit(matches[0])
|
||||
}
|
||||
|
||||
dst = e.Dst
|
||||
if e.NextEmit < len(src) {
|
||||
dst = append(dst, Match{
|
||||
Unmatched: len(src) - e.NextEmit,
|
||||
})
|
||||
}
|
||||
|
||||
return dst
|
||||
}
|
||||
|
||||
const hashMul64 = 0x1E35A7BD1E35A7BD
|
||||
|
||||
// extendMatch returns the largest k such that k <= len(src) and that
|
||||
// src[i:i+k-j] and src[j:k] have the same contents.
|
||||
//
|
||||
// It assumes that:
|
||||
//
|
||||
// 0 <= i && i < j && j <= len(src)
|
||||
func extendMatch(src []byte, i, j int) int {
|
||||
switch runtime.GOARCH {
|
||||
case "amd64":
|
||||
// As long as we are 8 or more bytes before the end of src, we can load and
|
||||
// compare 8 bytes at a time. If those 8 bytes are equal, repeat.
|
||||
for j+8 < len(src) {
|
||||
iBytes := binary.LittleEndian.Uint64(src[i:])
|
||||
jBytes := binary.LittleEndian.Uint64(src[j:])
|
||||
if iBytes != jBytes {
|
||||
// If those 8 bytes were not equal, XOR the two 8 byte values, and return
|
||||
// the index of the first byte that differs. The BSF instruction finds the
|
||||
// least significant 1 bit, the amd64 architecture is little-endian, and
|
||||
// the shift by 3 converts a bit index to a byte index.
|
||||
return j + bits.TrailingZeros64(iBytes^jBytes)>>3
|
||||
}
|
||||
i, j = i+8, j+8
|
||||
}
|
||||
case "386":
|
||||
// On a 32-bit CPU, we do it 4 bytes at a time.
|
||||
for j+4 < len(src) {
|
||||
iBytes := binary.LittleEndian.Uint32(src[i:])
|
||||
jBytes := binary.LittleEndian.Uint32(src[j:])
|
||||
if iBytes != jBytes {
|
||||
return j + bits.TrailingZeros32(iBytes^jBytes)>>3
|
||||
}
|
||||
i, j = i+4, j+4
|
||||
}
|
||||
}
|
||||
for ; j < len(src) && src[i] == src[j]; i, j = i+1, j+1 {
|
||||
}
|
||||
return j
|
||||
}
|
||||
|
||||
// Given a 4-byte match at src[start] and src[candidate], extendMatch2 extends it
|
||||
// upward as far as possible, and downward no farther than to min.
|
||||
func extendMatch2(src []byte, start, candidate, min int) absoluteMatch {
|
||||
end := extendMatch(src, candidate+4, start+4)
|
||||
for start > min && candidate > 0 && src[start-1] == src[candidate-1] {
|
||||
start--
|
||||
candidate--
|
||||
}
|
||||
return absoluteMatch{
|
||||
Start: start,
|
||||
End: end,
|
||||
Match: candidate,
|
||||
}
|
||||
}
|
||||
+103
@@ -0,0 +1,103 @@
|
||||
// The matchfinder package defines reusable components for data compression.
|
||||
//
|
||||
// Many compression libraries have two main parts:
|
||||
// - Something that looks for repeated sequences of bytes
|
||||
// - An encoder for the compressed data format (often an entropy coder)
|
||||
//
|
||||
// Although these are logically two separate steps, the implementations are
|
||||
// usually closely tied together. You can't use flate's matcher with snappy's
|
||||
// encoder, for example. This package defines interfaces and an intermediate
|
||||
// representation to allow mixing and matching compression components.
|
||||
package matchfinder
|
||||
|
||||
import "io"
|
||||
|
||||
// A Match is the basic unit of LZ77 compression.
|
||||
type Match struct {
|
||||
Unmatched int // the number of unmatched bytes since the previous match
|
||||
Length int // the number of bytes in the matched string; it may be 0 at the end of the input
|
||||
Distance int // how far back in the stream to copy from
|
||||
}
|
||||
|
||||
// A MatchFinder performs the LZ77 stage of compression, looking for matches.
|
||||
type MatchFinder interface {
|
||||
// FindMatches looks for matches in src, appends them to dst, and returns dst.
|
||||
FindMatches(dst []Match, src []byte) []Match
|
||||
|
||||
// Reset clears any internal state, preparing the MatchFinder to be used with
|
||||
// a new stream.
|
||||
Reset()
|
||||
}
|
||||
|
||||
// An Encoder encodes the data in its final format.
|
||||
type Encoder interface {
|
||||
// Encode appends the encoded format of src to dst, using the match
|
||||
// information from matches.
|
||||
Encode(dst []byte, src []byte, matches []Match, lastBlock bool) []byte
|
||||
|
||||
// Reset clears any internal state, preparing the Encoder to be used with
|
||||
// a new stream.
|
||||
Reset()
|
||||
}
|
||||
|
||||
// A Writer uses MatchFinder and Encoder to write compressed data to Dest.
|
||||
type Writer struct {
|
||||
Dest io.Writer
|
||||
MatchFinder MatchFinder
|
||||
Encoder Encoder
|
||||
|
||||
// BlockSize is the number of bytes to compress at a time. If it is zero,
|
||||
// each Write operation will be treated as one block.
|
||||
BlockSize int
|
||||
|
||||
err error
|
||||
inBuf []byte
|
||||
outBuf []byte
|
||||
matches []Match
|
||||
}
|
||||
|
||||
func (w *Writer) Write(p []byte) (n int, err error) {
|
||||
if w.err != nil {
|
||||
return 0, w.err
|
||||
}
|
||||
|
||||
if w.BlockSize == 0 {
|
||||
return w.writeBlock(p, false)
|
||||
}
|
||||
|
||||
w.inBuf = append(w.inBuf, p...)
|
||||
var pos int
|
||||
for pos = 0; pos+w.BlockSize <= len(w.inBuf) && w.err == nil; pos += w.BlockSize {
|
||||
w.writeBlock(w.inBuf[pos:pos+w.BlockSize], false)
|
||||
}
|
||||
if pos > 0 {
|
||||
n := copy(w.inBuf, w.inBuf[pos:])
|
||||
w.inBuf = w.inBuf[:n]
|
||||
}
|
||||
|
||||
return len(p), w.err
|
||||
}
|
||||
|
||||
func (w *Writer) writeBlock(p []byte, lastBlock bool) (n int, err error) {
|
||||
w.outBuf = w.outBuf[:0]
|
||||
w.matches = w.MatchFinder.FindMatches(w.matches[:0], p)
|
||||
w.outBuf = w.Encoder.Encode(w.outBuf, p, w.matches, lastBlock)
|
||||
_, w.err = w.Dest.Write(w.outBuf)
|
||||
return len(p), w.err
|
||||
}
|
||||
|
||||
func (w *Writer) Close() error {
|
||||
w.writeBlock(w.inBuf, true)
|
||||
w.inBuf = w.inBuf[:0]
|
||||
return w.err
|
||||
}
|
||||
|
||||
func (w *Writer) Reset(newDest io.Writer) {
|
||||
w.MatchFinder.Reset()
|
||||
w.Encoder.Reset()
|
||||
w.err = nil
|
||||
w.inBuf = w.inBuf[:0]
|
||||
w.outBuf = w.outBuf[:0]
|
||||
w.matches = w.matches[:0]
|
||||
w.Dest = newDest
|
||||
}
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
package matchfinder
|
||||
|
||||
import "fmt"
|
||||
|
||||
// A TextEncoder is an Encoder that produces a human-readable representation of
|
||||
// the LZ77 compression. Matches are replaced with <Length,Distance> symbols.
|
||||
type TextEncoder struct{}
|
||||
|
||||
func (t TextEncoder) Reset() {}
|
||||
|
||||
func (t TextEncoder) Encode(dst []byte, src []byte, matches []Match, lastBlock bool) []byte {
|
||||
pos := 0
|
||||
for _, m := range matches {
|
||||
if m.Unmatched > 0 {
|
||||
dst = append(dst, src[pos:pos+m.Unmatched]...)
|
||||
pos += m.Unmatched
|
||||
}
|
||||
if m.Length > 0 {
|
||||
dst = append(dst, []byte(fmt.Sprintf("<%d,%d>", m.Length, m.Distance))...)
|
||||
pos += m.Length
|
||||
}
|
||||
}
|
||||
if pos < len(src) {
|
||||
dst = append(dst, src[pos:]...)
|
||||
}
|
||||
return dst
|
||||
}
|
||||
|
||||
// A NoMatchFinder implements MatchFinder, but doesn't find any matches.
|
||||
// It can be used to implement the equivalent of the standard library flate package's
|
||||
// HuffmanOnly setting.
|
||||
type NoMatchFinder struct{}
|
||||
|
||||
func (n NoMatchFinder) Reset() {}
|
||||
|
||||
func (n NoMatchFinder) FindMatches(dst []Match, src []byte) []Match {
|
||||
return append(dst, Match{
|
||||
Unmatched: len(src),
|
||||
})
|
||||
}
|
||||
|
||||
// AutoReset wraps a MatchFinder that can return references to data in previous
|
||||
// blocks, and calls Reset before each block. It is useful for (e.g.) using a
|
||||
// snappy Encoder with a MatchFinder designed for flate. (Snappy doesn't
|
||||
// support references between blocks.)
|
||||
type AutoReset struct {
|
||||
MatchFinder
|
||||
}
|
||||
|
||||
func (a AutoReset) FindMatches(dst []Match, src []byte) []Match {
|
||||
a.Reset()
|
||||
return a.MatchFinder.FindMatches(dst, src)
|
||||
}
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2016 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/*
|
||||
Dynamically grows array capacity to at least the requested size
|
||||
T: data type
|
||||
A: array
|
||||
C: capacity
|
||||
R: requested size
|
||||
*/
|
||||
func brotli_ensure_capacity_uint8_t(a *[]byte, c *uint, r uint) {
|
||||
if *c < r {
|
||||
var new_size uint = *c
|
||||
if new_size == 0 {
|
||||
new_size = r
|
||||
}
|
||||
|
||||
for new_size < r {
|
||||
new_size *= 2
|
||||
}
|
||||
|
||||
if cap(*a) < int(new_size) {
|
||||
var new_array []byte = make([]byte, new_size)
|
||||
if *c != 0 {
|
||||
copy(new_array, (*a)[:*c])
|
||||
}
|
||||
|
||||
*a = new_array
|
||||
} else {
|
||||
*a = (*a)[:new_size]
|
||||
}
|
||||
|
||||
*c = new_size
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_ensure_capacity_uint32_t(a *[]uint32, c *uint, r uint) {
|
||||
var new_array []uint32
|
||||
if *c < r {
|
||||
var new_size uint = *c
|
||||
if new_size == 0 {
|
||||
new_size = r
|
||||
}
|
||||
|
||||
for new_size < r {
|
||||
new_size *= 2
|
||||
}
|
||||
|
||||
if cap(*a) < int(new_size) {
|
||||
new_array = make([]uint32, new_size)
|
||||
if *c != 0 {
|
||||
copy(new_array, (*a)[:*c])
|
||||
}
|
||||
|
||||
*a = new_array
|
||||
} else {
|
||||
*a = (*a)[:new_size]
|
||||
}
|
||||
*c = new_size
|
||||
}
|
||||
}
|
||||
+574
@@ -0,0 +1,574 @@
|
||||
package brotli
|
||||
|
||||
import (
|
||||
"sync"
|
||||
)
|
||||
|
||||
/* Copyright 2014 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Algorithms for distributing the literals and commands of a metablock between
|
||||
block types and contexts. */
|
||||
|
||||
type metaBlockSplit struct {
|
||||
literal_split blockSplit
|
||||
command_split blockSplit
|
||||
distance_split blockSplit
|
||||
literal_context_map []uint32
|
||||
literal_context_map_size uint
|
||||
distance_context_map []uint32
|
||||
distance_context_map_size uint
|
||||
literal_histograms []histogramLiteral
|
||||
literal_histograms_size uint
|
||||
command_histograms []histogramCommand
|
||||
command_histograms_size uint
|
||||
distance_histograms []histogramDistance
|
||||
distance_histograms_size uint
|
||||
}
|
||||
|
||||
var metaBlockPool sync.Pool
|
||||
|
||||
func getMetaBlockSplit() *metaBlockSplit {
|
||||
mb, _ := metaBlockPool.Get().(*metaBlockSplit)
|
||||
|
||||
if mb == nil {
|
||||
mb = &metaBlockSplit{}
|
||||
} else {
|
||||
initBlockSplit(&mb.literal_split)
|
||||
initBlockSplit(&mb.command_split)
|
||||
initBlockSplit(&mb.distance_split)
|
||||
mb.literal_context_map = mb.literal_context_map[:0]
|
||||
mb.literal_context_map_size = 0
|
||||
mb.distance_context_map = mb.distance_context_map[:0]
|
||||
mb.distance_context_map_size = 0
|
||||
mb.literal_histograms = mb.literal_histograms[:0]
|
||||
mb.command_histograms = mb.command_histograms[:0]
|
||||
mb.distance_histograms = mb.distance_histograms[:0]
|
||||
}
|
||||
return mb
|
||||
}
|
||||
|
||||
func freeMetaBlockSplit(mb *metaBlockSplit) {
|
||||
metaBlockPool.Put(mb)
|
||||
}
|
||||
|
||||
func initDistanceParams(params *encoderParams, npostfix uint32, ndirect uint32) {
|
||||
var dist_params *distanceParams = ¶ms.dist
|
||||
var alphabet_size uint32
|
||||
var max_distance uint32
|
||||
|
||||
dist_params.distance_postfix_bits = npostfix
|
||||
dist_params.num_direct_distance_codes = ndirect
|
||||
|
||||
alphabet_size = uint32(distanceAlphabetSize(uint(npostfix), uint(ndirect), maxDistanceBits))
|
||||
max_distance = ndirect + (1 << (maxDistanceBits + npostfix + 2)) - (1 << (npostfix + 2))
|
||||
|
||||
if params.large_window {
|
||||
var bound = [maxNpostfix + 1]uint32{0, 4, 12, 28}
|
||||
var postfix uint32 = 1 << npostfix
|
||||
alphabet_size = uint32(distanceAlphabetSize(uint(npostfix), uint(ndirect), largeMaxDistanceBits))
|
||||
|
||||
/* The maximum distance is set so that no distance symbol used can encode
|
||||
a distance larger than BROTLI_MAX_ALLOWED_DISTANCE with all
|
||||
its extra bits set. */
|
||||
if ndirect < bound[npostfix] {
|
||||
max_distance = maxAllowedDistance - (bound[npostfix] - ndirect)
|
||||
} else if ndirect >= bound[npostfix]+postfix {
|
||||
max_distance = (3 << 29) - 4 + (ndirect - bound[npostfix])
|
||||
} else {
|
||||
max_distance = maxAllowedDistance
|
||||
}
|
||||
}
|
||||
|
||||
dist_params.alphabet_size = alphabet_size
|
||||
dist_params.max_distance = uint(max_distance)
|
||||
}
|
||||
|
||||
func recomputeDistancePrefixes(cmds []command, orig_params *distanceParams, new_params *distanceParams) {
|
||||
if orig_params.distance_postfix_bits == new_params.distance_postfix_bits && orig_params.num_direct_distance_codes == new_params.num_direct_distance_codes {
|
||||
return
|
||||
}
|
||||
|
||||
for i := range cmds {
|
||||
var cmd *command = &cmds[i]
|
||||
if commandCopyLen(cmd) != 0 && cmd.cmd_prefix_ >= 128 {
|
||||
prefixEncodeCopyDistance(uint(commandRestoreDistanceCode(cmd, orig_params)), uint(new_params.num_direct_distance_codes), uint(new_params.distance_postfix_bits), &cmd.dist_prefix_, &cmd.dist_extra_)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func computeDistanceCost(cmds []command, orig_params *distanceParams, new_params *distanceParams, cost *float64) bool {
|
||||
var equal_params bool = false
|
||||
var dist_prefix uint16
|
||||
var dist_extra uint32
|
||||
var extra_bits float64 = 0.0
|
||||
var histo histogramDistance
|
||||
histogramClearDistance(&histo)
|
||||
|
||||
if orig_params.distance_postfix_bits == new_params.distance_postfix_bits && orig_params.num_direct_distance_codes == new_params.num_direct_distance_codes {
|
||||
equal_params = true
|
||||
}
|
||||
|
||||
for i := range cmds {
|
||||
cmd := &cmds[i]
|
||||
if commandCopyLen(cmd) != 0 && cmd.cmd_prefix_ >= 128 {
|
||||
if equal_params {
|
||||
dist_prefix = cmd.dist_prefix_
|
||||
} else {
|
||||
var distance uint32 = commandRestoreDistanceCode(cmd, orig_params)
|
||||
if distance > uint32(new_params.max_distance) {
|
||||
return false
|
||||
}
|
||||
|
||||
prefixEncodeCopyDistance(uint(distance), uint(new_params.num_direct_distance_codes), uint(new_params.distance_postfix_bits), &dist_prefix, &dist_extra)
|
||||
}
|
||||
|
||||
histogramAddDistance(&histo, uint(dist_prefix)&0x3FF)
|
||||
extra_bits += float64(dist_prefix >> 10)
|
||||
}
|
||||
}
|
||||
|
||||
*cost = populationCostDistance(&histo) + extra_bits
|
||||
return true
|
||||
}
|
||||
|
||||
var buildMetaBlock_kMaxNumberOfHistograms uint = 256
|
||||
|
||||
func buildMetaBlock(ringbuffer []byte, pos uint, mask uint, params *encoderParams, prev_byte byte, prev_byte2 byte, cmds []command, literal_context_mode int, mb *metaBlockSplit) {
|
||||
var distance_histograms []histogramDistance
|
||||
var literal_histograms []histogramLiteral
|
||||
var literal_context_modes []int = nil
|
||||
var literal_histograms_size uint
|
||||
var distance_histograms_size uint
|
||||
var i uint
|
||||
var literal_context_multiplier uint = 1
|
||||
var npostfix uint32
|
||||
var ndirect_msb uint32 = 0
|
||||
var check_orig bool = true
|
||||
var best_dist_cost float64 = 1e99
|
||||
var orig_params encoderParams = *params
|
||||
/* Histogram ids need to fit in one byte. */
|
||||
|
||||
var new_params encoderParams = *params
|
||||
|
||||
for npostfix = 0; npostfix <= maxNpostfix; npostfix++ {
|
||||
for ; ndirect_msb < 16; ndirect_msb++ {
|
||||
var ndirect uint32 = ndirect_msb << npostfix
|
||||
var skip bool
|
||||
var dist_cost float64
|
||||
initDistanceParams(&new_params, npostfix, ndirect)
|
||||
if npostfix == orig_params.dist.distance_postfix_bits && ndirect == orig_params.dist.num_direct_distance_codes {
|
||||
check_orig = false
|
||||
}
|
||||
|
||||
skip = !computeDistanceCost(cmds, &orig_params.dist, &new_params.dist, &dist_cost)
|
||||
if skip || (dist_cost > best_dist_cost) {
|
||||
break
|
||||
}
|
||||
|
||||
best_dist_cost = dist_cost
|
||||
params.dist = new_params.dist
|
||||
}
|
||||
|
||||
if ndirect_msb > 0 {
|
||||
ndirect_msb--
|
||||
}
|
||||
ndirect_msb /= 2
|
||||
}
|
||||
|
||||
if check_orig {
|
||||
var dist_cost float64
|
||||
computeDistanceCost(cmds, &orig_params.dist, &orig_params.dist, &dist_cost)
|
||||
if dist_cost < best_dist_cost {
|
||||
/* NB: currently unused; uncomment when more param tuning is added. */
|
||||
/* best_dist_cost = dist_cost; */
|
||||
params.dist = orig_params.dist
|
||||
}
|
||||
}
|
||||
|
||||
recomputeDistancePrefixes(cmds, &orig_params.dist, ¶ms.dist)
|
||||
|
||||
splitBlock(cmds, ringbuffer, pos, mask, params, &mb.literal_split, &mb.command_split, &mb.distance_split)
|
||||
|
||||
if !params.disable_literal_context_modeling {
|
||||
literal_context_multiplier = 1 << literalContextBits
|
||||
literal_context_modes = make([]int, (mb.literal_split.num_types))
|
||||
for i = 0; i < mb.literal_split.num_types; i++ {
|
||||
literal_context_modes[i] = literal_context_mode
|
||||
}
|
||||
}
|
||||
|
||||
literal_histograms_size = mb.literal_split.num_types * literal_context_multiplier
|
||||
literal_histograms = make([]histogramLiteral, literal_histograms_size)
|
||||
clearHistogramsLiteral(literal_histograms, literal_histograms_size)
|
||||
|
||||
distance_histograms_size = mb.distance_split.num_types << distanceContextBits
|
||||
distance_histograms = make([]histogramDistance, distance_histograms_size)
|
||||
clearHistogramsDistance(distance_histograms, distance_histograms_size)
|
||||
|
||||
mb.command_histograms_size = mb.command_split.num_types
|
||||
if cap(mb.command_histograms) < int(mb.command_histograms_size) {
|
||||
mb.command_histograms = make([]histogramCommand, (mb.command_histograms_size))
|
||||
} else {
|
||||
mb.command_histograms = mb.command_histograms[:mb.command_histograms_size]
|
||||
}
|
||||
clearHistogramsCommand(mb.command_histograms, mb.command_histograms_size)
|
||||
|
||||
buildHistogramsWithContext(cmds, &mb.literal_split, &mb.command_split, &mb.distance_split, ringbuffer, pos, mask, prev_byte, prev_byte2, literal_context_modes, literal_histograms, mb.command_histograms, distance_histograms)
|
||||
literal_context_modes = nil
|
||||
|
||||
mb.literal_context_map_size = mb.literal_split.num_types << literalContextBits
|
||||
if cap(mb.literal_context_map) < int(mb.literal_context_map_size) {
|
||||
mb.literal_context_map = make([]uint32, (mb.literal_context_map_size))
|
||||
} else {
|
||||
mb.literal_context_map = mb.literal_context_map[:mb.literal_context_map_size]
|
||||
}
|
||||
|
||||
mb.literal_histograms_size = mb.literal_context_map_size
|
||||
if cap(mb.literal_histograms) < int(mb.literal_histograms_size) {
|
||||
mb.literal_histograms = make([]histogramLiteral, (mb.literal_histograms_size))
|
||||
} else {
|
||||
mb.literal_histograms = mb.literal_histograms[:mb.literal_histograms_size]
|
||||
}
|
||||
|
||||
clusterHistogramsLiteral(literal_histograms, literal_histograms_size, buildMetaBlock_kMaxNumberOfHistograms, mb.literal_histograms, &mb.literal_histograms_size, mb.literal_context_map)
|
||||
literal_histograms = nil
|
||||
|
||||
if params.disable_literal_context_modeling {
|
||||
/* Distribute assignment to all contexts. */
|
||||
for i = mb.literal_split.num_types; i != 0; {
|
||||
var j uint = 0
|
||||
i--
|
||||
for ; j < 1<<literalContextBits; j++ {
|
||||
mb.literal_context_map[(i<<literalContextBits)+j] = mb.literal_context_map[i]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
mb.distance_context_map_size = mb.distance_split.num_types << distanceContextBits
|
||||
if cap(mb.distance_context_map) < int(mb.distance_context_map_size) {
|
||||
mb.distance_context_map = make([]uint32, (mb.distance_context_map_size))
|
||||
} else {
|
||||
mb.distance_context_map = mb.distance_context_map[:mb.distance_context_map_size]
|
||||
}
|
||||
|
||||
mb.distance_histograms_size = mb.distance_context_map_size
|
||||
if cap(mb.distance_histograms) < int(mb.distance_histograms_size) {
|
||||
mb.distance_histograms = make([]histogramDistance, (mb.distance_histograms_size))
|
||||
} else {
|
||||
mb.distance_histograms = mb.distance_histograms[:mb.distance_histograms_size]
|
||||
}
|
||||
|
||||
clusterHistogramsDistance(distance_histograms, mb.distance_context_map_size, buildMetaBlock_kMaxNumberOfHistograms, mb.distance_histograms, &mb.distance_histograms_size, mb.distance_context_map)
|
||||
distance_histograms = nil
|
||||
}
|
||||
|
||||
const maxStaticContexts = 13
|
||||
|
||||
/* Greedy block splitter for one block category (literal, command or distance).
|
||||
Gathers histograms for all context buckets. */
|
||||
type contextBlockSplitter struct {
|
||||
alphabet_size_ uint
|
||||
num_contexts_ uint
|
||||
max_block_types_ uint
|
||||
min_block_size_ uint
|
||||
split_threshold_ float64
|
||||
num_blocks_ uint
|
||||
split_ *blockSplit
|
||||
histograms_ []histogramLiteral
|
||||
histograms_size_ *uint
|
||||
target_block_size_ uint
|
||||
block_size_ uint
|
||||
curr_histogram_ix_ uint
|
||||
last_histogram_ix_ [2]uint
|
||||
last_entropy_ [2 * maxStaticContexts]float64
|
||||
merge_last_count_ uint
|
||||
}
|
||||
|
||||
func initContextBlockSplitter(self *contextBlockSplitter, alphabet_size uint, num_contexts uint, min_block_size uint, split_threshold float64, num_symbols uint, split *blockSplit, histograms *[]histogramLiteral, histograms_size *uint) {
|
||||
var max_num_blocks uint = num_symbols/min_block_size + 1
|
||||
var max_num_types uint
|
||||
assert(num_contexts <= maxStaticContexts)
|
||||
|
||||
self.alphabet_size_ = alphabet_size
|
||||
self.num_contexts_ = num_contexts
|
||||
self.max_block_types_ = maxNumberOfBlockTypes / num_contexts
|
||||
self.min_block_size_ = min_block_size
|
||||
self.split_threshold_ = split_threshold
|
||||
self.num_blocks_ = 0
|
||||
self.split_ = split
|
||||
self.histograms_size_ = histograms_size
|
||||
self.target_block_size_ = min_block_size
|
||||
self.block_size_ = 0
|
||||
self.curr_histogram_ix_ = 0
|
||||
self.merge_last_count_ = 0
|
||||
|
||||
/* We have to allocate one more histogram than the maximum number of block
|
||||
types for the current histogram when the meta-block is too big. */
|
||||
max_num_types = brotli_min_size_t(max_num_blocks, self.max_block_types_+1)
|
||||
|
||||
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, max_num_blocks)
|
||||
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, max_num_blocks)
|
||||
split.num_blocks = max_num_blocks
|
||||
*histograms_size = max_num_types * num_contexts
|
||||
if histograms == nil || cap(*histograms) < int(*histograms_size) {
|
||||
*histograms = make([]histogramLiteral, (*histograms_size))
|
||||
} else {
|
||||
*histograms = (*histograms)[:*histograms_size]
|
||||
}
|
||||
self.histograms_ = *histograms
|
||||
|
||||
/* Clear only current histogram. */
|
||||
clearHistogramsLiteral(self.histograms_[0:], num_contexts)
|
||||
|
||||
self.last_histogram_ix_[1] = 0
|
||||
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
|
||||
}
|
||||
|
||||
/* Does either of three things:
|
||||
(1) emits the current block with a new block type;
|
||||
(2) emits the current block with the type of the second last block;
|
||||
(3) merges the current block with the last block. */
|
||||
func contextBlockSplitterFinishBlock(self *contextBlockSplitter, is_final bool) {
|
||||
var split *blockSplit = self.split_
|
||||
var num_contexts uint = self.num_contexts_
|
||||
var last_entropy []float64 = self.last_entropy_[:]
|
||||
var histograms []histogramLiteral = self.histograms_
|
||||
|
||||
if self.block_size_ < self.min_block_size_ {
|
||||
self.block_size_ = self.min_block_size_
|
||||
}
|
||||
|
||||
if self.num_blocks_ == 0 {
|
||||
var i uint
|
||||
|
||||
/* Create first block. */
|
||||
split.lengths[0] = uint32(self.block_size_)
|
||||
|
||||
split.types[0] = 0
|
||||
|
||||
for i = 0; i < num_contexts; i++ {
|
||||
last_entropy[i] = bitsEntropy(histograms[i].data_[:], self.alphabet_size_)
|
||||
last_entropy[num_contexts+i] = last_entropy[i]
|
||||
}
|
||||
|
||||
self.num_blocks_++
|
||||
split.num_types++
|
||||
self.curr_histogram_ix_ += num_contexts
|
||||
if self.curr_histogram_ix_ < *self.histograms_size_ {
|
||||
clearHistogramsLiteral(self.histograms_[self.curr_histogram_ix_:], self.num_contexts_)
|
||||
}
|
||||
|
||||
self.block_size_ = 0
|
||||
} else if self.block_size_ > 0 {
|
||||
var entropy [maxStaticContexts]float64
|
||||
var combined_histo []histogramLiteral = make([]histogramLiteral, (2 * num_contexts))
|
||||
var combined_entropy [2 * maxStaticContexts]float64
|
||||
var diff = [2]float64{0.0}
|
||||
/* Try merging the set of histograms for the current block type with the
|
||||
respective set of histograms for the last and second last block types.
|
||||
Decide over the split based on the total reduction of entropy across
|
||||
all contexts. */
|
||||
|
||||
var i uint
|
||||
for i = 0; i < num_contexts; i++ {
|
||||
var curr_histo_ix uint = self.curr_histogram_ix_ + i
|
||||
var j uint
|
||||
entropy[i] = bitsEntropy(histograms[curr_histo_ix].data_[:], self.alphabet_size_)
|
||||
for j = 0; j < 2; j++ {
|
||||
var jx uint = j*num_contexts + i
|
||||
var last_histogram_ix uint = self.last_histogram_ix_[j] + i
|
||||
combined_histo[jx] = histograms[curr_histo_ix]
|
||||
histogramAddHistogramLiteral(&combined_histo[jx], &histograms[last_histogram_ix])
|
||||
combined_entropy[jx] = bitsEntropy(combined_histo[jx].data_[0:], self.alphabet_size_)
|
||||
diff[j] += combined_entropy[jx] - entropy[i] - last_entropy[jx]
|
||||
}
|
||||
}
|
||||
|
||||
if split.num_types < self.max_block_types_ && diff[0] > self.split_threshold_ && diff[1] > self.split_threshold_ {
|
||||
/* Create new block. */
|
||||
split.lengths[self.num_blocks_] = uint32(self.block_size_)
|
||||
|
||||
split.types[self.num_blocks_] = byte(split.num_types)
|
||||
self.last_histogram_ix_[1] = self.last_histogram_ix_[0]
|
||||
self.last_histogram_ix_[0] = split.num_types * num_contexts
|
||||
for i = 0; i < num_contexts; i++ {
|
||||
last_entropy[num_contexts+i] = last_entropy[i]
|
||||
last_entropy[i] = entropy[i]
|
||||
}
|
||||
|
||||
self.num_blocks_++
|
||||
split.num_types++
|
||||
self.curr_histogram_ix_ += num_contexts
|
||||
if self.curr_histogram_ix_ < *self.histograms_size_ {
|
||||
clearHistogramsLiteral(self.histograms_[self.curr_histogram_ix_:], self.num_contexts_)
|
||||
}
|
||||
|
||||
self.block_size_ = 0
|
||||
self.merge_last_count_ = 0
|
||||
self.target_block_size_ = self.min_block_size_
|
||||
} else if diff[1] < diff[0]-20.0 {
|
||||
split.lengths[self.num_blocks_] = uint32(self.block_size_)
|
||||
split.types[self.num_blocks_] = split.types[self.num_blocks_-2]
|
||||
/* Combine this block with second last block. */
|
||||
|
||||
var tmp uint = self.last_histogram_ix_[0]
|
||||
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
|
||||
self.last_histogram_ix_[1] = tmp
|
||||
for i = 0; i < num_contexts; i++ {
|
||||
histograms[self.last_histogram_ix_[0]+i] = combined_histo[num_contexts+i]
|
||||
last_entropy[num_contexts+i] = last_entropy[i]
|
||||
last_entropy[i] = combined_entropy[num_contexts+i]
|
||||
histogramClearLiteral(&histograms[self.curr_histogram_ix_+i])
|
||||
}
|
||||
|
||||
self.num_blocks_++
|
||||
self.block_size_ = 0
|
||||
self.merge_last_count_ = 0
|
||||
self.target_block_size_ = self.min_block_size_
|
||||
} else {
|
||||
/* Combine this block with last block. */
|
||||
split.lengths[self.num_blocks_-1] += uint32(self.block_size_)
|
||||
|
||||
for i = 0; i < num_contexts; i++ {
|
||||
histograms[self.last_histogram_ix_[0]+i] = combined_histo[i]
|
||||
last_entropy[i] = combined_entropy[i]
|
||||
if split.num_types == 1 {
|
||||
last_entropy[num_contexts+i] = last_entropy[i]
|
||||
}
|
||||
|
||||
histogramClearLiteral(&histograms[self.curr_histogram_ix_+i])
|
||||
}
|
||||
|
||||
self.block_size_ = 0
|
||||
self.merge_last_count_++
|
||||
if self.merge_last_count_ > 1 {
|
||||
self.target_block_size_ += self.min_block_size_
|
||||
}
|
||||
}
|
||||
|
||||
combined_histo = nil
|
||||
}
|
||||
|
||||
if is_final {
|
||||
*self.histograms_size_ = split.num_types * num_contexts
|
||||
split.num_blocks = self.num_blocks_
|
||||
}
|
||||
}
|
||||
|
||||
/* Adds the next symbol to the current block type and context. When the
|
||||
current block reaches the target size, decides on merging the block. */
|
||||
func contextBlockSplitterAddSymbol(self *contextBlockSplitter, symbol uint, context uint) {
|
||||
histogramAddLiteral(&self.histograms_[self.curr_histogram_ix_+context], symbol)
|
||||
self.block_size_++
|
||||
if self.block_size_ == self.target_block_size_ {
|
||||
contextBlockSplitterFinishBlock(self, false) /* is_final = */
|
||||
}
|
||||
}
|
||||
|
||||
func mapStaticContexts(num_contexts uint, static_context_map []uint32, mb *metaBlockSplit) {
|
||||
var i uint
|
||||
mb.literal_context_map_size = mb.literal_split.num_types << literalContextBits
|
||||
if cap(mb.literal_context_map) < int(mb.literal_context_map_size) {
|
||||
mb.literal_context_map = make([]uint32, (mb.literal_context_map_size))
|
||||
} else {
|
||||
mb.literal_context_map = mb.literal_context_map[:mb.literal_context_map_size]
|
||||
}
|
||||
|
||||
for i = 0; i < mb.literal_split.num_types; i++ {
|
||||
var offset uint32 = uint32(i * num_contexts)
|
||||
var j uint
|
||||
for j = 0; j < 1<<literalContextBits; j++ {
|
||||
mb.literal_context_map[(i<<literalContextBits)+j] = offset + static_context_map[j]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func buildMetaBlockGreedyInternal(ringbuffer []byte, pos uint, mask uint, prev_byte byte, prev_byte2 byte, literal_context_lut contextLUT, num_contexts uint, static_context_map []uint32, commands []command, mb *metaBlockSplit) {
|
||||
var lit_blocks struct {
|
||||
plain blockSplitterLiteral
|
||||
ctx contextBlockSplitter
|
||||
}
|
||||
var cmd_blocks blockSplitterCommand
|
||||
var dist_blocks blockSplitterDistance
|
||||
var num_literals uint = 0
|
||||
for i := range commands {
|
||||
num_literals += uint(commands[i].insert_len_)
|
||||
}
|
||||
|
||||
if num_contexts == 1 {
|
||||
initBlockSplitterLiteral(&lit_blocks.plain, 256, 512, 400.0, num_literals, &mb.literal_split, &mb.literal_histograms, &mb.literal_histograms_size)
|
||||
} else {
|
||||
initContextBlockSplitter(&lit_blocks.ctx, 256, num_contexts, 512, 400.0, num_literals, &mb.literal_split, &mb.literal_histograms, &mb.literal_histograms_size)
|
||||
}
|
||||
|
||||
initBlockSplitterCommand(&cmd_blocks, numCommandSymbols, 1024, 500.0, uint(len(commands)), &mb.command_split, &mb.command_histograms, &mb.command_histograms_size)
|
||||
initBlockSplitterDistance(&dist_blocks, 64, 512, 100.0, uint(len(commands)), &mb.distance_split, &mb.distance_histograms, &mb.distance_histograms_size)
|
||||
|
||||
for _, cmd := range commands {
|
||||
var j uint
|
||||
blockSplitterAddSymbolCommand(&cmd_blocks, uint(cmd.cmd_prefix_))
|
||||
for j = uint(cmd.insert_len_); j != 0; j-- {
|
||||
var literal byte = ringbuffer[pos&mask]
|
||||
if num_contexts == 1 {
|
||||
blockSplitterAddSymbolLiteral(&lit_blocks.plain, uint(literal))
|
||||
} else {
|
||||
var context uint = uint(getContext(prev_byte, prev_byte2, literal_context_lut))
|
||||
contextBlockSplitterAddSymbol(&lit_blocks.ctx, uint(literal), uint(static_context_map[context]))
|
||||
}
|
||||
|
||||
prev_byte2 = prev_byte
|
||||
prev_byte = literal
|
||||
pos++
|
||||
}
|
||||
|
||||
pos += uint(commandCopyLen(&cmd))
|
||||
if commandCopyLen(&cmd) != 0 {
|
||||
prev_byte2 = ringbuffer[(pos-2)&mask]
|
||||
prev_byte = ringbuffer[(pos-1)&mask]
|
||||
if cmd.cmd_prefix_ >= 128 {
|
||||
blockSplitterAddSymbolDistance(&dist_blocks, uint(cmd.dist_prefix_)&0x3FF)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if num_contexts == 1 {
|
||||
blockSplitterFinishBlockLiteral(&lit_blocks.plain, true) /* is_final = */
|
||||
} else {
|
||||
contextBlockSplitterFinishBlock(&lit_blocks.ctx, true) /* is_final = */
|
||||
}
|
||||
|
||||
blockSplitterFinishBlockCommand(&cmd_blocks, true) /* is_final = */
|
||||
blockSplitterFinishBlockDistance(&dist_blocks, true) /* is_final = */
|
||||
|
||||
if num_contexts > 1 {
|
||||
mapStaticContexts(num_contexts, static_context_map, mb)
|
||||
}
|
||||
}
|
||||
|
||||
func buildMetaBlockGreedy(ringbuffer []byte, pos uint, mask uint, prev_byte byte, prev_byte2 byte, literal_context_lut contextLUT, num_contexts uint, static_context_map []uint32, commands []command, mb *metaBlockSplit) {
|
||||
if num_contexts == 1 {
|
||||
buildMetaBlockGreedyInternal(ringbuffer, pos, mask, prev_byte, prev_byte2, literal_context_lut, 1, nil, commands, mb)
|
||||
} else {
|
||||
buildMetaBlockGreedyInternal(ringbuffer, pos, mask, prev_byte, prev_byte2, literal_context_lut, num_contexts, static_context_map, commands, mb)
|
||||
}
|
||||
}
|
||||
|
||||
func optimizeHistograms(num_distance_codes uint32, mb *metaBlockSplit) {
|
||||
var good_for_rle [numCommandSymbols]byte
|
||||
var i uint
|
||||
for i = 0; i < mb.literal_histograms_size; i++ {
|
||||
optimizeHuffmanCountsForRLE(256, mb.literal_histograms[i].data_[:], good_for_rle[:])
|
||||
}
|
||||
|
||||
for i = 0; i < mb.command_histograms_size; i++ {
|
||||
optimizeHuffmanCountsForRLE(numCommandSymbols, mb.command_histograms[i].data_[:], good_for_rle[:])
|
||||
}
|
||||
|
||||
for i = 0; i < mb.distance_histograms_size; i++ {
|
||||
optimizeHuffmanCountsForRLE(uint(num_distance_codes), mb.distance_histograms[i].data_[:], good_for_rle[:])
|
||||
}
|
||||
}
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2015 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Greedy block splitter for one block category (literal, command or distance).
|
||||
*/
|
||||
type blockSplitterCommand struct {
|
||||
alphabet_size_ uint
|
||||
min_block_size_ uint
|
||||
split_threshold_ float64
|
||||
num_blocks_ uint
|
||||
split_ *blockSplit
|
||||
histograms_ []histogramCommand
|
||||
histograms_size_ *uint
|
||||
target_block_size_ uint
|
||||
block_size_ uint
|
||||
curr_histogram_ix_ uint
|
||||
last_histogram_ix_ [2]uint
|
||||
last_entropy_ [2]float64
|
||||
merge_last_count_ uint
|
||||
}
|
||||
|
||||
func initBlockSplitterCommand(self *blockSplitterCommand, alphabet_size uint, min_block_size uint, split_threshold float64, num_symbols uint, split *blockSplit, histograms *[]histogramCommand, histograms_size *uint) {
|
||||
var max_num_blocks uint = num_symbols/min_block_size + 1
|
||||
var max_num_types uint = brotli_min_size_t(max_num_blocks, maxNumberOfBlockTypes+1)
|
||||
/* We have to allocate one more histogram than the maximum number of block
|
||||
types for the current histogram when the meta-block is too big. */
|
||||
self.alphabet_size_ = alphabet_size
|
||||
|
||||
self.min_block_size_ = min_block_size
|
||||
self.split_threshold_ = split_threshold
|
||||
self.num_blocks_ = 0
|
||||
self.split_ = split
|
||||
self.histograms_size_ = histograms_size
|
||||
self.target_block_size_ = min_block_size
|
||||
self.block_size_ = 0
|
||||
self.curr_histogram_ix_ = 0
|
||||
self.merge_last_count_ = 0
|
||||
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, max_num_blocks)
|
||||
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, max_num_blocks)
|
||||
self.split_.num_blocks = max_num_blocks
|
||||
*histograms_size = max_num_types
|
||||
if histograms == nil || cap(*histograms) < int(*histograms_size) {
|
||||
*histograms = make([]histogramCommand, (*histograms_size))
|
||||
} else {
|
||||
*histograms = (*histograms)[:*histograms_size]
|
||||
}
|
||||
self.histograms_ = *histograms
|
||||
|
||||
/* Clear only current histogram. */
|
||||
histogramClearCommand(&self.histograms_[0])
|
||||
|
||||
self.last_histogram_ix_[1] = 0
|
||||
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
|
||||
}
|
||||
|
||||
/* Does either of three things:
|
||||
(1) emits the current block with a new block type;
|
||||
(2) emits the current block with the type of the second last block;
|
||||
(3) merges the current block with the last block. */
|
||||
func blockSplitterFinishBlockCommand(self *blockSplitterCommand, is_final bool) {
|
||||
var split *blockSplit = self.split_
|
||||
var last_entropy []float64 = self.last_entropy_[:]
|
||||
var histograms []histogramCommand = self.histograms_
|
||||
self.block_size_ = brotli_max_size_t(self.block_size_, self.min_block_size_)
|
||||
if self.num_blocks_ == 0 {
|
||||
/* Create first block. */
|
||||
split.lengths[0] = uint32(self.block_size_)
|
||||
|
||||
split.types[0] = 0
|
||||
last_entropy[0] = bitsEntropy(histograms[0].data_[:], self.alphabet_size_)
|
||||
last_entropy[1] = last_entropy[0]
|
||||
self.num_blocks_++
|
||||
split.num_types++
|
||||
self.curr_histogram_ix_++
|
||||
if self.curr_histogram_ix_ < *self.histograms_size_ {
|
||||
histogramClearCommand(&histograms[self.curr_histogram_ix_])
|
||||
}
|
||||
self.block_size_ = 0
|
||||
} else if self.block_size_ > 0 {
|
||||
var entropy float64 = bitsEntropy(histograms[self.curr_histogram_ix_].data_[:], self.alphabet_size_)
|
||||
var combined_histo [2]histogramCommand
|
||||
var combined_entropy [2]float64
|
||||
var diff [2]float64
|
||||
var j uint
|
||||
for j = 0; j < 2; j++ {
|
||||
var last_histogram_ix uint = self.last_histogram_ix_[j]
|
||||
combined_histo[j] = histograms[self.curr_histogram_ix_]
|
||||
histogramAddHistogramCommand(&combined_histo[j], &histograms[last_histogram_ix])
|
||||
combined_entropy[j] = bitsEntropy(combined_histo[j].data_[0:], self.alphabet_size_)
|
||||
diff[j] = combined_entropy[j] - entropy - last_entropy[j]
|
||||
}
|
||||
|
||||
if split.num_types < maxNumberOfBlockTypes && diff[0] > self.split_threshold_ && diff[1] > self.split_threshold_ {
|
||||
/* Create new block. */
|
||||
split.lengths[self.num_blocks_] = uint32(self.block_size_)
|
||||
|
||||
split.types[self.num_blocks_] = byte(split.num_types)
|
||||
self.last_histogram_ix_[1] = self.last_histogram_ix_[0]
|
||||
self.last_histogram_ix_[0] = uint(byte(split.num_types))
|
||||
last_entropy[1] = last_entropy[0]
|
||||
last_entropy[0] = entropy
|
||||
self.num_blocks_++
|
||||
split.num_types++
|
||||
self.curr_histogram_ix_++
|
||||
if self.curr_histogram_ix_ < *self.histograms_size_ {
|
||||
histogramClearCommand(&histograms[self.curr_histogram_ix_])
|
||||
}
|
||||
self.block_size_ = 0
|
||||
self.merge_last_count_ = 0
|
||||
self.target_block_size_ = self.min_block_size_
|
||||
} else if diff[1] < diff[0]-20.0 {
|
||||
split.lengths[self.num_blocks_] = uint32(self.block_size_)
|
||||
split.types[self.num_blocks_] = split.types[self.num_blocks_-2]
|
||||
/* Combine this block with second last block. */
|
||||
|
||||
var tmp uint = self.last_histogram_ix_[0]
|
||||
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
|
||||
self.last_histogram_ix_[1] = tmp
|
||||
histograms[self.last_histogram_ix_[0]] = combined_histo[1]
|
||||
last_entropy[1] = last_entropy[0]
|
||||
last_entropy[0] = combined_entropy[1]
|
||||
self.num_blocks_++
|
||||
self.block_size_ = 0
|
||||
histogramClearCommand(&histograms[self.curr_histogram_ix_])
|
||||
self.merge_last_count_ = 0
|
||||
self.target_block_size_ = self.min_block_size_
|
||||
} else {
|
||||
/* Combine this block with last block. */
|
||||
split.lengths[self.num_blocks_-1] += uint32(self.block_size_)
|
||||
|
||||
histograms[self.last_histogram_ix_[0]] = combined_histo[0]
|
||||
last_entropy[0] = combined_entropy[0]
|
||||
if split.num_types == 1 {
|
||||
last_entropy[1] = last_entropy[0]
|
||||
}
|
||||
|
||||
self.block_size_ = 0
|
||||
histogramClearCommand(&histograms[self.curr_histogram_ix_])
|
||||
self.merge_last_count_++
|
||||
if self.merge_last_count_ > 1 {
|
||||
self.target_block_size_ += self.min_block_size_
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if is_final {
|
||||
*self.histograms_size_ = split.num_types
|
||||
split.num_blocks = self.num_blocks_
|
||||
}
|
||||
}
|
||||
|
||||
/* Adds the next symbol to the current histogram. When the current histogram
|
||||
reaches the target size, decides on merging the block. */
|
||||
func blockSplitterAddSymbolCommand(self *blockSplitterCommand, symbol uint) {
|
||||
histogramAddCommand(&self.histograms_[self.curr_histogram_ix_], symbol)
|
||||
self.block_size_++
|
||||
if self.block_size_ == self.target_block_size_ {
|
||||
blockSplitterFinishBlockCommand(self, false) /* is_final = */
|
||||
}
|
||||
}
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2015 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Greedy block splitter for one block category (literal, command or distance).
|
||||
*/
|
||||
type blockSplitterDistance struct {
|
||||
alphabet_size_ uint
|
||||
min_block_size_ uint
|
||||
split_threshold_ float64
|
||||
num_blocks_ uint
|
||||
split_ *blockSplit
|
||||
histograms_ []histogramDistance
|
||||
histograms_size_ *uint
|
||||
target_block_size_ uint
|
||||
block_size_ uint
|
||||
curr_histogram_ix_ uint
|
||||
last_histogram_ix_ [2]uint
|
||||
last_entropy_ [2]float64
|
||||
merge_last_count_ uint
|
||||
}
|
||||
|
||||
func initBlockSplitterDistance(self *blockSplitterDistance, alphabet_size uint, min_block_size uint, split_threshold float64, num_symbols uint, split *blockSplit, histograms *[]histogramDistance, histograms_size *uint) {
|
||||
var max_num_blocks uint = num_symbols/min_block_size + 1
|
||||
var max_num_types uint = brotli_min_size_t(max_num_blocks, maxNumberOfBlockTypes+1)
|
||||
/* We have to allocate one more histogram than the maximum number of block
|
||||
types for the current histogram when the meta-block is too big. */
|
||||
self.alphabet_size_ = alphabet_size
|
||||
|
||||
self.min_block_size_ = min_block_size
|
||||
self.split_threshold_ = split_threshold
|
||||
self.num_blocks_ = 0
|
||||
self.split_ = split
|
||||
self.histograms_size_ = histograms_size
|
||||
self.target_block_size_ = min_block_size
|
||||
self.block_size_ = 0
|
||||
self.curr_histogram_ix_ = 0
|
||||
self.merge_last_count_ = 0
|
||||
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, max_num_blocks)
|
||||
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, max_num_blocks)
|
||||
self.split_.num_blocks = max_num_blocks
|
||||
*histograms_size = max_num_types
|
||||
if histograms == nil || cap(*histograms) < int(*histograms_size) {
|
||||
*histograms = make([]histogramDistance, *histograms_size)
|
||||
} else {
|
||||
*histograms = (*histograms)[:*histograms_size]
|
||||
}
|
||||
self.histograms_ = *histograms
|
||||
|
||||
/* Clear only current histogram. */
|
||||
histogramClearDistance(&self.histograms_[0])
|
||||
|
||||
self.last_histogram_ix_[1] = 0
|
||||
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
|
||||
}
|
||||
|
||||
/* Does either of three things:
|
||||
(1) emits the current block with a new block type;
|
||||
(2) emits the current block with the type of the second last block;
|
||||
(3) merges the current block with the last block. */
|
||||
func blockSplitterFinishBlockDistance(self *blockSplitterDistance, is_final bool) {
|
||||
var split *blockSplit = self.split_
|
||||
var last_entropy []float64 = self.last_entropy_[:]
|
||||
var histograms []histogramDistance = self.histograms_
|
||||
self.block_size_ = brotli_max_size_t(self.block_size_, self.min_block_size_)
|
||||
if self.num_blocks_ == 0 {
|
||||
/* Create first block. */
|
||||
split.lengths[0] = uint32(self.block_size_)
|
||||
|
||||
split.types[0] = 0
|
||||
last_entropy[0] = bitsEntropy(histograms[0].data_[:], self.alphabet_size_)
|
||||
last_entropy[1] = last_entropy[0]
|
||||
self.num_blocks_++
|
||||
split.num_types++
|
||||
self.curr_histogram_ix_++
|
||||
if self.curr_histogram_ix_ < *self.histograms_size_ {
|
||||
histogramClearDistance(&histograms[self.curr_histogram_ix_])
|
||||
}
|
||||
self.block_size_ = 0
|
||||
} else if self.block_size_ > 0 {
|
||||
var entropy float64 = bitsEntropy(histograms[self.curr_histogram_ix_].data_[:], self.alphabet_size_)
|
||||
var combined_histo [2]histogramDistance
|
||||
var combined_entropy [2]float64
|
||||
var diff [2]float64
|
||||
var j uint
|
||||
for j = 0; j < 2; j++ {
|
||||
var last_histogram_ix uint = self.last_histogram_ix_[j]
|
||||
combined_histo[j] = histograms[self.curr_histogram_ix_]
|
||||
histogramAddHistogramDistance(&combined_histo[j], &histograms[last_histogram_ix])
|
||||
combined_entropy[j] = bitsEntropy(combined_histo[j].data_[0:], self.alphabet_size_)
|
||||
diff[j] = combined_entropy[j] - entropy - last_entropy[j]
|
||||
}
|
||||
|
||||
if split.num_types < maxNumberOfBlockTypes && diff[0] > self.split_threshold_ && diff[1] > self.split_threshold_ {
|
||||
/* Create new block. */
|
||||
split.lengths[self.num_blocks_] = uint32(self.block_size_)
|
||||
|
||||
split.types[self.num_blocks_] = byte(split.num_types)
|
||||
self.last_histogram_ix_[1] = self.last_histogram_ix_[0]
|
||||
self.last_histogram_ix_[0] = uint(byte(split.num_types))
|
||||
last_entropy[1] = last_entropy[0]
|
||||
last_entropy[0] = entropy
|
||||
self.num_blocks_++
|
||||
split.num_types++
|
||||
self.curr_histogram_ix_++
|
||||
if self.curr_histogram_ix_ < *self.histograms_size_ {
|
||||
histogramClearDistance(&histograms[self.curr_histogram_ix_])
|
||||
}
|
||||
self.block_size_ = 0
|
||||
self.merge_last_count_ = 0
|
||||
self.target_block_size_ = self.min_block_size_
|
||||
} else if diff[1] < diff[0]-20.0 {
|
||||
split.lengths[self.num_blocks_] = uint32(self.block_size_)
|
||||
split.types[self.num_blocks_] = split.types[self.num_blocks_-2]
|
||||
/* Combine this block with second last block. */
|
||||
|
||||
var tmp uint = self.last_histogram_ix_[0]
|
||||
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
|
||||
self.last_histogram_ix_[1] = tmp
|
||||
histograms[self.last_histogram_ix_[0]] = combined_histo[1]
|
||||
last_entropy[1] = last_entropy[0]
|
||||
last_entropy[0] = combined_entropy[1]
|
||||
self.num_blocks_++
|
||||
self.block_size_ = 0
|
||||
histogramClearDistance(&histograms[self.curr_histogram_ix_])
|
||||
self.merge_last_count_ = 0
|
||||
self.target_block_size_ = self.min_block_size_
|
||||
} else {
|
||||
/* Combine this block with last block. */
|
||||
split.lengths[self.num_blocks_-1] += uint32(self.block_size_)
|
||||
|
||||
histograms[self.last_histogram_ix_[0]] = combined_histo[0]
|
||||
last_entropy[0] = combined_entropy[0]
|
||||
if split.num_types == 1 {
|
||||
last_entropy[1] = last_entropy[0]
|
||||
}
|
||||
|
||||
self.block_size_ = 0
|
||||
histogramClearDistance(&histograms[self.curr_histogram_ix_])
|
||||
self.merge_last_count_++
|
||||
if self.merge_last_count_ > 1 {
|
||||
self.target_block_size_ += self.min_block_size_
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if is_final {
|
||||
*self.histograms_size_ = split.num_types
|
||||
split.num_blocks = self.num_blocks_
|
||||
}
|
||||
}
|
||||
|
||||
/* Adds the next symbol to the current histogram. When the current histogram
|
||||
reaches the target size, decides on merging the block. */
|
||||
func blockSplitterAddSymbolDistance(self *blockSplitterDistance, symbol uint) {
|
||||
histogramAddDistance(&self.histograms_[self.curr_histogram_ix_], symbol)
|
||||
self.block_size_++
|
||||
if self.block_size_ == self.target_block_size_ {
|
||||
blockSplitterFinishBlockDistance(self, false) /* is_final = */
|
||||
}
|
||||
}
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2015 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Greedy block splitter for one block category (literal, command or distance).
|
||||
*/
|
||||
type blockSplitterLiteral struct {
|
||||
alphabet_size_ uint
|
||||
min_block_size_ uint
|
||||
split_threshold_ float64
|
||||
num_blocks_ uint
|
||||
split_ *blockSplit
|
||||
histograms_ []histogramLiteral
|
||||
histograms_size_ *uint
|
||||
target_block_size_ uint
|
||||
block_size_ uint
|
||||
curr_histogram_ix_ uint
|
||||
last_histogram_ix_ [2]uint
|
||||
last_entropy_ [2]float64
|
||||
merge_last_count_ uint
|
||||
}
|
||||
|
||||
func initBlockSplitterLiteral(self *blockSplitterLiteral, alphabet_size uint, min_block_size uint, split_threshold float64, num_symbols uint, split *blockSplit, histograms *[]histogramLiteral, histograms_size *uint) {
|
||||
var max_num_blocks uint = num_symbols/min_block_size + 1
|
||||
var max_num_types uint = brotli_min_size_t(max_num_blocks, maxNumberOfBlockTypes+1)
|
||||
/* We have to allocate one more histogram than the maximum number of block
|
||||
types for the current histogram when the meta-block is too big. */
|
||||
self.alphabet_size_ = alphabet_size
|
||||
|
||||
self.min_block_size_ = min_block_size
|
||||
self.split_threshold_ = split_threshold
|
||||
self.num_blocks_ = 0
|
||||
self.split_ = split
|
||||
self.histograms_size_ = histograms_size
|
||||
self.target_block_size_ = min_block_size
|
||||
self.block_size_ = 0
|
||||
self.curr_histogram_ix_ = 0
|
||||
self.merge_last_count_ = 0
|
||||
brotli_ensure_capacity_uint8_t(&split.types, &split.types_alloc_size, max_num_blocks)
|
||||
brotli_ensure_capacity_uint32_t(&split.lengths, &split.lengths_alloc_size, max_num_blocks)
|
||||
self.split_.num_blocks = max_num_blocks
|
||||
*histograms_size = max_num_types
|
||||
if histograms == nil || cap(*histograms) < int(*histograms_size) {
|
||||
*histograms = make([]histogramLiteral, *histograms_size)
|
||||
} else {
|
||||
*histograms = (*histograms)[:*histograms_size]
|
||||
}
|
||||
self.histograms_ = *histograms
|
||||
|
||||
/* Clear only current histogram. */
|
||||
histogramClearLiteral(&self.histograms_[0])
|
||||
|
||||
self.last_histogram_ix_[1] = 0
|
||||
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
|
||||
}
|
||||
|
||||
/* Does either of three things:
|
||||
(1) emits the current block with a new block type;
|
||||
(2) emits the current block with the type of the second last block;
|
||||
(3) merges the current block with the last block. */
|
||||
func blockSplitterFinishBlockLiteral(self *blockSplitterLiteral, is_final bool) {
|
||||
var split *blockSplit = self.split_
|
||||
var last_entropy []float64 = self.last_entropy_[:]
|
||||
var histograms []histogramLiteral = self.histograms_
|
||||
self.block_size_ = brotli_max_size_t(self.block_size_, self.min_block_size_)
|
||||
if self.num_blocks_ == 0 {
|
||||
/* Create first block. */
|
||||
split.lengths[0] = uint32(self.block_size_)
|
||||
|
||||
split.types[0] = 0
|
||||
last_entropy[0] = bitsEntropy(histograms[0].data_[:], self.alphabet_size_)
|
||||
last_entropy[1] = last_entropy[0]
|
||||
self.num_blocks_++
|
||||
split.num_types++
|
||||
self.curr_histogram_ix_++
|
||||
if self.curr_histogram_ix_ < *self.histograms_size_ {
|
||||
histogramClearLiteral(&histograms[self.curr_histogram_ix_])
|
||||
}
|
||||
self.block_size_ = 0
|
||||
} else if self.block_size_ > 0 {
|
||||
var entropy float64 = bitsEntropy(histograms[self.curr_histogram_ix_].data_[:], self.alphabet_size_)
|
||||
var combined_histo [2]histogramLiteral
|
||||
var combined_entropy [2]float64
|
||||
var diff [2]float64
|
||||
var j uint
|
||||
for j = 0; j < 2; j++ {
|
||||
var last_histogram_ix uint = self.last_histogram_ix_[j]
|
||||
combined_histo[j] = histograms[self.curr_histogram_ix_]
|
||||
histogramAddHistogramLiteral(&combined_histo[j], &histograms[last_histogram_ix])
|
||||
combined_entropy[j] = bitsEntropy(combined_histo[j].data_[0:], self.alphabet_size_)
|
||||
diff[j] = combined_entropy[j] - entropy - last_entropy[j]
|
||||
}
|
||||
|
||||
if split.num_types < maxNumberOfBlockTypes && diff[0] > self.split_threshold_ && diff[1] > self.split_threshold_ {
|
||||
/* Create new block. */
|
||||
split.lengths[self.num_blocks_] = uint32(self.block_size_)
|
||||
|
||||
split.types[self.num_blocks_] = byte(split.num_types)
|
||||
self.last_histogram_ix_[1] = self.last_histogram_ix_[0]
|
||||
self.last_histogram_ix_[0] = uint(byte(split.num_types))
|
||||
last_entropy[1] = last_entropy[0]
|
||||
last_entropy[0] = entropy
|
||||
self.num_blocks_++
|
||||
split.num_types++
|
||||
self.curr_histogram_ix_++
|
||||
if self.curr_histogram_ix_ < *self.histograms_size_ {
|
||||
histogramClearLiteral(&histograms[self.curr_histogram_ix_])
|
||||
}
|
||||
self.block_size_ = 0
|
||||
self.merge_last_count_ = 0
|
||||
self.target_block_size_ = self.min_block_size_
|
||||
} else if diff[1] < diff[0]-20.0 {
|
||||
split.lengths[self.num_blocks_] = uint32(self.block_size_)
|
||||
split.types[self.num_blocks_] = split.types[self.num_blocks_-2]
|
||||
/* Combine this block with second last block. */
|
||||
|
||||
var tmp uint = self.last_histogram_ix_[0]
|
||||
self.last_histogram_ix_[0] = self.last_histogram_ix_[1]
|
||||
self.last_histogram_ix_[1] = tmp
|
||||
histograms[self.last_histogram_ix_[0]] = combined_histo[1]
|
||||
last_entropy[1] = last_entropy[0]
|
||||
last_entropy[0] = combined_entropy[1]
|
||||
self.num_blocks_++
|
||||
self.block_size_ = 0
|
||||
histogramClearLiteral(&histograms[self.curr_histogram_ix_])
|
||||
self.merge_last_count_ = 0
|
||||
self.target_block_size_ = self.min_block_size_
|
||||
} else {
|
||||
/* Combine this block with last block. */
|
||||
split.lengths[self.num_blocks_-1] += uint32(self.block_size_)
|
||||
|
||||
histograms[self.last_histogram_ix_[0]] = combined_histo[0]
|
||||
last_entropy[0] = combined_entropy[0]
|
||||
if split.num_types == 1 {
|
||||
last_entropy[1] = last_entropy[0]
|
||||
}
|
||||
|
||||
self.block_size_ = 0
|
||||
histogramClearLiteral(&histograms[self.curr_histogram_ix_])
|
||||
self.merge_last_count_++
|
||||
if self.merge_last_count_ > 1 {
|
||||
self.target_block_size_ += self.min_block_size_
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if is_final {
|
||||
*self.histograms_size_ = split.num_types
|
||||
split.num_blocks = self.num_blocks_
|
||||
}
|
||||
}
|
||||
|
||||
/* Adds the next symbol to the current histogram. When the current histogram
|
||||
reaches the target size, decides on merging the block. */
|
||||
func blockSplitterAddSymbolLiteral(self *blockSplitterLiteral, symbol uint) {
|
||||
histogramAddLiteral(&self.histograms_[self.curr_histogram_ix_], symbol)
|
||||
self.block_size_++
|
||||
if self.block_size_ == self.target_block_size_ {
|
||||
blockSplitterFinishBlockLiteral(self, false) /* is_final = */
|
||||
}
|
||||
}
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2017 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Parameters for the Brotli encoder with chosen quality levels. */
|
||||
type hasherParams struct {
|
||||
type_ int
|
||||
bucket_bits int
|
||||
block_bits int
|
||||
hash_len int
|
||||
num_last_distances_to_check int
|
||||
}
|
||||
|
||||
type distanceParams struct {
|
||||
distance_postfix_bits uint32
|
||||
num_direct_distance_codes uint32
|
||||
alphabet_size uint32
|
||||
max_distance uint
|
||||
}
|
||||
|
||||
/* Encoding parameters */
|
||||
type encoderParams struct {
|
||||
mode int
|
||||
quality int
|
||||
lgwin uint
|
||||
lgblock int
|
||||
size_hint uint
|
||||
disable_literal_context_modeling bool
|
||||
large_window bool
|
||||
hasher hasherParams
|
||||
dist distanceParams
|
||||
dictionary encoderDictionary
|
||||
}
|
||||
+103
@@ -0,0 +1,103 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
func brotli_min_double(a float64, b float64) float64 {
|
||||
if a < b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_max_double(a float64, b float64) float64 {
|
||||
if a > b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_min_float(a float32, b float32) float32 {
|
||||
if a < b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_max_float(a float32, b float32) float32 {
|
||||
if a > b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_min_int(a int, b int) int {
|
||||
if a < b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_max_int(a int, b int) int {
|
||||
if a > b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_min_size_t(a uint, b uint) uint {
|
||||
if a < b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_max_size_t(a uint, b uint) uint {
|
||||
if a > b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_min_uint32_t(a uint32, b uint32) uint32 {
|
||||
if a < b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_max_uint32_t(a uint32, b uint32) uint32 {
|
||||
if a > b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_min_uint8_t(a byte, b byte) byte {
|
||||
if a < b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
|
||||
func brotli_max_uint8_t(a byte, b byte) byte {
|
||||
if a > b {
|
||||
return a
|
||||
} else {
|
||||
return b
|
||||
}
|
||||
}
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
package brotli
|
||||
|
||||
/* Copyright 2013 Google Inc. All Rights Reserved.
|
||||
|
||||
Distributed under MIT license.
|
||||
See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
|
||||
*/
|
||||
|
||||
/* Functions for encoding of integers into prefix codes the amount of extra
|
||||
bits, and the actual values of the extra bits. */
|
||||
|
||||
/* Here distance_code is an intermediate code, i.e. one of the special codes or
|
||||
the actual distance increased by BROTLI_NUM_DISTANCE_SHORT_CODES - 1. */
|
||||
func prefixEncodeCopyDistance(distance_code uint, num_direct_codes uint, postfix_bits uint, code *uint16, extra_bits *uint32) {
|
||||
if distance_code < numDistanceShortCodes+num_direct_codes {
|
||||
*code = uint16(distance_code)
|
||||
*extra_bits = 0
|
||||
return
|
||||
} else {
|
||||
var dist uint = (uint(1) << (postfix_bits + 2)) + (distance_code - numDistanceShortCodes - num_direct_codes)
|
||||
var bucket uint = uint(log2FloorNonZero(dist) - 1)
|
||||
var postfix_mask uint = (1 << postfix_bits) - 1
|
||||
var postfix uint = dist & postfix_mask
|
||||
var prefix uint = (dist >> bucket) & 1
|
||||
var offset uint = (2 + prefix) << bucket
|
||||
var nbits uint = bucket - postfix_bits
|
||||
*code = uint16(nbits<<10 | (numDistanceShortCodes + num_direct_codes + ((2*(nbits-1) + prefix) << postfix_bits) + postfix))
|
||||
*extra_bits = uint32((dist - offset) >> postfix_bits)
|
||||
}
|
||||
}
|
||||
Loaded 100 of 1085 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user