Tokenizer

The public tokenizer surface is built around TokenizerModel and TokenCounter. Default mode uses cl100k_base; custom mode falls back to a chars-per-token estimate.

Tokenizer models#

Variant	Meaning	When to use it
`TokenizerModel::Default`	Uses `cl100k_base` through `tiktoken-rs`.	Best default for modern chat models.
`TokenizerModel::Custom { chars_per_token_x100 }`	Uses a lightweight approximation.	Good when you need cheap counting without BPE.

Token counter#

rust

use forgetless::{TokenCounter, TokenizerModel};

let exact = TokenCounter::new(TokenizerModel::Default)?;
let rough = TokenCounter::new(TokenizerModel::Custom {
    chars_per_token_x100: 400,
})?;

let fits = exact.fits_budget("hello world", 32);
let tokens = exact.count("release notes and code review summary");
let clipped = exact.truncate_to_budget("very long text ...", 128);

Image estimation#

TokenCounter::count_image mirrors the vision-token heuristic in the crate: images are scaled to fit a maximum side of 2048, then the shortest side is scaled toward 768 before 512x512 tiles are counted.

Detail	Cost model
Low	Always 85 tokens.
High	`170 * number_of_tiles + 85` after the crate's resize pipeline.
Auto	Uses the same branch as `High`.

Configuration

Configure Forgetless token budgets, scoring weights, chunk behavior, local model options, and feature toggles to tune LLM context for your workload.

Results

Read the Forgetless result types: OptimizedContext, selected chunks, score breakdowns, token statistics, and the compression ratio returned after each run.

HTTP Server

Run the optional Forgetless HTTP server built on Axum: a multipart API for language-agnostic context optimization with a health endpoint and file priorities.