Chunking

Chunking

Chunking is content-aware. forgetless picks a strategy from ContentType, then applies token targets, overlap, and deduplication before scoring begins.

Chunk presets#

Preset	Target	Max	Good for
`ChunkConfig::default()`	512	1024	General text and mixed project context.
`ChunkConfig::for_code()`	256	512	Codebases where smaller functions need independent ranking.
`ChunkConfig::for_conversation()`	200	400	Chat transcripts and message histories.
`ChunkConfig::for_speed()`	1000	2000	Fast coarse compression with larger chunks.
`ChunkConfig::for_quality()`	256	512	More selective ranking with finer chunk boundaries.

Content types#

Type	Detected from	Notes
Text	Fallback default	Used for plain text and unknown extensions.
Code	`.rs`, `.py`, `.ts`, `.tsx`, `.go`, `.java`, and similar	Optimized for source-oriented chunking.
Markdown	`.md`, `.markdown`	Keeps markdown documents out of the plain-text path.
Conversation	Explicit config path	Best for message-oriented histories.
Structured	`.json`, `.yaml`, `.toml`, `.xml`	Useful for config and data files.

Customization#

rust

use forgetless::{ChunkConfig, Config, ForgetlessConfig, ScoringConfig};

let advanced = ForgetlessConfig::new(
    Config::default().context_limit(64_000),
)
.with_chunk(
    ChunkConfig::for_quality()
        .with_target_tokens(256)
        .with_max_tokens(512)
        .with_min_tokens(10)
        .with_deduplication(true),
)
.with_scoring(ScoringConfig {
    semantic_weight: 0.6,
    keyword_weight: 0.25,
    priority_weight: 0.15,
});

Inputs and Priorities

Control what the LLM context optimizer keeps using WithPriority, FileWithPriority, byte payloads, and lazy file inputs to rank and preserve critical content.

Pipeline

See how the Forgetless context optimization pipeline runs: preview filtering, parallel file reads, chunking, relevance scoring, and budget-based selection.

Introduction

Learn how Forgetless, a Rust LLM context optimizer, ranks and compresses oversized prompts and files into budget-aware context for production workflows.