AI Helpers

forgetless re-exports local ML helpers for embeddings, text generation, and optional image description. These match the modules under src/ai.

Embeddings#

rust
use forgetless::{cosine_similarity, embed_batch, embed_text, EmbeddingCache};

let query = embed_text("release summary")?;
let docs = embed_batch(&["design review", "old notes"])?;
let score = cosine_similarity(&query, &docs[0]);

let mut cache = EmbeddingCache::new(10_000);

The embedding backend uses FastEmbed with the quantized all-MiniLM-L6-v2 model and an in-process LRU cache.

Local LLM#

PresetModel ID
LLMConfig::smollm2()HuggingFaceTB/SmolLM2-135M-Instruct
LLMConfig::smollm2_360m()HuggingFaceTB/SmolLM2-360M-Instruct
LLMConfig::qwen_0_5b()Qwen/Qwen2.5-0.5B-Instruct
LLMConfig::phi3_mini()microsoft/Phi-3-mini-4k-instruct
rust
use forgetless::{LLM, LLMConfig, Quantization};

LLM::init_with_config(
    LLMConfig::qwen_0_5b()
        .with_quantization(Quantization::Q4)
        .with_max_tokens(256),
).await?;

Vision#

rust
use forgetless::{describe_image, describe_image_with_prompt, init_vision, is_vision_ready};

if !is_vision_ready() {
    init_vision().await?;
}

let caption = describe_image(&image_bytes).await?;
let answer = describe_image_with_prompt(&image_bytes, "What text is visible?").await?;

The vision helper uses HuggingFaceTB/SmolVLM-256M-Instruct through mistralrs and resizes inputs to a fixed 224x224 shape before inference.

Configuration

Configure Forgetless token budgets, scoring weights, chunk behavior, local model options, and feature toggles to tune LLM context for your workload.

Tokenizer

Count LLM tokens in Rust with cl100k_base or a custom chars-per-token mode, plus truncation helpers and image token estimation to plan context budgets.

Results

Read the Forgetless result types: OptimizedContext, selected chunks, score breakdowns, token statistics, and the compression ratio returned after each run.