Chunking & vector index

This pipeline turns "the text of a note, file or URL" into "retrievable chunks with vectors": chunk first, embed per chunk, then write both the relational tables and the sqlite-vec virtual tables. Every step can degrade, because the goal is indexing may be slow, but data must never be corrupted.

Chunking strategy

Item	Value
Preferred	With a chunk model configured and a working provider, LLM chunking runs (`ChunkMethod = llm`)
Fallback	No model, a failed call or an unparsable answer falls back to structural chunking (`ChunkMethod = structure`) splitting on headings and paragraphs
Structural values	`MaxChunkSize = 1500`, `MinChunkSize = 200`, `OverlapSize = 100` characters
LLM constraints	The prompt asks for 300–1500 characters per chunk; text under 500 characters may return a single chunk

Write order

Write NoteChunks keyed by knowledge item and chunk index (content, method, summary).
Embed each chunk and write NoteChunkEmbeddings (model, dimensions, timestamp) plus the vector itself.
Mirror into the sqlite-vec table (vec_chunk_embeddings); the virtual table does not support REPLACE, so rows are always deleted and re-inserted.
For single-chunk content an extra whole-note vector is written (vec_note_embeddings) so the note itself can be recalled directly.

Reuse: chunks whose content and summary are unchanged keep their existing vectors, so editing a note does not re-embed the whole document. The note-level check also repairs vectors that exist in the relational table but are missing from the vec table after an interrupted run.

Dimension resolution

Order	Source
1	`Dimensions` of the most recent embedding row (most trustworthy)
2	The model configured for the `embedding` purpose
3	Setting `Embedding:Dimensions` (default 1536)

When a connection opens and the dimensions no longer match, the four vec tables (notes, chunks, memories, code) are dropped and rebuilt — that is the mechanism behind "change the embedding model and you must re-index".

Failure and degradation

Case	Behaviour
No embedding provider	The whole pipeline is skipped; no half-written data
LLM chunking fails	Falls back to structural chunking, so a flaky chunk model never loses the index
`sqlite-vec` missing / table missing / dimension mismatch	Retrieval falls back to in-memory cosine similarity; relational data stays complete
A single vector write fails	Logged; three consecutive failures in a batch abort that batch to avoid a cascade
Cancellation	`OperationCanceledException` propagates and is not treated as a failure

Observability

Log prefixes [Chunk] and [KI Embed], including chunk method, chunk count, dimensions and failure reasons.
NoteChunkDto exposes ChunkIndex / ChunkMethod / HasEmbedding / Summary for the chunk list and index status in the UI.
Background task types: GenerateEmbedding (notes) and GenerateKnowledgeItemEmbedding (files / URLs), with duration and errors visible on the Tasks page.

Practical notes

Chunking decides index quality: structural chunking splits on headings, so keep subheadings in long notes.
After switching embedding models you must re-index and update Embedding:Dimensions.
Vectors serve retrieval only; NoteChunkEmbeddings carries metadata and powers the fallback — both must be written.
Code chunks live in their own vec_work_code_chunks table and never interfere with note vectors.