AI Engineering & GEO Glossary

A website optimization methodology targeting generative AI engines (ChatGPT, Perplexity, Claude) — the goal is to be correctly crawled, understood, and cited by AI. Unlike SEO which targets search engine spiders, GEO targets AI engines.

A deep learning model based on the Transformer architecture, trained on massive text corpora to understand and generate natural language. Notable models include GPT series, Claude series, Qwen, DeepSeek, and others.

A JSON-based structured data format for embedding machine-readable semantic information in web pages. Search engines and AI engines use JSON-LD to understand entity types (Organization, Article, FAQ, Product, etc.).

A structured data vocabulary jointly initiated by Google, Microsoft, Yahoo and Yandex, defining hundreds of entity types and properties. JSON-LD is the recommended implementation method for Schema.org.

A content structure where the core conclusion comes first, followed by supporting arguments. AI engines prioritize content at the beginning of a page — BLUF structure increases the probability of being correctly cited, making it a key GEO content strategy.

A text file placed at the website root that provides a site guide for AI agents (e.g., ChatGPT and Claude crawlers), listing core pages and crawling rules. It complements robots.txt in GEO infrastructure.

An automated program used by AI engines to crawl web content. Unlike search engine crawlers (Googlebot, Bingbot), AI crawlers (GPTBot, ClaudeBot, PerplexityBot) primarily collect data for LLM training and inference.

An AI architecture combining information retrieval with text generation. The LLM first retrieves relevant documents from an external knowledge base, then injects the results as context into the prompt — reducing hallucinations and improving timeliness and accuracy.

The basic unit of text processing in LLMs. A token can be a word, subword, or character. In Chinese, one character typically corresponds to 1-2 tokens. Tokens are also the billing unit for API calls — both input and output are charged per token.

The practice of designing input text (prompts) to guide LLMs toward desired outputs. Techniques include Few-shot (providing examples), Chain-of-Thought (guiding reasoning steps), and System Prompt (setting role and behavior boundaries).

Machine-readable data embedded in web pages in standardized formats (JSON-LD, Microdata), helping search engines and AI engines understand page content. Core types include Organization, WebPage, Article, FAQPage, BreadcrumbList, and DefinedTermSet.

A build-time approach that pre-renders all pages into static HTML files, requiring no server-side runtime at deploy time. Frameworks like Astro, Next.js (static export), Hugo, and 11ty support SSG. Static sites are most AI-crawler-friendly — content is fully readable without JavaScript.

An HTML language annotation attribute that tells search engines and AI engines the relationship between different language versions of the same content. Correct hreflang configuration prevents multilingual sites from being flagged as duplicate content.

A declaration in the HTML head (`<link rel="canonical">`) specifying the authoritative version URL of the current page. When the same content is accessible via multiple URLs, canonical tells search engines and AI engines which one is the original source.

A file (sitemap.xml) listing all important page URLs, last modification times, and update frequencies. Search engines and AI crawlers use sitemaps to discover and index site content — a standard GEO infrastructure component.

A Schema.org structured data type for FAQ pages. Each Q&A pair (Question + Answer) is independently marked, allowing AI engines to directly cite FAQ answers when answering related questions. One of the most directly effective structured data types for GEO.

A structured knowledge base organized around entities and their relationships. Search engines and AI engines use knowledge graphs to understand entity associations. Websites can output entity relationships via JSON-LD to help AI engines build more accurate knowledge connections.

A technique for mapping unstructured data (text, images) into high-dimensional vector spaces. Semantically similar texts are closer in vector space. Embeddings are the foundation of RAG, semantic search, and recommendation systems.

Further training a pre-trained LLM on domain-specific data to improve performance on specific tasks. The key difference from RAG: fine-tuning changes model parameters; RAG changes only the input context without modifying parameters.

A database specialized for storing and retrieving high-dimensional vectors (embeddings), supporting approximate nearest neighbor (ANN) search with indexes like HNSW and IVF. Mainstream choices: Qdrant (fastest adoption, simplest ops), Milvus (distributed, billion-scale), pgvector (reuse existing PostgreSQL), Elasticsearch (full-text + vector in one). The core storage component of RAG knowledge bases.

The process of splitting long documents into retrieval units (chunks) along semantic boundaries. Production practice: split by heading hierarchy and paragraph boundaries, overlap adjacent chunks by 50-100 characters, keep chunks at 200-1000 characters. Chunking quality directly determines retrieval recall — hard-splitting by fixed character count cuts semantic units in half and is the most common RAG pitfall.

A retrieval strategy combining vector search (semantic similarity) with keyword search (BM25 exact match). Vector search handles "asked a different way" semantics; keyword search handles exact matches like contract clause numbers and model names. Merged results are typically reranked — the standard configuration for production RAG.

A model that re-orders initial retrieval results, typically a cross-encoder (e.g. bge-reranker) that scores each query-document pair to surface what truly matters. Small VRAM footprint (a few GB) with outsized retrieval gains — the highest-ROI investment in the RAG pipeline.

A classic keyword retrieval ranking algorithm scoring documents by term frequency (TF) and inverse document frequency (IDF). Complementary to vector search: vectors handle semantic similarity, BM25 handles exact matches (numbers, proper nouns, abbreviations). Lucene/Elasticsearch default relevance is based on improved BM25.

The positional bias documented in the 2023 "Lost in the Middle" paper: for a long document, models recall information at the beginning and end significantly better than information in the middle — an intrinsic property of transformer attention distribution. 2026 long-context models have improved but not eliminated it, only narrowing the drop. Three engineering mitigations: ① sandwich structure — system prompt at the very front, the user’s latest question at the very end; ② when RAG injects content, place the most relevant chunks at the first/last positions of the context; ③ pre-extract key facts (amounts, dates, clause numbers) into structured fields instead of relying on the model to find them in a long text.

The degradation in effective utilization of context as input length grows — "fits in the window" does not mean "works reliably when stuffed in". Three causes: ① positional bias (lower recall for the middle section); ② distractors (semantically similar noise dilutes attention to the right answer); ③ insufficient length generalization (some models’ practically usable window is below the advertised value). It is the direct reason long-context models score 10-30 precision points lower on real document QA than on NIAH, and the hidden risk item of the "full-stuff" approach.

The idealized benchmark for long-context capability: bury a sentence containing a random code (the needle) in a long document (the haystack) and ask the model to find it. Unique answer, pure-noise background, precise question — pass rates of 90-100%. But real business documents have answers spanning multiple passages, noise, and reasoning; real precision typically runs 10-30 points lower. Conclusion: NIAH scores can only eliminate the worst models, not guarantee production precision — selection requires a golden dataset (50-100 items) built from your own real business documents, with regression on every model or prompt change, and no release if precision drops more than 3 points.

A relevance scorer that encodes the "query + candidate document" pair together and produces a relevance score, in contrast to bi-encoders (query and document encoded separately — vectors are precomputable but scoring is coarse). Cross-encoders (bge-reranker, cohere rerank, etc.) score candidates one by one: slow (50-100ms each) but accurate. Standard use is the reranking stage: hybrid retrieval recalls top-20 → cross-encoder ranks → top-8~12 injected into context. Small VRAM footprint (a few GB) and the highest-ROI step in RAG tuning.

The first phase of LLM inference: the entire input context is processed in one forward pass to build the KV cache. TTFT is dominated by prefill time, whose compute cost grows roughly quadratically with input length — a 100K-token input means prefill in the seconds-to-tens-of-seconds depending on hardware, the direct cause of first-token latency degradation in long-context scenarios. Prefill compute also directly squeezes concurrency: on the same inference GPU, QPS at 8K input can be 5-10x the QPS at 100K input, so "input length × concurrency target" must be computed together in on-premise sizing.

A retrieval strategy running vector search (semantic similarity) and BM25 (exact keyword matching) in parallel, merging and deduplicating, then reranking. In legal/finance/technical documents, exact terms (clause number "4.2.1", amount "8.6M", contract name) are a vector-retrieval blind spot — semantically adjacent but far in vector space, so BM25 must catch them. Production parameters: top-10 recall per path → merge → rerank to top-8~12; below the similarity threshold (cosine ≥ 0.75) prefer no injection. Lifts recall 15-30% over vector-only top-5, far cheaper than upgrading the embedding model.

The RAG alternative of stuffing the entire document into the context window for the model to read through, used in long-document QA. Advantage: information is naturally complete with zero pipeline build. Three hard constraints: ① attention decay (lost in the middle — lower recall for the middle section); ② token billing — input is billed and grows linearly with document length, so high-frequency bills run 5-10x above RAG; ③ prefill latency — long inputs start TTFT in the seconds, an immediate disqualifier for online scenarios (TTFT < 3s required). Decision rule of thumb: document ≤ 30% of the window, or low-frequency/offline → full-stuff; high-frequency online → RAG; the middle zone is decided by a call-volume cost calculation.

The engineering requirement that answers must cite source chunks and citations are programmatically verified: cited source IDs must fall within the chunks injected in this call; out-of-range citations are classified as hallucinated citations and the answer is rejected. Companion practice: every injected chunk carries a "document name + section + page/line number" tag, generation is required to cite IDs, and the program verifies ID validity. It is standard for auditable long-document QA — users can be pointed to the original source when they challenge an answer, it suppresses "confident nonsense", and it is the trust infrastructure when delivering to customers.

Rewriting the user's raw question before retrieval to fix failures caused by colloquial references and vague wording. E.g. "is that policy from last time still valid?" → "policy validity 2026". Implementable with a small model or rules; noticeably improves hit rate and is a common preprocessing step in production RAG.

An information retrieval metric measuring where the first correct answer appears in results. MRR = 1/rank — rank 1 scores 1.0, rank 5 scores 0.2. Used with recall to evaluate RAG retrieval quality: below 80% recall, optimize retrieval before touching generation.

An architecture where multiple specialized AI agents collaborate to complete complex tasks. Three signals you need it: ① independent subtasks that can run in parallel; ② different steps needing different tool sets with permission isolation; ③ adversarial generate→review loops. More flexible but with significantly higher token cost and debugging difficulty — prefer a single agent for fixed processes.

Also called Function Calling. The mechanism by which an LLM selects and invokes external functions/APIs. The model emits a structured call (function name + arguments) that the application executes and returns the result of. Design points: single-purpose tools, clear parameter descriptions (the model decides when to call from them), structured return values, and exceptions explicitly returned to the model rather than swallowed.

The standard agent execution loop: think (Reason) → call a tool (Act) → observe the result → think again until the task completes. ReAct grounds each step in tool results, correcting judgment as it goes — the foundational mechanism for reducing hallucination: answers must cite tool-call evidence, and evidence-free answers should be rejected.

Structuring multiple agents/steps into a workflow by dependency. Three mainstream patterns: linear pipeline (fixed order, most predictable), DAG graph orchestration (parallelism, branches and joins — the production default), and Planner-Executor (dynamic task decomposition, most flexible but least predictable). Order: use a pipeline when you can, a DAG when you must, planner-executor only when nothing else fits.

An agent orchestration framework by the LangChain team that describes workflows as graphs, with state persistence, checkpoint recovery, parallelism and conditional branching. Suited to engineering-capable teams building production multi-agent systems — the productionization path after Dify validation.

An open-source LLM application development platform with visual drag-and-drop orchestration for agent workflows, RAG pipelines and conversational apps, plus model integration, tooling, monitoring and publishing. Fits fast business-flow validation; deep customization (complex state, custom scheduling) hits a wall, and production multi-agent often migrates to LangGraph.

Breaking a complex goal into an executable sequence of subtasks. Granularity directly affects success rate and cost: too coarse and the model cannot execute; too fine and token overhead explodes. In multi-agent systems the planner handles decomposition and dispatch — the first quality gate in agent engineering.

The mechanism where an agent reviews its own output after generation, or has an independent reviewer agent validate it: checking whether citations are real, conclusions contradict data, or key constraints are missed — rolling back on error. Reflection cannot eliminate hallucination, but together with evidence chains and human fallback it forms the three-layer hallucination-control structure.

An open protocol proposed by Anthropic that defines a standardized interaction method between AI models and external tools/data sources. MCP enables LLMs to safely call APIs, query databases, and read/write files — infrastructure for AI agents.

The server-side process in the MCP protocol, exposing three capability primitives to models/apps: Tools (executable actions, e.g. query orders), Resources (read-only data, e.g. knowledge-base documents), and Prompts (pre-built prompt templates). A Server written once is reusable by every MCP-capable client — the key to "write tools once, use them everywhere." Production deployment essentials: authentication (OAuth/API key), timeout & retry, idempotency, model-readable errors, and call observability.

The client-side component in the MCP protocol, running inside the AI application (Host) and communicating with one or more MCP Servers. Host-to-Server is one-to-many: one app can mount several Servers, and one Server can serve several Hosts. Two transports: stdio (local subprocess, zero network exposure) and HTTP+SSE (remote deployment, requiring auth and TLS).

The engineering practice of systematically designing and orchestrating the context fed to an LLM (system prompt, retrieval results, tool-call outputs, conversation history) to control model behavior. The difference from Prompt Engineering: a prompt is writing one good instruction; Context Engineering manages the entire context lifecycle — what goes in, how much, in what order, and at what cost (token spend is often dominated by tool results re-entering context).

The mechanism that dispatches each request by its characteristics to the most suitable model size. Three implementations: rule-based routing (by endpoint/field/user tier), semantic routing (embedding-based intent matching, then dispatch), and cascade routing (start small, escalate only when quality is insufficient). Value: splitting traffic that used to go 100% to a flagship model into an 8:2 mix cuts overall cost by 50%–80% while complex-task quality stays unchanged. Precondition: a capability-cost evaluation baseline per task class × per model.

The vendor-side KV cache: repeated requests with the same prefix (system prompt, long documents) are billed at the cache-hit rate (typically 10%–25%) from the second request on. Engineering note: structure prompts "static first, dynamic last" — fixed content in the prefix, varying content behind — to maximize hits. For long-context workloads (hundreds-of-pages Q&A, long system prompts, multi-turn conversations) input-token cost drops 75%–90%.

The application-side cache: embedding similarity determines whether request intents match; on a hit, the stored answer is returned and the model call is skipped entirely. Difference from Prompt Caching: the former saves the entire call; the latter only saves recomputing input tokens. The core is invalidation — version the cache key on knowledge updates, shorten TTL on hot entries, and skip the cache for sensitive categories.

A cost-saving strategy where a cheap small model answers first and escalates to mid-tier, then flagship, only when quality is suspect. Three quality-detection approaches: structured self-check (model outputs a confidence field), rule validation (JSON Schema / required-field presence), and a discriminator model (a small classifier judging whether the answer passes). Upside: average cost approaches the small model while worst-case quality holds at flagship level.

An automated software delivery practice. CI means frequently merging code changes into the main branch with automated testing; CD means deploying verified code to production through an automated pipeline. GitHub Actions and GitLab CI are mainstream CI/CD tools.

A containerization platform that packages applications and their dependencies into lightweight, portable containers, ensuring consistent operation across different environments. Docker Compose orchestrates multi-container applications (e.g., web service + database + cache) and is the mainstream choice for small-to-medium project deployment.

A specification standard for describing RESTful APIs (formerly the Swagger specification). OpenAPI spec files enable auto-generation of API documentation, client SDKs, and mock servers — the core tool for contract-first API development.

An architectural pattern that decomposes an application into multiple independently deployable services, each corresponding to a business subdomain, communicating via lightweight protocols (REST/gRPC/message queues). Microservices enable independent deployment, scaling, and teams, but introduce distributed transactions and service governance complexity.

An intermediary layer between clients and backend services, handling cross-cutting concerns such as request routing, authentication, rate limiting, circuit breaking, and protocol translation. In multi-provider Token aggregation scenarios, the API gateway also handles routing decisions (distributing requests by price/latency) and fault tolerance (circuit breaking and degradation).

An open-source automated tool by Google for auditing web page quality across five dimensions: Performance, Accessibility, SEO, Best Practices, and PWA. Lighthouse scores serve as a key reference metric for measuring frontend performance optimization effectiveness.

A set of core web performance metrics defined by Google, including LCP (Largest Contentful Paint, measuring loading performance), CLS (Cumulative Layout Shift, measuring visual stability), and TBT (Total Blocking Time, measuring interactivity). These metrics directly impact search rankings and user experience scores.

A next-generation image format developed by Google, supporting both lossy and lossless compression, transparency, and animation. Compared to PNG, WebP lossless compression averages 26% smaller; compared to JPEG, lossy compression averages 25-34% smaller. All images on this site have been converted from PNG to WebP, achieving 89% total size reduction.

A rendering pattern where pages are rendered into complete HTML on the server before being sent to the client. The key difference from SSG (Static Site Generation): SSR generates HTML dynamically on each request, suitable for pages needing real-time data; SSG pre-generates HTML at build time, suitable for relatively static content. Next.js supports both SSR and SSG modes.

A high-performance web server and reverse proxy server, also commonly used for load balancing, HTTP caching, SSL termination, and static file serving. This site uses Nginx as the front-end server, handling HTTPS termination, static asset serving, security header injection, and reverse proxy configuration.

An open standard (RFC 7519) for securely transmitting information between parties as a JSON object. JWT consists of three parts: Header, Payload, and Signature. It is commonly used for API authentication and authorization. Stateless (no server-side session storage needed), but cannot be actively revoked — suitable for short-lived token scenarios.

An open standard for access delegation, allowing users to grant third-party applications access to specific resources without sharing their passwords. OAuth 2.0 defines four authorization flows: Authorization Code (most common, for third-party apps), Client Credentials (service-to-service), Password (first-party apps, not recommended), and Implicit (deprecated, replaced by PKCE).

An HTTP-header-based mechanism that allows a server to declare which origins (domain, protocol, port) are permitted to access its resources. Browsers block cross-origin requests by default; CORS safely relaxes this restriction by setting Access-Control-Allow-Origin headers. Production environments should configure CORS as a whitelist, allowing only trusted origins.

A protocol enabling full-duplex communication over a single TCP connection. Unlike HTTP's request-response model, WebSocket allows both client and server to push data at any time after the connection is established. Suitable for chat, real-time collaboration, and gaming scenarios requiring low-latency bidirectional communication. Higher implementation complexity — requires connection management, heartbeats, and horizontal scaling.

A lightweight real-time communication protocol based on HTTP, allowing servers to push data to clients over a long-lived connection. Clients use the EventSource API to receive pushes. Compared to WebSocket, SSE is simpler to implement, has native browser auto-reconnection, but only supports unidirectional communication (server to client). Suitable for notifications, data push, and log streams.

An event-driven callback mechanism where a server sends an HTTP POST request to a pre-registered URL when a specific event occurs. Unlike long-lived connections, Webhooks are short-lived callbacks that require no persistent connection and scale horizontally by nature. Commonly used for payment callbacks, CI/CD notifications, and third-party event subscriptions. Requires retry logic and signature verification for reliability.

An open-source programming language developed by Microsoft that extends JavaScript with a static type system. TypeScript performs type checking at compile time, catching type errors early and improving code maintainability and refactoring safety. The frontend of this site is written in TypeScript with Astro and Tailwind CSS.

An open-source in-memory data structure store supporting strings, hashes, lists, sets, and sorted sets. Commonly used for caching, session management, message queues, and real-time leaderboards. Known for sub-millisecond read/write latency and rich data structures — the most popular caching layer solution in web applications.

A powerful open-source relational database management system known for ACID transaction support, extensibility, and SQL standard compliance. Supports JSON data types, full-text search, geospatial data (PostGIS), and window functions. The primary database choice for this site's backend services.

An API query language and runtime developed by Facebook, allowing clients to precisely specify the data structure they need, avoiding over-fetching and under-fetching. Unlike REST, GraphQL serves through a single endpoint with the client declaring required fields in the query. Suitable for multi-client aggregation and fast-iterating frontend scenarios.

A high-performance remote procedure call (RPC) framework developed by Google, based on HTTP/2 and Protocol Buffers serialization. Unlike REST's JSON text transport, gRPC uses binary serialization for higher performance, suitable for high-frequency internal service-to-service calls. Supports bidirectional streaming — a mainstream communication protocol in microservice architectures.

The secure version of HTTP, adding a TLS/SSL encryption layer underneath HTTP to ensure data confidentiality, integrity, and server identity verification. HTTPS prevents man-in-the-middle attacks, data tampering, and eavesdropping. Every production website and API should enforce HTTPS. This site uses Let's Encrypt certificates with automatic renewal.

The process of improving the internal structure of code without changing its external behavior. The core principle is small steps — one refactoring operation at a time, running tests after each step to ensure behavior is unchanged.

The strategy for identifying and managing different versions of an API. Common strategies include URL path versioning (/v1/users), header versioning, and parameter versioning. URL path versioning is recommended for its transparency.

The practice of transforming data into visual formats (charts, graphs) to make patterns and trends immediately understandable. Key principles include choosing the right chart type, maintaining a high data-ink ratio, and using colorblind-friendly palettes.

A powerful open-source JavaScript charting library by Baidu, offering 60+ chart types. Uses Canvas rendering for large dataset performance. The most popular data visualization library in the Chinese ecosystem.

A JavaScript library for producing dynamic, interactive data visualizations by binding data to the DOM. Offers maximum flexibility with a steep learning curve. Best suited for highly customized visualizations.

The ability of a newer API version to work correctly with older clients. Compatible changes include adding fields and optional parameters; incompatible changes include removing fields and changing field types. Good API design maximizes backward compatibility.

A lightweight documentation method for recording architecture decisions — context, alternatives, final choice, and consequences. ADRs are version-controlled alongside the codebase under docs/adr/, numbered sequentially. The core value is helping future team members understand "why was this chosen" months later.

A software development process with a "red-green-refactor" cycle: write a failing test (red), write the minimum code to pass the test (green), then refactor for optimization. The value of TDD is not "write tests first" — it is "think about expected behavior before writing code."

A network of geographically distributed edge servers that cache content closer to end users, accelerating content delivery. CDNs significantly reduce user latency, reduce origin server load, and provide DDoS protection. Popular CDN providers include Cloudflare, Alibaba Cloud CDN, and AWS CloudFront.

The practice of designing products, devices, services, or environments to be usable by as many people as possible, with particular focus on users with disabilities. Core frontend a11y practices include: semantic HTML, ARIA labels, keyboard navigation, color contrast (WCAG 2.1 AA requires text ≥ 4.5:1), and screen reader support.

An open-source log management solution. Elasticsearch handles log storage and full-text search, Logstash handles log collection and transformation, Kibana handles log visualization and querying. ELK indexes log content for full-text search — flexible queries but higher storage cost. Suitable for scenarios requiring full-text search and complex aggregation analysis.

An asynchronous communication mechanism for passing messages between services. Core model: producers send messages to a queue, consumers receive messages from the queue. Mainstream message queues include RabbitMQ (queue model, best for task distribution), Kafka (log model, best for event streams), and Redis Streams (lightweight, best for simple scenarios).

The process of migrating database schemas or data from one structure to another. The key to zero-downtime migration is "dual-running" — having old and new structures coexist while the application layer is compatible with both. Popular tools: gh-ost (MySQL), pgroll (PostgreSQL).

A technique for modifying table structures without stopping database service. Works by creating a shadow table, syncing incremental changes (via trigger or binlog), then atomically swapping table names. gh-ost uses binlog streaming without triggers, putting less load on the primary.

A measurable goal for system service quality, defined from the user perspective. Common SLOs include API availability (≥ 99.9%), response time P99 (≤ 500ms), and data freshness (≤ 5 min). SLOs are the foundation of a monitoring and alerting system — alert only on SLO violations.

A state where operators become desensitized due to receiving too many irrelevant alerts, eventually missing critical ones. Mitigation strategies: SLO-driven alerting (alert only on user-perceivable anomalies), dependency inhibition (auto-suppress dependent alerts), and group aggregation (merge similar alerts into one notification).

The daily quota of pages a search engine crawler will crawl for a specific site. If a site has many low-value pages (duplicate content, parameterized URLs), the crawler wastes budget on those. Management: use robots.txt and noindex to block low-value pages, focusing crawler attention on core content.

Google Quality Rater Guidelines framework. While not a direct ranking algorithm factor, content quality signals affect rankings. Improve by: attributing articles to real authors, citing authoritative sources, displaying team credentials, keeping content update dates visible, and maintaining complete contact and About pages.

A build optimization that automatically removes unused code during bundling. Relies on ES Module static analysis — at bundle time, the tool identifies and eliminates code that was imported but never called. All major 2026 build tools (Vite, Webpack, Turbopack) support Tree Shaking by default.

A technique that splits application code into multiple independent chunks for on-demand loading. The most common granularity is route-level splitting — code for each route is loaded only when the user visits that route. Vite and Webpack both automatically split dynamic imports (import()) into separate files.

A software architecture pattern where events serve as the core communication mechanism. Components do not call each other directly but communicate asynchronously by publishing and subscribing to events. EDA's core advantage is loose coupling — the sender does not need to know who consumes the event or whether consumption succeeds. Typical implementations include message queues (RabbitMQ) and event streams (Kafka).

A message model where events are durably stored in a log. Unlike message queues, events in a stream are not deleted after consumption but retained until expiry, allowing consumers to replay at any point. Apache Kafka is the representative implementation, known for high throughput and persistence.

A data persistence pattern that records state changes as a sequence of events rather than storing the current state. To get the current state, all historical events must be replayed. Advantages include a complete audit log, time travel (reconstructing any historical state), and event-driven architecture support. Drawbacks include consistency latency and event schema version management complexity.

A pattern for managing distributed transactions in microservice architectures. A Saga splits a large transaction into multiple local transactions, each publishing an event to trigger the next step. If a step fails, the Saga executes compensating transactions to roll back completed steps. Two orchestration approaches: choreography (event-driven) and orchestration (central coordinator).

A design pattern that protects distributed systems from cascading failures. The circuit breaker monitors downstream call failure rates — when the rate exceeds a threshold, it "opens" the circuit, and subsequent requests return errors immediately without actually calling the downstream service, giving it time to recover. Three states: closed (normal), open (fast-fail), half-open (probing recovery).

A fault isolation design pattern named after a ship's bulkhead compartments — one leaking compartment does not sink the entire ship. In systems, Bulkhead isolates failures by allocating independent thread pools or connection pools to different components. When one component exhausts its resources, only that component is affected.

The property of an operation producing the same result whether executed once or multiple times. In distributed systems, network glitches and retries may cause duplicate message delivery, so consumers must be idempotent. Common implementations: naturally idempotent operations (SET key = value), dedup tables (event ID unique constraint), optimistic locking (version checking), and state machines.

A flow control mechanism where consumers actively signal back to producers when processing speed cannot keep up with production rate, causing producers to slow down or pause. Backpressure is a core mechanism in Reactive Systems, preventing consumers from being overwhelmed by data floods. In Kafka, consumers control backpressure via max.poll.records and fetch.max.bytes parameters.

A technique for tracking the complete path of a request across microservices. Each request receives a unique Trace ID at the entry point, propagated via headers across service boundaries. A Trace ID chains all service calls a request passes through, enabling rapid identification of performance bottlenecks and failure points. Jaeger and Zipkin are mainstream distributed tracing systems.

A zero-downtime deployment strategy maintaining two identical production environments (Blue and Green). Only one environment serves production traffic at any time. The new version is deployed to the idle environment; once verified, the traffic gateway instantaneously switches (updating routes/load balancer), with the old environment retained as rollback capacity. The switch is instantaneous with no gradual rollout window.

A gradual rollout strategy that first deploys the new version to a small subset of instances (e.g., 5%), monitors health and error rates for a period, then gradually increases the traffic percentage to 100% if no anomalies are detected. If issues are found, traffic can be immediately redirected back to the old version. More granular but has a longer rollout cycle compared to Blue-Green Deployment.

A technique for controlling feature visibility at runtime without requiring code redeployment. Common use cases include: canary launches (enabling new features for specific users or regions), A/B testing (random group testing), and operational toggles (emergency feature disable). LaunchDarkly and Unleash are dedicated feature flag management platforms.

A system design strategy where, when some components or dependencies become unavailable, the system continues serving at a reduced functionality level rather than crashing outright. Examples: fall back to popular rankings when recommendation service is down, show placeholder images when image loading fails, postpone feature activation when payment is unavailable. Degradation strategies should be predefined during architecture design, not improvised during an outage.

A system protection mechanism that limits the number of requests allowed within a specified time window. Common algorithms: Token Bucket (allows bursts), Leaky Bucket (processes requests at a constant rate), and Sliding Window (precise counting). Rate limiting dimensions include IP, user ID, and API Key. API gateways (Kong, APISIX) typically have built-in rate limiting plugins.

A mechanism for detecting whether a service instance is functioning properly. Two types: Liveness Probe (whether the service process is still running) and Readiness Probe (whether the service can handle requests). Kubernetes uses Pod health checks to decide whether to restart containers or remove traffic. Health check endpoints typically return 200 OK or JSON-formatted status details.

An architectural pattern where auxiliary functions (logging, monitoring, proxy, configuration) are extracted from the main application into separate companion containers. The sidecar shares the same network and storage volumes as the main application but is deployed and updated independently. Service Mesh (Istio) Envoy proxies are classic sidecar implementations — application code does not need to be aware of the service mesh.

A distributed systems algorithm that selects one leader among multiple replicas to handle write operations or coordinate tasks while others remain as followers. If the leader crashes, a new election is automatically triggered. ZooKeeper, etcd, and Redis Sentinel all provide Leader Election capabilities. Algorithms include Paxos, Raft, and Bully.

An algorithm for achieving agreement on a single value among multiple nodes in a distributed system. Consensus algorithms are the foundation of distributed reliability — used in leader election, distributed locking, and atomic broadcast. The most famous are Paxos (rigorous theory, implemented in Chubby/ZooKeeper) and Raft (easier to understand, implemented in etcd/Consul).

The fundamental theorem of distributed system design stating that a distributed system can satisfy at most two of three properties simultaneously: Consistency, Availability, and Partition Tolerance. In practice, Partition Tolerance is mandatory, so the trade-off is between Consistency and Availability.

A process that moves data from source through extraction, transformation, and loading stages to a destination. Data pipelines typically involve source extraction → cleaning → transformation → loading into a data warehouse, running on batch schedules or real-time stream processing. Apache Kafka, Apache Flink, and Apache Airflow are common data pipeline tooling.

A distributed system consistency model where replicas may temporarily hold different data values, but given enough time without new updates, all replicas will eventually converge to the same state. Eventual consistency contrasts with strong consistency and is the core of BASE theory. In event-driven architectures, read models typically lag behind write models by seconds to minutes.

An architectural style oriented toward elasticity and responsiveness, following four core principles: Responsive (responding in a timely manner), Resilient (remaining available under failure), Elastic (automatically scaling with load), and Message Driven (async message passing for loose coupling and backpressure). The Reactive Manifesto defines the core philosophy of reactive systems.

In Saga patterns, an operation that undoes previously completed steps when a step fails. A compensating transaction does not "restore to original state" — other operations may have occurred in between — but rather "counteracts" the effects of the previous operations. Example: issuing a refund after a failed debit (not "reverse-debiting"), restoring inventory after order cancellation (not "changing order status"). Compensating transactions must be idempotent.

A core metric for measuring system throughput. Capacity planning typically distinguishes between average QPS and peak QPS. Average QPS = total daily requests / 86400, peak QPS = average QPS × peak factor (5-8 depending on business type). Load test targets are typically set at 3-5× peak QPS to allow headroom.

The latency value below which 99% of requests complete. P99 is more indicative of real user experience than averages — averages can be pulled down by the majority, masking a small percentage of slow requests. P99 exceeding 1000ms at more than 5% frequency should trigger alerting. Related metrics include P50 (median) and P95 (95th percentile).

A key performance metric for AI inference, measuring the time from submitting a request to receiving the first output token. TTFT directly impacts perceived response speed. For streaming output, TTFT should be kept under 500ms. Factors affecting TTFT include model size, hardware configuration, batching strategy, and KV Cache hit rate.

A key performance metric for AI inference, measuring the average time required to generate each output token. TPOT determines generation speed — a 500-token output at 10ms TPOT takes 5 seconds. TPOT is affected by GPU compute, memory bandwidth, quantization method, and batch size.

Validating system performance under expected load by simulating real user requests. The testing pyramid: component-level (single endpoint/query) → module-level (single service) → full-chain (end-to-end). Common tools: wrk (lightweight HTTP), k6 (scenario scripting), pgbench (database), vLLM benchmark_serving.py (AI inference). Must cover baseline, limit, endurance, and burst testing scenarios.

A common database performance bottleneck where all database connections are occupied and new requests time out waiting for a connection. At 350 QPS, a connection pool of 50-80 is recommended. Causes include slow queries holding connections too long, connection leaks (incorrectly returned connections), or burst traffic exceeding pool capacity. Monitoring pool utilization and wait time is key to early bottleneck detection.

AI-generated summaries displayed at the top of Google search results, automatically composed by Google's generative AI from top-ranking web content to directly answer user queries. AI Overviews cite source pages — cited sites gain high visibility but usually no clicks (zero-click search). GEO optimization makes site content more likely to be selected for citation in AI Overviews.

A class of AI information-retrieval products that directly provide answers instead of listing links, the representative being Perplexity. Unlike traditional search engines, answer engines first retrieve relevant web pages, then an LLM synthesizes a cited answer. GEO (Generative Engine Optimization) aims to get your site crawled, understood and cited by answer engines.

Optimization aimed at answer engines (such as Perplexity), targeting citations of your site in AI answers. Highly overlapping with GEO: both require structured data, bottom-line-up-front (BLUF) content, crawlable content and clear entity relationships. AEO is GEO applied to the specific channel of answer engines.

The AI crawler used by the Perplexity search engine to fetch web content as citation sources for its AI answers. Like GPTBot (OpenAI) and ClaudeBot (Anthropic), it is a crawler dedicated to AI engines. Your robots.txt should explicitly allow PerplexityBot, or your site will be completely invisible in Perplexity answers.

Enhanced fragments shown in search results, driven by structured data (JSON-LD), displaying ratings, prices, FAQ Q&A, breadcrumbs and more. Rich snippets significantly improve click-through rates, and FAQPage-type snippets are also more likely to be cited directly by AI engines — a win-win infrastructure for both SEO and GEO.

Building page structure with meaningful semantic tags (article, section, nav, header, h1-h6, time, etc.) rather than div/span everywhere. Semantic HTML helps search engines and AI engines understand content hierarchy and meaning, and is part of GEO infrastructure — AI crawlers rely on semantic tags to judge content importance even more than browsers do.

An information box in search engine result pages (Google, Baidu) summarizing an entity (brand, person, organization), sourced from the knowledge graph. Knowledge panels enhance brand authority and are usually tied to Wikipedia, Organization Schema on the official site, and authoritative media citations — part of E-E-A-T signals.

A search where users obtain the complete answer directly on the results page or from an AI answer, without clicking through to any website. AI Overviews and answer engines amplify this trend — many informational queries no longer generate traffic. For websites, the counter-strategy is getting brand names and core value propositions cited in AI answers, turning "zero clicks" into brand exposure.

OpenAI's AI crawler for fetching web content, used for training and inference. Its User-Agent contains "GPTBot" and can be allowed or blocked independently via robots.txt. ChatGPT's browsing feature also uses OAI-SearchBot to fetch real-time content. Monitoring GPTBot crawl volume is the first-layer signal for measuring GEO effectiveness on the OpenAI side.

The crawler used by OpenAI's search engine (ChatGPT Search) to fetch real-time web content as citation sources for answers. Unlike the training-oriented GPTBot, OAI-SearchBot serves retrieval during inference — its GEO value is more direct, since being crawled by it means a chance to appear in ChatGPT's cited answers.

The crawler used by Anthropic's Claude AI assistants to fetch web content for training and inference. Claude's search products (Claude-SearchBot, Claude-User) also fetch real-time content. User-Agent contains "ClaudeBot"; along with GPTBot and PerplexityBot, it belongs on the AI crawler whitelist that should be explicitly allowed.

A dedicated crawler identifier Google uses to train generative AI models such as Gemini, independent of regular Googlebot. Site owners can control Google-Extended access separately in robots.txt. Observing Google-Extended crawl data in Search Console provides a proxy for how much of your content is being read by Google's AI side.

The crawler protocol file at a site root, declaring which User-Agents may crawl which paths. A common GEO pitfall is reusing old block rules that accidentally block AI crawlers (e.g., blocking the User-Agent segment containing GPTBot/PerplexityBot). GEO infrastructure requires explicitly allowing all major AI crawlers while keeping the sitemap reference.

Google's official tool for monitoring site indexing and search performance. In GEO scenarios: filter the performance report by Googlebot and Google-Extended to see which URLs are crawled and their status codes, providing a proxy for the scale and quality of content being read by the AI side.

Tracking parameters appended to URLs (e.g., ?utm_source=perplexity&utm_medium=organic) that label traffic sources in analytics tools. The standard GEO attribution practice: add UTM parameters to landing-page links appearing in AI answers, then measure visits and conversions from AI channels like ChatGPT and Perplexity.

The overall degree to which a website is crawled, understood and cited in AI engine answers. The GEO counterpart to SEO "keyword rankings". AI visibility is measured through a four-layer pipeline: crawler volume (server logs) → index coverage (Search Console) → citation (brand tests and business-term tests) → attribution (AI-source traffic and leads).

A set of test cases with gold answers used to evaluate AI system quality. The correct way to build one: hand-label a seed set of 50-100 cases covering main scenarios (stratified sampling, double labeling) → after launch, feed 10-20 real failure cases back from production logs weekly → forming a regression-prevention loop. The eval set is the foundation of the "offline technical quality" layer: run regression before every change, and releases that fail regression cases do not ship.

The core evaluation metrics for classification and extraction tasks. Precision = share of positive predictions that are actually positive; Recall = share of actual positives that were predicted; F1 is their harmonic mean. With imbalanced classes, accuracy alone is meaningless — you must look at the Precision/Recall combination.

An open-source evaluation framework for RAG projects with four metrics: Context Precision (share of retrieved content that is relevant), Context Recall (did it retrieve what it should have), Faithfulness (is the answer grounded in the retrieved context — hallucination control), and Answer Relevancy (does the answer address the question). Only the four metrics together localize the problem to the retrieval or generation layer — a single accuracy number tells you nothing.

The share of AI answers containing unsupported or factually wrong content — the key quality metric for RAG/agent projects. Measurement: annotators (or an independent evaluation model) judge each answer in the eval set for factual errors and tally the share. Hallucination rate is compressed by "evidence chains + review + human fallback"; it cannot be eliminated with prompts.

Using a large language model as a grader that scores answer quality against a rubric, for generative tasks that rules cannot judge automatically. Practice points: ① the rubric must be operational; ② the judge model should differ from the evaluated model to avoid same-source bias; ③ periodically sample and manually verify the judge's stability.

An experiment method that randomly splits users and runs new and old versions concurrently, comparing key metrics — the gold standard for validating AI system launches. The AI pitfall: the cold-start period (indexes and user behavior still settling) causes metric noise, so run at least 2-4 weeks before concluding; and watch quality and cost on both sides, lest "cost dropped but quality dropped too".

A mechanism where the live version is the Champion, and a new model/prompt runs as a Challenger in parallel gray release, compared by A/B metrics — only switch when the challenger clearly wins. It prevents "changed and made it worse": every AI system change should first run as a challenger instead of replacing the live version outright.

The phenomenon where changes in input distribution or retrieved content silently degrade model quality over time. AI system quality is dynamic after launch: user phrasing changes, knowledge-base updates, and business rule changes all trigger drift. Countermeasures: a baseline metric report, monitoring input distribution shifts, and regularly re-running the eval set.

A mechanism where high-risk steps in an AI workflow (orders, deletions, large decisions) require human confirmation before execution. The intervention rate is a key operational metric: too high means the AI saves no labor, too low means risk is out of control. It ties directly to ROI — human fallback cost must be counted in the AI project's total cost.

A method of accounting cost and benefit per unit of processing — the hard standard for whether AI is worth doing: total cost per AI-processed unit (tokens + fallback labor + amortized development) must be lower than cost per manually processed unit. Projects that "saved people but saved no money" usually have an intervention rate so high that fallback labor eats the entire benefit.

The most common Approximate Nearest Neighbor (ANN) index algorithm for vector search. Builds a multi-layer graph: coarse navigation at the top, precise search at the bottom. Queries enter from top to bottom with O(log N) complexity. Two parameters control the precision-speed trade-off: ef_construction and M. Higher M improves recall but increases index size — start with M=16. HNSW builds slowly but queries extremely fast — ideal for write-once-read-many online retrieval.

A classic vector search index algorithm. Uses K-Means clustering to partition the vector space into nlist cells; queries search only the nearest nprobe cells. IVF builds much faster than HNSW but queries are slower. Suitable for batch ingestion, offline retrieval, or resource-constrained scenarios. The nlist parameter controls the number of partitions: more partitions means slower build but more accurate query.

A measure of directional similarity between two vectors, range [-1, 1]. Values closer to 1 indicate more aligned directions. In vector search, Cosine Similarity is equivalent to inner product (IP) after vector normalization — the most common distance metric. Mainstream embedding models (bge-m3, text-embedding-3) default to Cosine Similarity.

The fundamental paradigm of vector search — does not guarantee the absolute nearest neighbor, but trades a small recall loss for orders-of-magnitude speed gains. Compared to brute-force (exact) search, ANN is 100-1000× faster on million-scale datasets while maintaining 90-99% recall. HNSW, IVF, and DiskANN are all concrete ANN index implementations.

A vector compression technique that splits high-dimensional vectors into sub-vectors, clusters each sub-space independently, and stores cluster IDs instead of original vectors. Compression ratios of 4-32× dramatically reduce memory usage and query distance computation cost. The trade-off is a slight recall loss (typically 1-3%) — suitable for resource-constrained large-scale deployments.

A disk-based ANN algorithm proposed by Microsoft. Stores the vector index on SSD rather than in memory, with only the graph structure (roughly 1/10 of the raw data) in RAM — dramatically reducing single-node deployment cost. Supports billion-scale vector search with recall close to in-memory indexes. Ideal for massive datasets with limited memory budgets.

Retrieval using embedding models to map text into dense vectors and search by semantic similarity in vector space. Complementary to Sparse Retrieval: dense retrieval excels at "asked a different way" semantic recall but is less stable with proper nouns, IDs, and abbreviations. Production RAG typically combines both in hybrid search.

A retrieval method based on term frequency statistics, represented by BM25. Unlike Dense Retrieval's semantic matching, sparse retrieval performs exact term matching — contract clause numbers, product model IDs, and name abbreviations that dense vectors blur are precisely hit by sparse retrieval. The standard production RAG practice is dense + sparse hybrid search for complementary gains.

The core evaluation metric for the vector search / RAG retrieval layer: given a query, the proportion of correct answers appearing in the Top-K results. K is typically 5/10/20, used with MRR (Mean Reciprocal Rank). Healthy baseline: Recall@10 ≥ 80%. Below that, fix retrieval before touching generation. Ensure the eval set does not overlap with the model's training data, or metrics will be inflated.

A technique for narrowing the vector search scope by metadata (source, date, permissions, type, etc.). Two modes: Pre-filtering — filters at the index level before vector search, represented by Qdrant's payload index, maintains stable recall under strict filters; Post-filtering — vector search Top-K first, then filter results, the default behavior of pgvector and most vector DBs, risks empty result sets under strict filters. Pre-filtering is the more reliable production approach.

The first layer of a dialogue system: determining what the user wants (order lookup, refund, complaint). Implementations range from rules/keywords and small-model classification to large-model zero-shot classification. In AI customer service, intention recognition feeds graded routing — high-frequency standardized questions get automated, high-risk ones go to humans. It is what lets the AI operate freely.

Graded routing that dispatches each message by intention level: high-frequency standardized questions (order lookup, address change, invoicing) go fully automated; mid-frequency semi-standard requests (refunds, exchanges) are handled by AI with human confirmation for key actions; high-risk/complaint cases go straight to humans. The value: 70% of high-frequency questions get automated safely while the 10% of high-risk cases stay with humans. It is Layer 1 of the four-layer AI customer service architecture.

The mechanism for transferring a conversation to a human agent when the AI cannot handle it or the scenario is high-risk. Three requirements: handoff must carry full context (conversation summary + attempted solutions + user info) so the user never repeats themselves; handoff labor must be counted in the AI project's total cost (otherwise you "saved headcount but saved no money"); the handoff rate is a double-edged sword — too high means the AI saves no labor, too low means risk is uncontrolled.

The key operational metric of AI customer service: the proportion of conversations requiring human takeover or confirmation. Industry reference: 10-20% for simple Q&A bots, 30-40% is normal for complex business support (after-sales, complaints, multi-step requests). Together with task success rate and cost per ticket, it forms the three numbers to watch after launch — and the basis for deciding whether to advance to the next phase in progressive deployment.

The technology area that enables computers to understand the meaning of human language, including intention recognition, slot filling, entity extraction, and sentiment analysis. Traditional dialogue systems centered on NLU; in the LLM era much of NLU capability has been absorbed into large models, but "intention recognition + graded routing" remains the key design of production AI customer service.

The multi-turn dialogue process of progressively collecting the key information (slots) needed to complete a task. For example, an address-change task requires two slots: order number and new address — the AI fills them through follow-up questions. Design points: actively ask when a slot is missing instead of guessing; sensitive operations (e.g., changing the payout account) still require human confirmation even after slots are filled.

The technique for judging the emotional tendency of text (positive/neutral/negative). In AI customer service it is used for risk control: when the customer is detected as agitated or complaint-inclined, immediately switch strategy — stop standard scripts and prioritize human handoff, preventing the AI from misspeaking in a reputation-sensitive scenario. Sentiment is a key signal for high-risk interception.

The core quality metric of AI customer service: the proportion of user-initiated tasks (order lookup, refund, booking) that are ultimately completed successfully. Different from handoff rate — a low handoff rate does not mean a high task success rate; the AI may be giving irrelevant answers and the customer gives up. Measure at the task granularity (not message granularity) and distinguish "completed by AI alone" from "completed by AI + human collaboration."

The engineering process of moving production workloads from closed-source APIs to open-weight models or additional providers. The 2026 trend: OpenRouter data shows US enterprise token share for Chinese open models rose from 4.5% in H1 2025 to 30%+ (peak 46%). Correct process: build an eval baseline (score old vs new models on real samples) → stand up a multi-provider gateway (unified protocol + routing + circuit breaking and fallback) → phased canary rollout (endpoint by endpoint) → cost attribution. The precondition is "you can switch back at any time" — a gateway drives switching cost toward zero.

A model whose weights are publicly downloadable and can be self-hosted (e.g., DeepSeek, Qwen, GLM, Kimi). The essential difference from closed APIs: data never leaves the enterprise intranet and you are not hostage to vendor rate limits, price hikes, or version churn — but you must run and maintain your own inference infrastructure. One of the three drivers of the 2026 enterprise migration wave: 60%-90% cheaper, capable enough (91% of engineers do not need peak performance), and deployment freedom.

An intermediary layer that unifies access to multiple model providers: business traffic uses the OpenAI-compatible protocol throughout, while the gateway handles routing (dispatch by endpoint/task/user tier), rate limiting, circuit breaking, and automatic fallback. Value: no business interruption when a single provider fails or raises prices; the "rollback" precondition for model migration; the landing point for tiered multi-model selection. Difference from API Gateway: the former is purpose-built for model providers and includes model routing and cost attribution.

The compliance requirement that data storage and processing must reside within a specific geographic or legal jurisdiction. The key constraint in enterprise AI adoption: when data cannot leave the network (intranet/border), public APIs are off the table and self-hosted open-weight models are required. The first factor deciding "API vs self-host" — for finance, healthcare, and government it outranks all cost considerations.

A production-grade open-source LLM inference serving framework with an OpenAI-compatible API out of the box. Two core mechanisms: PagedAttention (page-based KV Cache management — higher VRAM utilization and more concurrency) and Continuous Batching (new requests join in-flight batches anytime; short requests finish first). Officially supports mainstream open models (Qwen/DeepSeek/Llama) and is the default choice for production self-hosted inference. Key launch flags: --gpu-memory-utilization (default 0.9), --max-num-seqs (effective concurrency cap), --enable-prefix-caching.

The mechanism that caches attention Key/Value tensors of historical tokens during Transformer inference — generating each new token only requires a dot product against the full history, avoiding repeated forward passes. VRAM formula: 2 × layers × KV heads × head dim × bytes per token × context tokens. It is the VRAM hog of self-hosted deployment (the main usage besides weights) and the determinant of the concurrency ceiling; PagedAttention manages it in pages to reduce fragmentation.

Dynamic batch scheduling where the inference engine admits new requests into in-flight batches as completed ones leave — short requests finish and exit first while long requests continue. Compared to static batching (waiting for a full batch, where long requests block short ones), GPU utilization rises significantly; it is the core mechanism behind the throughput lead of production engines like vLLM. Caveat: when VRAM is tight, effective concurrency is capped by the KV Cache limit and batching has no room to work.

The KV Cache memory management technique introduced by vLLM: like OS paging, KV Cache is split into fixed-size pages and only the actually used portion is allocated, sharply cutting fragmentation and waste and letting far more concurrent requests fit on one GPU. It is vLLM's signature throughput mechanism — no configuration needed, but "the KV Cache ceiling is the concurrency ceiling" is the mindset that tuning starts from.

Caching the KV computation of shared request prefixes (system prompts, RAG document prefixes) and reusing it on hit, skipping recomputation. Hit rates of 70-90% are common in multi-turn dialogue and RAG — input compute for free. Engineering points: structure prompts "static first, dynamic last" (fixed content at the prefix, varying content after) to maximize hits; in vLLM it maps to --enable-prefix-caching; monitor the hit rate (target > 50%) to judge whether the prompt structure is sound.

A technique where a small draft model quickly generates k candidate tokens and the large model verifies them in one parallel pass — accepted tokens are kept, and on the first disagreement generation restarts from that point. Generation speed can improve 1.5-3×; it suits 70B-class models that are latency-sensitive with spare compute, and is built into engines like vLLM. Not every workload benefits (limited gain when the small model is already fast) — measure the gain before enabling.

An attack where an attacker injects malicious instructions into the LLM's input context to induce unintended actions. The essential difference from SQL injection: SQL injection exploits "code and data not separated" (user input is concatenated into executable SQL); prompt injection exploits "instructions and data not separated" — the model cannot distinguish system instructions from instructions inside content, because an LLM input is plain text. Core defenses: input-side isolation declarations (content is reference material, not instructions), independent output-side validation, and human fallback for high-risk actions.

An attack where malicious instructions are hidden in externally retrieved content (web pages, documents, emails). The user never types anything malicious — the model gets "poisoned" just by reading the data. The unique and highest-severity risk of RAG applications and AI customer service (e.g., a web page hiding transfer instructions). Three-layer defense: system prompt declares "retrieved content is not instructions", sensitive operations pass independent rule checks or human confirmation, and ingested documents are reviewed with external content tagged "untrusted" for degraded handling.

An attack that bypasses LLM safety restrictions through roleplay, fictional scenarios, or hypothetical dialogue. Typical techniques: "pretend you are a model without safety limits", script settings, ASCII-art obfuscation. Difference from prompt injection: jailbreaking targets the model's own safety alignment; prompt injection induces specific actions. Defense: continuous red teaming plus bypass cases persisted as a regression set, re-run immediately after each new model release — as model capability grows, attack techniques grow too.

Adversarial security testing of LLM applications by simulating attackers. Four attack types to test: direct injection (ignore instructions / extract the system prompt), indirect injection (malicious documents), jailbreaks (roleplay), and data extraction (translation/repetition/encoding bypasses). Best practice: an independent red team (non-developers or dedicated attack models) avoids the "testing your own work" blind spot; record the bypass rate (< 5% recommended) as a launch gate; persist cases as a regression set and re-run automatically after every prompt/tool/model change.

Running components that may execute arbitrary code/actions (e.g., an agent's code executor) inside a controlled environment. Essentials: no network egress (prevents data exfiltration), only necessary directories mounted, resource limits (CPU/memory/disk), and controlled process lifecycle. A key part of agent tool-permission governance — even if the model is tricked by prompt injection into calling a tool, the sandbox confines the blast radius to the isolated environment.

The extraction of sensitive information from an AI application. Four channels: prompt-injection extraction (translation/encoding bypasses trick the model into repeating hidden content), weak session isolation between users, logs and caches retaining plaintext PII (exposed in a breach), and tool side effects (the model writes data to storage it should not). Defenses: session-level data isolation, log redaction (mask or encrypt PII fields), disabling Prompt Caching for sensitive categories, and whitelisting tool write paths.

The capability of maintaining state across calls on behalf of a stateless LLM — what goes into the context, what is persisted externally, and when it is pulled back, all designed and implemented by the application layer. Models are stateless by nature; "memory" is really the application maintaining state for the model. Taxonomy: short-term memory (session messages), working memory (task state), long-term memory (cross-session user profiles and conclusions). Good design is what turns an agent from a "single-turn Q&A machine" into a "workflow that works continuously".

The raw message sequence of the current session, living inside the LLM context, invalidated when the session ends. Engineering: appended automatically before each model call, with context-window trimming (keep only the most recent N turns) or rolling summarization to control token usage. Loss consequence: "amnesia" mid-session and incoherent answers. Together with working memory (task state) and long-term memory (cross-session profile) it forms the three-layer agent memory model.

The state of an in-flight task — executed steps, intermediate results, to-dos. Its lifecycle is bound to the task: write the goal at task start, write results at each step completion, write the reason on failure; archive when the task completes or times out (e.g. 24h). Engineering: Redis Hash or a database table keyed by task_id, with crash recovery support. Loss consequence: an interrupted task cannot resume and must start over.

Cross-session user profile, preferences and historical conclusions. Only "conclusion-level information" is stored (explicit user preferences, post-task summaries, facts reused across sessions); process-level information never enters the store to avoid noise. Storage: semantic memories go to a vector DB (Qdrant/Milvus/pgvector) as summarized vectors retrieved on demand; structured facts go to PostgreSQL keyed by user_id. Loss consequence: the agent reintroduces the user every session — poor experience.

Dialogue compression: once a conversation passes a threshold, old turns are rolled into a summary placed in the context, keeping "conclusions" rather than "process". Key points: trigger proactively instead of every turn (e.g. at 20 accumulated turns or 70% context consumption) to avoid pointless model calls; summaries inherently lose detail over time. Combined with truncation (drop the oldest) and retrieval (recall on demand), it is the default solution for controlling token cost in long conversations.

The technique of recalling memory fragments from the storage layer by relevance and injecting them into the context at answer time. Trigger points: pull the user profile at session start, pull relevant historical conclusions at task start, trigger vector retrieval when needed information is missing from context. Key points: conflicting retrieved memories follow "newer wins + proactively confirm with the user" — never let the model guess; retrieval relevance ≠ importance, so hybrid retrieval or reranking may be required.

Forgetting strategies — a memory system that actively "forgets": short-term memories are cleared when the session ends (TTL); working memories are archived when the task completes or times out; long-term memories not retrieved for N days are downweighted or expired; when a user corrects a preference, the old memory is marked invalid rather than coexisting (otherwise the model sees contradictions every time). Compliance requirement: user-requested deletions must actually be deletable (delete API plus a user-visible "clear memory" entry).

Row-level isolation of memories by user_id in multi-tenant scenarios — A user's memories must never be retrievable by B, the security baseline. Supporting governance: sanitize on write (PII detection — IDs, phone numbers, bank cards are redacted or encrypted on hit; plaintext never enters the storage layer), store only summaries rather than raw messages in long-term memory (shrinking the leak surface), and provide a delete API for compliance.

A high-quality, human-annotated evaluation sample set with "expected answers" — the foundation of trustworthy LLM evaluation. Three building principles: sample from real traffic rather than writing your own examples (self-written samples tend to be "designed to pass"); align annotators before scaling (double annotation + dispute arbitration, otherwise disagreement becomes evaluation noise); keep expanding (every production issue and every customer complaint becomes a sample). Eval-set quality caps evaluation credibility — if the eval set is biased, every downstream metric is self-deception.

An evaluation subset made of reproduction samples from historically fixed issues, used to prevent rollbacks — "the bug I fixed last week is back". How to build: fold the reproduction sample into the regression set every time a production issue is fixed; auto-rerun it after every prompt/model/tool change and block rollbacks immediately. Regression testing must be automated — remembering "what we broke last time" by hand has zero productivity. Together with the capability set (baseline abilities) and the adversarial set (boundary/malicious inputs), it forms the three-layer eval set.

An evaluation subset made of boundary/malicious inputs (over-length, typos, inducement, jailbreaks) that tests LLM application robustness under extreme conditions. Sources: red-team output + collection of anomalous production inputs. With the capability set (normal-scenario baselines) and the regression set (historical-issue protection), it completes the eval-set trio. It is the intersection of security evaluation (red teaming) and quality evaluation — the four attack categories in LLM security practice can be folded directly into adversarial-set samples.

The phenomenon where eval samples appear in model training data, inflating metrics — the model has "seen the answers" and results are untrustworthy. Mitigations: ① keep refreshing the eval set with fresh real data, avoiding overused public-dataset samples; ② evaluate with online data published after the model's training cutoff; ③ build your own eval set for sensitive or unique scenarios; ④ periodically ask the same question rephrased to check for memorization. A contaminated eval set is more dangerous than no evaluation — it provides false confidence.

One of the common LLM-as-a-Judge biases: the judge tends to score answers at a fixed position (e.g. the first) higher, regardless of content quality. Fix: rotate answer order and average two comparisons, or use single-sample scoring instead of pairwise comparison. When calibrating a judge, check position bias together with self-preference bias (favoring outputs that "sound like itself" — fix with a rubric instead of open-ended "which is better") and verbosity bias (longer answers score higher — fix with per-dimension scoring or length normalization). Human agreement ≥ 85% counts as calibrated.

One of the common LLM-as-a-Judge biases: the judge model favors outputs matching its own style — stylistically similar but factually worse answers score higher. Fix: score against a rubric by dimension instead of open-ended "which is better" comparison; cross-validate with a different judge model when needed. Alongside position bias and verbosity bias it forms the judge's three blind spots; the unified calibration method is measuring agreement against 30-50 human-labeled samples, and switching to a stronger judge or keeping human spot checks if it falls below 85%.

One of the common LLM-as-a-Judge biases: longer answers score inflated, even when redundant or off-topic. The cause is the judge's perception of "informativeness" correlating strongly with length. Fix: score per dimension (relevance / completeness / conciseness), or normalize by length; alternatively state "conciseness gains points, redundancy loses points" explicitly in the rubric. In practice it is calibrated together with position bias and self-preference bias — the standard checklist for judge trustworthiness.

Explicit scoring dimensions and rules given to LLM-as-a-Judge (e.g. relevance 0-5, completeness 0-5, conciseness 0-3), replacing open-ended "which is better" comparisons. Benefits: ① reduces self-preference bias and verbosity bias — with dimensions and anchors, scoring becomes reproducible; ② makes results explainable — clients and teams can see "why this version is better than the last"; ③ enables calibration — compare rubric output against human annotations to quantify judge agreement. A rubric is what turns a judge from "scoring by feel" into an engineering metric.

The stage of continuing model training on "instruction-expected answer" paired data — the main battleground of fine-tuning. Data-volume reference: 500-5,000 high-quality pairs produce clear gains; quality beats quantity by far (1,000 carefully cleaned pairs beat 50,000 crawled ones). Source priority: real traffic > synthetic data + human cleaning > generic public data. Before/after evaluation is mandatory after training — otherwise fine-tuning is alchemy, not engineering.

The mainstream parameter-efficient fine-tuning method: the original weights are frozen and only a small set of injected low-rank adapter matrices is trained (typically <1% of parameters). A single 24 GB consumer GPU can fine-tune 7B-32B models with near full-parameter quality — and because the change footprint is small, catastrophic-forgetting risk is inherently lower. It is the de facto default in 2026; full-parameter fine-tuning has a higher ceiling but much higher cost and risk — do not reach for it without strong reasons.

LoRA fine-tuning applied on a 4-bit quantized model, letting a single 12-16 GB GPU fine-tune 7B-14B-class models — the default starting point for small teams and resource-constrained scenarios. The cost is slightly lower training quality than full-precision LoRA. Selection order: start with QLoRA when VRAM is tight, and only upgrade to LoRA or full-parameter if results are insufficient.

Fine-tuning on "instruction-answer" formatted data to teach a model to follow instructions and align output format — the dominant form of SFT. The core is data: instructions should mirror real user phrasing (not self-written examples "designed to pass") and answers should include format samples. Difference from domain fine-tuning: instruction tuning teaches "how to do the job", domain fine-tuning teaches "domain expertise" — production practice usually mixes both.

The alignment training stack that uses human preference data to make model outputs more aligned with human expectations (helpful, harmless, honest). Three steps: SFT baseline → reward model training (human ranking/scoring) → PPO reinforcement learning. High cost and engineering complexity — enterprise fine-tuning generally stops at SFT; the RLHF layer is usually done by model vendors. Self-run RLHF needs strong justification (e.g. strict value-constraint scenarios).

The fine-tuning side effect where a model improves on new data while degrading on existing general capabilities (arithmetic, common sense, other-domain Q&A). Three standard mitigations: ① data mixing — blend 10-20% general data into the fine-tuning set so the model "reviews old knowledge while learning new"; ② parameter-efficient fine-tuning — LoRA/QLoRA change little, so forgetting risk is naturally lower; ③ evaluate after training — before/after comparison on the same eval set; if general scores drop too much, fix the data mix before shipping.

Training data generated by large models, used to bootstrap scenarios without labeled data. Standard practice: have a large model batch-generate "question-answer" pairs for your business scenarios, then spot-check and clean them; real samples can also be paraphrased to expand coverage. Watch the self-loop risk — synthetic data fed back into models regenerates more synthetic data, and errors compound generation after generation; human quality gates are mandatory, never train on it directly.

The compression technique of training a smaller model (student) on the outputs of a larger model (teacher) — easy to get backwards: the student learns from the teacher. Common uses: distilling flagship-model capability into a 7B-class model for low-cost routing, offline classification, or self-hosted deployment. Caveats: the distilled model inherits the teacher's biases and hallucination patterns — evaluation cannot be skipped; and when the capability gap is too large, distillation gains are limited.

Corporate data leaving your controlled environment and being accessed or processed by an external provider (cloud vendor, AI model provider). Rule of thumb: if data passes through a third-party server at any stage — transit, storage, or processing — it counts as cross-border. Common cases: calling public-cloud AI APIs, SaaS AI tools, outsourced AI training/fine-tuning. Non-cross-border options: private deployment, local inference, federated learning.

Information recorded in electronic or other form relating to an identified or identifiable natural person, including names, ID numbers, phone numbers, biometrics, and location data. PIPL requires: cross-border personal information transfer requires a standard contract, certification, or security assessment; processing must follow the minimum-necessity principle; sensitive personal information (financial accounts, biometrics, health) requires separate consent. Common AI risks: training data, conversation logs, and user feedback may contain personal information — de-identify before cross-border transfer or use.

Data that, if tampered with, destroyed, leaked, or illegally obtained or used, could endanger national security, economic operation, social stability, public health, or safety. Typical scope: undisclosed financial data, customer lists, technical documents, commercial strategy, infrastructure operational data, population health data. Cross-border transfer of important data requires a CAC security assessment; transfer without assessment is prohibited. Difference from national core data: core data is in principle not allowed to leave, important data may leave after assessment.

Technically deforming, replacing, masking, or pseudo-anonymizing personal or sensitive information in data to reduce leakage risk. Common techniques: substitution (phone number 138****1234), truncation (first 6 digits of ID), hashing (irreversible mapping), generalization (age range instead of exact age), perturbation (adding noise to values). Note: de-identification is not anonymization — de-identified data can still be re-identified through correlation analysis; cross-border scenarios require residual-risk assessment. Difference from anonymization: anonymization is irreversible; de-identification is reversible or can be correlated back.

A distributed machine learning paradigm: multiple parties train models locally and exchange only model parameters (gradients); raw data never leaves local storage. Core advantage: data does not cross borders, satisfying compliance requirements; multi-source data collaboratively trains, improving model generalization. Three architectures: horizontal federation (different samples, same features), vertical federation (different features, overlapping samples), federated transfer learning (different samples and different features). Limitations: high communication overhead, high coordination cost, model parameters may still carry information about training data (gradient leakage attacks); use with secure aggregation and differential privacy.

The system requiring enterprises providing algorithmic recommendation services or generative AI services to the public to submit filing materials to the CAC. Algorithm filing is governed by the Provisions on the Administration of Algorithmic Recommendation Internet Information Services; generative AI service filing is governed by the Interim Measures on the Administration of Generative AI Services. Filing materials include: algorithm safety assessment report, data-source description, content-filtering mechanism, user-rights protection plan. Review cycle is 2-3 months; public-facing services cannot launch without filing. Enterprise internal AI use is currently not mandatory, but public-facing generated content must be labeled.

Filing specifically for generative AI (large language models, image generation, video generation, etc.) offered to the public, governed by the Interim Measures on the Administration of Generative AI Services. Parallel to algorithm filing, but with emphasis on generative AI-specific risks: generated content labeling, false-information prevention, training-data provenance compliance. Lists of approved models are on the CAC website. Using an already-filed model simplifies the process; self-developed or fine-tuned models require separate application.

Embedding an identifiable "AI-generated" mark into AI-generated content (text, images, video, audio) to satisfy compliance requirements. Three approaches: ① explicit labeling (watermark/badge) — intuitive but affects visuals; ② implicit labeling (steganography/metadata) — invisible but requires dedicated tools to detect; ③ structured labeling (API response field) — suitable for programmatic consumption. 2026 compliance requirement: content facing the public must be identifiable; explicit + implicit belt-and-suspenders is recommended. Relationship to watermarking: watermarking is one implementation of explicit labeling; steganography is one implementation of implicit labeling.

Role-Based Access Control: permissions are granted to roles (e.g. dept:hr, role:executive) and users inherit them via role membership, instead of per-user grants. The workhorse permission model for RAG knowledge bases — doc metadata stores a list of visible roles (acl), and at retrieval the user’s roles are intersected with acl. Fits organizations with clear departments/levels; coarser than ABAC but cheapest to implement and audit.

Attribute-Based Access Control: permissions are decided at runtime by attribute policy expressions (user attrs × resource attrs × environment attrs), e.g. "doc.sensitivity <= user.clearance AND same department". More expressive than RBAC (conditional logic) but policies are costlier to write, debug, and audit. RAG guidance: start with two levels — RBAC + sensitivity metadata — and evolve to ABAC only when a real conditional requirement appears; avoid over-design.

Access Control List: the list of principals allowed to see a resource, attached directly to the resource (document). In RAG it is typically a vector-store metadata field (e.g. acl: ["dept:hr", "role:executive"]). Design points: ① store IDs, not names (a department rename should not trigger a rebuild); ② flatten inherited permissions at index time; ③ keep temporary grants in a separate grant table with TTL — never pollute acl.

The principle that every principal gets only the minimal set of permissions needed to do the job. Three layers in RAG: ① data layer — doc acl scoped to the smallest visible audience; ② application layer — retrieval permission params come only from the server-side session, never from user input; ③ model layer — no document list in the system prompt, no "memory" beyond the retrieved context. Closed loop completed by audit logging (the permission decision recorded on every retrieval).

Row-Level Security: a database-layer permission mechanism where rows in the same table have different visibility for different users (e.g. PostgreSQL RLS policies). In RAG, vector-store metadata filtering is "row-level security for vectors" — chunks of different users are isolated inside the same collection by metadata predicates. ES document-level security and pgvector + RLS are two typical implementations.

Physical isolation in a vector store: partition data into separate collections (or namespaces) per tenant / user group / sensitivity tier. Shared docs are stored once per partition — write cost rises and permission changes must sync N copies, but isolation is strongest: cross-partition data leakage is architecturally impossible. Standard practice for multi-tenant SaaS knowledge bases and high-compliance industries; Milvus partitions and ES indices both fit.

The mechanism that propagates permission changes from the source document system (Confluence / SharePoint / Feishu) into vector-store metadata. Recommended: event-driven (webhook/MQ → incremental acl metadata update) plus a daily full reconciliation as a safety net. Iron rules: ① incremental updates, never reindex; ② fail closed on sync errors (downgrade visibility) — miss a recall rather than over-authorize; ③ add real-time re-authorization (call the source authz API after a retrieval hit) for high-risk docs. Index once and never update permissions = a ticking time bomb.

Financial operations practice for AI workloads: extends cloud FinOps (resource layer) to the token-billing layer; the core is "cost must be measurable, cap-able, alert-able". Three parts: ① metering — the gateway logs model/key/scenario/task_id/tokens per call, never trusting client-reported counts; ② unit economics — Cost per Task instead of $/request for business cost accounting; ③ control — Token Budget hard caps + Cost SLO + anomaly detection. Maps to the classic FinOps pillars: Inform (visibility) → Optimize (the four cost levers) → Operate (budget & alerts run like an SLA).

A monthly hard token cap set per scenario × tenant/project, metered and accumulated in real time at the gateway; 80% triggers a warning, 100% executes the on_exceed policy (degrade to small model / reject / approval). Iron rules: ① dual gates — scenario-level (fine-grained degrade) plus project-level (master breaker); with only one, a bug in one scenario can burn the whole project; ② metering at the gateway, never client-reported token counts; ③ budget values must carry 3× traffic headroom — first-week bill volatility far exceeds estimates.

The core unit-economics metric for AI projects: Cost per Task = Σ(cost of every model call in the task chain) / business tasks completed. The denominator is "business completed" (one closed ticket, one produced summary), not API request count; human-rescued tasks count as completed with the human cost included. Only this metric answers "pre-AI (human cost) vs post-AI (model + ops + fallback) — which is cheaper?", and exposes the classic trap where the cheaper model is actually more expensive due to lower success rate. Companions: Token Efficiency (tasks per million tokens) and Quality-Adjusted Cost (cost ÷ quality score, preventing "cost cut = quality cut" self-deception).

The vendor’s offline batch inference API: submit a batch request file, results returned within 24h, unit price typically 50% of the realtime API. Fits every non-realtime scenario (summarization / data labeling / bulk generation / eval runs). Engineering notes: ① the business side needs "completion callback + timeout requeue" (24h is an SLA, not a guarantee); ② split batch files by scenario — never mix high- and low-priority jobs (a file returns together); ③ mark batch unit prices separately in cost accounting, or Cost per Task comes out wrong.

Automatic switching when the primary model fails or is downgraded, in three senses: ① error fallback — on timeout / rate-limit / 5xx, switch to a backup provider or model for availability; ② budget fallback — on Token Budget on_exceed, drop from flagship to small model without interrupting business; ③ quality fallback — in cascade routing, escalate to the large model when the small model’s confidence is insufficient (the reverse, "escalation fallback"). Iron rule: every fallback hop needs its own independent timeout (cascading waits amplify latency), and fallback events must land on the monitoring dashboard — frequent fallback is a leading indicator of primary-model pricing / quota / quality problems.

Cost allocation and settlement when multiple teams/projects share AI capability. Three tiers: ① <5 teams — one shared budget + per-scenario cost reports (no attribution); ② 5-20 teams — project-level API keys + gateway metering per key + independent budgets + overage approval; ③ 20+ teams (AI platform) — internal pricing settlement (settlement price = vendor cost × 1.1-1.3, covering ops amortization). Iron rule: attribute to "scenario", not "team" — one key mixing functions never adds up; the gateway must require a scenario tag.

The internal thinking tokens of o1/R1-class reasoning models, generated before the answer: billed at the output rate, but invisible in content and not directly controllable in quantity (the model decides how deep to think). Cost risk: the same question can cost 40× depending on 200 vs 8,000 thinking tokens, and the bill only shows total output_tokens. Management: ① restrict usage — only tasks genuinely needing deep reasoning go to reasoning models; ② monitor the reasoning-token share (available separately in the API usage response); ③ mis-routing simple tasks to reasoning models is the #1 cause of bill anomalies — give it a dedicated anomaly-detection rule.

Treating cost as an SLA component: not "cheaper is better", but a target "cost per task ≤ X" placed on the monitoring dashboard, paged when breached just like latency. Sits alongside latency/success-rate SLOs (e.g. Cost per Task ≤ ¥0.8, alert when 7-day mean > ¥1.0). Four causes to check on breach: routing change (more large-model share), longer prompts, traffic-mix shift (fewer easy questions), vendor price change — the first two are team-controllable; the last two belong in contracts and change management.

The constraint mechanism where LLM output is 100% schema-conformant: field names, types and enum values are enforced (OpenAI structured_outputs, Anthropic tool use and the Qwen/DeepSeek json_schema variants all belong here). Key insight: you constrain shape, not facts — a schema blocks format errors but not semantic hallucinations (a well-formatted order_id that does not exist), so execution-layer parameter validation must be added. The default for cloud APIs; pure prompt + "please output JSON" is deprecated.

The mechanism where the LLM emits a structured call request (tool name + arguments) per a declared tool schema, which the program then executes. Engineering points: ① the tool description matters more than the schema — spell out "when to call / where the argument comes from / what if missing"; misfire rates drop by an order of magnitude. ② Narrow granularity — ≤4 arguments per tool; more arguments means more to hallucinate. ③ Verify argument existence in the data layer before execution; invalid arguments are not exceptions but fed back as "does not exist + reason + candidates" so the model can retry.

The lightweight response_format: json_object constraint that guarantees the output is valid JSON — but not that it matches a given schema (misspelled fields, extra or missing fields still happen). Position: fallback for older or smaller models, always paired with a parse-retry chain. The 2026 baseline is Structured Outputs (schema-level constraints); JSON Mode sits in between — "legal, but not necessarily right".

The standard vocabulary for describing JSON structure (type / enum / required / maxLength / additionalProperties) and the constraint medium for LLM structured output. Six design iron rules: ① nesting ≤2 levels, ≤8 fields per level (deep nesting is where small models collapse); ② enum over open fields — every closed value set is an enum; ③ self-explanatory names + descriptions carrying executable decision criteria; ④ 1–2 complete few-shot examples in the prompt; ⑤ minimize required, the rest optional with defaults; ⑥ no 2-D arrays (nested arrays are the reliability cliff) — flat array plus foreign key.

The JSON-Schema keyword that, set to false, forbids the output from carrying any field outside the schema. Under prompt-only or JSON Mode the model freely invents extra fields; it is only truly enforced under Structured Outputs (illegal fields are masked by the decoder). Iron rule: set it false on every business schema, or the downstream validator will be chasing free-styled fields forever.

Grammar constraints enforced at the inference-engine level: illegal tokens are masked one by one during generation, guaranteeing 100% conformance (outlines / guidance / lm-format-enforcer). The standard answer for private deployments: vLLM/Ollama are OpenAI-compatible but lack a structured-outputs switch, so the constraint must move down to the decoding layer (vLLM natively supports guided_json / guided_choice). Cost: a slight per-token slowdown, negligible in most scenarios; applies only where the schema compiles to a grammar (JSON Schema / regex), not free narrative.

The structured-output fallback mechanism: when parsing or validation fails, do not blindly resend the same prompt (~60% success) — feed back "validation error + original input" (e.g. "your last output was missing the required field urgency") and regenerate, lifting success above 95%. Iron rules: ① cap retries at 2 — a third attempt almost certainly fails, so degrade; ② truncation failures are unrecoverable (missing tokens cannot be patched), retry wholesale, so size max_tokens at schema-max ×1.5; ③ instrument retry success — a spike signals a silent model version upgrade or a prompt change.

The pattern for long-structure generation: call 1 emits only the structured skeleton (section array + per-section outline), calls 2..N fill each section (parallelizable), and the program assembles on the id anchor. Versus cramming the whole document into one output field: ① avoids max_tokens truncation (truncated JSON is unrecoverable); ② a failed section re-runs in isolation; ③ the skeleton is reviewable — restructure sections before filling starts. Fits reports and long-form "skeleton + long text" tasks; with parallel section fills, total latency is often shorter.

The total time from user request to complete response. In Agent scenarios: LLM inference latency + tool call latency + network latency + queue wait latency. Targets: < 5 seconds for simple tasks, < 30 seconds for complex tasks. E2E latency is the comprehensive user-experience metric — users do not care which stage is slow, only "how long until I get the result".

The technique of returning LLM output token by token, so users see content without waiting for the full response. Implementation: SSE (Server-Sent Events) or WebSocket push. Effect: perceived latency drops 50-80% (3s full wait → 0.5s to first visible output). Limitations: tool call results must wait for complete return (cannot stream); error handling is unified after streaming ends. In Agent scenarios, streaming output is the first priority for latency optimization.

Executing multiple independent tool calls simultaneously — total latency equals the slowest single call, not the sum. Implementation: Promise.all (JavaScript) or asyncio.gather (Python). Eligibility: calls with no dependency between them (e.g. fetching user info, order info, and recommendations simultaneously). Measured: three 1-second calls complete in 1 second (vs 3 seconds serial). The single highest-impact optimization point in Agent performance tuning.

The response time of an Agent calling external tools (APIs, databases, search engines). Common bottlenecks: external API calls (1-5s), database queries (0.1-1s), file I/O (0.5-2s). Optimization strategies: ① parallelization (independent calls run concurrently); ② caching (same-parameter results cached 5-30 min); ③ timeout control (5-10s timeout with degradation); ④ batch calls (merge multiple queries into one request). In practice, tool call latency often accounts for 60-80% of E2E latency — the most easily overlooked optimization point.

The proportion of cache requests that hit an existing cached entry — the core metric for cache effectiveness. Formula: hits / total requests × 100%. Three Agent-scenario caches: ① LLM response cache (same input + params, TTL 1 hour); ② tool call result cache (same params, TTL 5-30 min); ③ high-frequency question cache (common Q&A, TTL 24 hours). Targets: overall hit rate > 60%, tool call cache hit rate > 40%. Low hit rates call for checking cache key design (too granular) or TTL settings (too short).

The engineering practice of optimizing AI Agent production performance across three dimensions: latency (TTFT, E2E latency — user experience lifeline), throughput (concurrent capacity — system capacity ceiling), and cost (token consumption, tool call overhead — operational efficiency baseline). Optimization order: first ensure latency (user experience), then improve throughput (system capacity), finally reduce cost (operational efficiency). Supporting infrastructure: production monitoring (TTFT, E2E latency, tool call success rate, token consumption, concurrent users) + tiered alerting (P0/P1/P2).

The number of business tasks completed per million tokens — the core metric for token utilization efficiency. Formula: tasks completed / total tokens consumed (millions). Three causes of low token efficiency: ① prompt bloat (system prompt > 500 tokens); ② poor context management (retaining too much conversation history); ③ tool call results re-entering context repeatedly. Optimization gain: one customer service Agent project improved from 50 to 120 tasks per million tokens through context compression, reducing costs by 58%. Used together with Cost per Task.

An AI system capable of autonomously perceiving its environment, formulating plans, and executing actions. AI Agents typically use an LLM as the core reasoning engine, combined with tool calling (Function Calling) and memory systems, to complete multi-step complex tasks such as automatic code writing, database operations, and API calls.

A prompt engineering technique that guides LLMs to show their reasoning process before giving a final answer. By adding "let's think step by step" or providing reasoning examples in the prompt, accuracy on complex tasks like math and logical reasoning can be significantly improved.

A hyperparameter controlling the randomness of LLM output. Lower values (e.g. 0.1) produce more deterministic and repeatable output; higher values (e.g. 0.9) produce more diverse and creative output. Temperature 0 suits scenarios requiring precise answers (code generation), while 0.7-0.9 suits creative writing.

The phenomenon where an LLM generates content that appears plausible but is actually incorrect or fabricated. Hallucinations are an inherent LLM issue, stemming from the fact that models generate text based on probability rather than understanding facts. Mitigation methods include RAG (retrieval-augmented generation), prompt constraints, and output verification.

The maximum number of tokens an LLM can process at once. A larger context window allows the model to handle longer text. Current mainstream models range from 8K to 200K tokens. Context window size is a key LLM selection criterion, directly affecting the model's ability to handle long documents and multi-turn conversations.

A model compression technique that reduces model size and inference time by lowering the numerical precision of model parameters (e.g. from FP32 to INT8). Quantized models can be 75% smaller with 2-4× faster inference, though with slight accuracy loss. Common quantization methods include GPTQ, GGUF, and AWQ.

In the AI context, an intelligent entity capable of autonomously executing tasks. AI Agents typically possess three core capabilities: Perception (sensing the environment), Reasoning (thinking and planning), and Action (calling tools, accessing knowledge bases, executing code to complete tasks).

A prompt engineering technique that guides LLMs to understand task requirements and output format by providing a small number of examples (typically 2-5) in the prompt. Compared to Zero-shot (no examples), Few-shot typically produces more accurate and expected output.

Performing tasks without providing any examples to the LLM, directly requesting it to complete the task. Zero-shot performance depends on model capability and task clarity — simple tasks (translation, classification) work well with Zero-shot, while complex tasks (formatted data extraction) are more reliable with Few-shot.

Five principles of object-oriented design: Single Responsibility, Open-Closed, Liskov Substitution, Interface Segregation, and Dependency Inversion. Following SOLID principles improves code maintainability, extensibility, and testability.

A software development principle emphasizing avoidance of code duplication. Duplicated code increases maintenance costs — modifying one logic requires updating multiple places. Common implementations include function extraction, abstract classes, and design patterns. However, over-pursuing DRY may lead to over-abstraction.

Reusable solutions to common problems in software engineering. Design patterns are not code snippets but general templates for solving specific problems. Classic patterns are divided into creational (Singleton, Factory), structural (Adapter, Proxy), and behavioral (Observer, Strategy) categories.

An architectural pattern that separates an application's read operations (queries) from write operations (commands) into different models. CQRS allows read and write models to be independently optimized — read models can use caches and materialized views, while write models focus on business logic and consistency.

A classic software architecture pattern dividing an application into three core components: Model (data and business logic), View (user interface), and Controller (handles user input and coordinates Model and View). MVC achieves separation of concerns, improving code maintainability.

A measure of the degree of dependency between modules. High coupling means modifying one module affects others, reducing system maintainability and testability. Low coupling is a hallmark of good design; common decoupling methods include interface abstraction, dependency injection, and message queues.

A measure of how closely related elements within a module are. High cohesion means the elements within a module are functionally closely related with clear responsibilities. High cohesion and low coupling are the core goals of modular design.

A behavioral design pattern that defines a one-to-many dependency. When an object (subject) changes state, all dependent objects (observers) are automatically notified. In JavaScript, EventEmitter is a classic implementation of the Observer Pattern.

A creational design pattern that ensures a class has only one instance and provides a global access point. In TypeScript/JavaScript, module-level variables are naturally singletons. Singleton is commonly used for loggers, configuration managers, and database connection pools.

An open-source container orchestration platform for automating deployment, scaling, and management of containers. Kubernetes organizes multiple servers into a cluster for unified container scheduling, networking, storage, and health checking. Suitable for large-scale microservice deployments, though with higher operational complexity — small projects should consider Docker Compose instead.

An open-source Infrastructure as Code (IaC) tool that uses declarative configuration files (HCL language) to define and manage cloud infrastructure. Terraform supports AWS, Azure, Alibaba Cloud and other major platforms, enabling automated creation, modification, and version-controlled management of cloud resources.

An open-source systems monitoring and alerting toolkit originally developed by SoundCloud, now a CNCF graduated project. Prometheus collects metrics via pull mode, supports multi-dimensional data models and powerful PromQL query language. Commonly paired with Grafana for monitoring solutions.

An open-source data visualization and monitoring dashboard tool supporting multiple data sources including Prometheus, Elasticsearch, InfluxDB, and MySQL. Grafana provides rich chart types and alerting functionality, serving as a standard tool for operations monitoring and business data analysis.

An infrastructure layer for handling service-to-service communication. Service Mesh uses Sidecar proxies (e.g. Envoy) to intercept all network traffic between services, providing service discovery, load balancing, circuit breaking, retry logic, and observability — transparent to application code.

A cloud computing execution model where the cloud provider dynamically manages server allocation and resources. Developers only write and upload code without worrying about server configuration, scaling, or maintenance. Common Serverless products include AWS Lambda, Alibaba Cloud Function Compute, and Cloudflare Workers.

A device or software that distributes network traffic across multiple backend servers. Load balancing improves system availability and scalability. Common algorithms include Round Robin, Least Connections, and IP Hash.

An intermediary server between clients and backend servers that receives client requests on behalf of backend servers. Reverse proxies are commonly used for load balancing, SSL termination, caching, and security. Nginx is one of the most popular reverse proxy servers.

Four properties of database transactions: Atomicity (transactions either fully execute or fully rollback), Consistency (data remains consistent before and after transactions), Isolation (concurrent transactions don't interfere with each other), and Durability (committed data is permanently saved). ACID is the core feature of relational databases like PostgreSQL and MySQL.

A distributed system design philosophy, opposite to ACID. BASE emphasizes Basically Available, Soft state, and Eventual consistency. NoSQL databases (like MongoDB, Cassandra) typically adopt the BASE model.

A horizontal database scaling technique that distributes data across multiple database instances based on a key (e.g. user ID). Each shard holds a portion of the data, with all shards forming the complete dataset. Sharding突破单机存储和性能限制，但引入了跨分片查询和分布式事务的复杂度。

The process of synchronizing data from one database instance to another. Replication improves data availability and read performance. Master-Slave replication is the most common pattern: the master handles writes, while slaves handle reads.

A technique for mapping database tables to programming language objects. ORM frameworks (like TypeORM, Prisma, SQLAlchemy) let developers operate databases using object-oriented approaches without writing SQL. ORM improves development efficiency, though complex queries may require raw SQL.

A database optimization technique that accelerates data queries by creating ordered data structures on specified columns. Indexes can dramatically improve SELECT query speed but reduce INSERT/UPDATE performance and consume extra storage. Common index types include B-Tree, Hash, GiST, and GIN.

A database design method that organizes data by eliminating data redundancy and dependencies. Common normal forms include 1NF (eliminate repeating columns), 2NF (eliminate partial dependencies), and 3NF (eliminate transitive dependencies). Normalization reduces redundancy but may increase query complexity.

Intentionally introducing data redundancy to optimize query performance. Denormalization reduces JOIN operations by storing related data in the same table, suitable for read-heavy scenarios. The trade-off between normalization and denormalization depends on actual use cases.

A collective term for non-relational databases that, unlike SQL databases, do not use fixed table structures. Common NoSQL types include document (MongoDB), key-value (Redis), column-family (Cassandra), and graph (Neo4j) databases. NoSQL suits flexible data models and horizontal scaling scenarios.

A database connection management technique that pre-creates a set of database connections for reuse, avoiding the overhead of creating new connections for each request. Connection pools can significantly improve database access performance. Common implementations include PgBouncer (PostgreSQL) and HikariCP (Java).

A common security vulnerability where attackers inject malicious scripts into web pages. When other users visit the page, the scripts execute, potentially stealing cookies, redirecting to malicious sites, or tampering with page content. Defenses include input validation, output escaping, and Content-Security-Policy headers.

An attack that tricks users into clicking malicious links while logged in, exploiting their session to perform unintended actions. Defenses include SameSite Cookie attributes, CSRF tokens, and Referer header validation.

An attack where attackers control massive botnets to flood target servers with requests, exhausting network bandwidth or compute resources, preventing legitimate users from accessing the service. Defenses include CDN protection, traffic scrubbing, rate limiting, and WAF.

A security device or software that monitors and filters HTTP/HTTPS traffic, protecting web applications from common attacks (SQL injection, XSS, file inclusion). WAFs can be deployed at the application front-end (Cloudflare WAF, Alibaba Cloud WAF) or server-side.

An HTTP security header that controls which resources (scripts, styles, images) the browser can load through a whitelist mechanism. CSP effectively prevents XSS attacks and data injection. Example configuration: Content-Security-Policy: default-src 'self'.

A security architecture model based on the principle "never trust, always verify." Zero Trust assumes both internal and external networks are untrusted, performing identity verification and authorization checks on every access request. Key components include microsegmentation, least privilege, and continuous verification.

The underlying encryption protocol of HTTPS, ensuring data confidentiality, integrity, and identity verification during transmission. TLS is the successor to SSL, with TLS 1.3 being the current mainstream version. The TLS handshake process includes key exchange, certificate verification, and encryption parameter negotiation.

An authentication mechanism that allows users to access multiple independent systems with a single set of credentials. SSO is typically implemented based on OAuth 2.0 or SAML protocols, improving both user experience and security.

A security mechanism requiring users to provide two or more verification methods to log in. Common factors include knowledge (passwords), possession (phone codes, hardware keys), and inherence (fingerprints, facial recognition). MFA significantly improves account security.

A security flaw in a system, software, or network that attackers could exploit to gain unauthorized access, destroy data, or cause service disruption. Common vulnerability types include buffer overflow, SQL injection, privilege escalation, and remote code execution. Vulnerability management encompasses discovery, assessment, remediation, and disclosure.

A high-level programming language known for its clean, readable syntax. Python is widely used in data science, machine learning, web development (Django, FastAPI), and automation scripting. Its rich third-party library ecosystem (NumPy, Pandas, PyTorch) makes it the language of choice in AI.

A systems programming language known for memory safety, zero-cost abstractions, and concurrency safety. Rust's ownership system and borrow checker prevent memory errors at compile time without a garbage collector. Suitable for performance-critical scenarios (WebAssembly, system tools, game engines).

A compiled programming language developed by Google, known for its clean syntax, built-in concurrency support, and fast compilation. Go's goroutines and channels make concurrent programming simple. Widely used in cloud-native tools (Docker, Kubernetes, Terraform) and microservices.

A high-level interpreted programming language that is the native scripting language of web browsers. JavaScript supports object-oriented, functional, and event-driven programming paradigms. Node.js brought JavaScript to the server side, making it the language of choice for full-stack development.

A JavaScript runtime environment built on Chrome's V8 engine, enabling JavaScript execution on the server side. Node.js uses an event-driven, non-blocking I/O model, suitable for building high-concurrency, I/O-intensive applications (web servers, API gateways, real-time applications).

A JavaScript front-end library developed by Meta (Facebook) for building user interfaces. React uses a component-based architecture and virtual DOM technology, improving UI development efficiency and performance. The React ecosystem includes React Router, Next.js framework, and state management libraries (Redux, Zustand).

A progressive JavaScript front-end framework known for being lightweight, flexible, and easy to learn. Vue.js's core library focuses on the view layer and can be incrementally integrated into projects. The Vue ecosystem includes Vue Router, Pinia state management, and Nuxt.js framework.

A lightweight web application framework built on Node.js, providing concise APIs for building web servers and APIs. Express.js handles requests and responses through middleware mechanisms and is one of the most popular web frameworks in the Node.js ecosystem.

A full-stack web framework built on React, supporting server-side rendering (SSR), static generation (SSG), incremental static regeneration (ISR), and other rendering modes. Next.js provides file-system routing, API routes, and middleware out of the box, making it the preferred framework for enterprise React applications.

A modern static site generator focused on content-driven websites. Astro ships zero JavaScript by default, using partial hydration to add interactivity only where needed. Supports multiple UI frameworks (React, Vue, Svelte) in a single project. Ideal for documentation sites, blogs, and marketing pages where performance is critical.

Letting users query a database in plain language: the system generates SQL, executes it, and returns the result together with the metric definition. The production bottleneck is not “generating SQL” but “understanding the business”: roughly 80% of project effort goes to the semantic layer (business term to column/definition mapping), 15% to engineering architecture (schema linking, generation constraints, permission injection, execution limits), and 5% to prompts. Go-live bar: top-20 high-frequency questions at 95%+ semantic accuracy, and every answer ships with “expandable SQL + metric definition”.

Synonym for NL2SQL, emphasizing the natural-language-to-SQL translation step. In academic benchmarks (Spider/BIRD) it is a standard evaluation task; in engineering practice it usually denotes the full production pipeline: semantic layer, schema linking, constrained generation, permission-injected execution, result explanation. Three classic failures: 1) stuffing the entire schema into the prompt (context overflow plus same-named-column confusion); 2) relying on prompts for permissions (the model accommodates escalation requests); 3) testing only on curated standard questions, then collapsing under real user traffic.

The recall step in NL2SQL: for each question, surface only the relevant 3-8 tables and 10-20 columns into context instead of the whole schema. A three-stage funnel (vector coarse recall, permission whitelist filter, LLM fine ranking) is the common production implementation. Key details: 1) generate vectors from table/column business annotations, not table names (names are technical identifiers and match poorly); 2) drop unauthorized tables at recall time, so the model cannot see them at all; 3) place metric-dictionary columns at the front of the candidate list with an emphasis marker, because LLMs weight earlier, emphasized context more heavily.

The unified business-to-data mapping layer above physical tables: a business dictionary (synonyms/jargon to columns), a metric dictionary (metric name to formula to standard filters), and table/column-level business annotations. It is the single source of truth for metric definitions, so BI dashboards and NL2SQL both draw definitions from it and the same metric never gets two formulas. In NL2SQL projects it is the moat and the biggest work item (roughly 80% of effort); the standard action is a 2-3 day workshop with the business side (not the data team), walking the top-30 questions one by one and settling definitions into the dictionary.

The structured asset of business metrics in the semantic layer: each entry is a metric name, its formula, standard filters, and the confirmation time/owner of the definition. It is the antidote to the most frequent NL2SQL error, definition mismatch (the model invents its own repeat purchase rate). Three engineering points: 1) the dictionary is confirmed item-by-item with the business side, never written in a closed room by the data team; 2) one business term can carry multiple definition namespaces (operations/finance), with the right one injected by user role at generation; 3) few-shot examples are auto-assembled from the dictionary (business question plus canonical SQL), keeping definitions and examples in sync forever.

An interaction paradigm where business users ask data questions in plain language and get a result chart plus the metric definition directly, skipping the BI scheduling chain. Correct positioning is complement, not replace: high-frequency fixed questions go to BI dashboards (fast, accurate, cheap); long-tail exploratory questions go to conversational BI (flexible, instant, with explanation); both sit on the same semantic layer. In practice, dashboard usage typically does not drop after launch, because business asks the AI first and the AI routes fixed questions back to ready-made dashboards.

Online Analytical Processing: the technology and modeling approaches for multi-dimensional aggregation over large data volumes, as opposed to OLTP (online transaction processing). Characteristics: complex aggregation queries, read-heavy/write-light, and dimensional modeling (star/snowflake). In NL2SQL and conversational BI scenarios, OLAP is the typical target database: query patterns are aggregation-heavy, full-table scans and Cartesian joins are expensive, so permission predicate injection and row caps are mandatory.

The classic data-warehouse dimensional modeling pattern: a central fact table (orders, etc.) fanning out to dimension tables (time, region, product, etc.) like a star. Versus third-normalized models, it reduces join counts for aggregation queries and tolerates column redundancy (storage is cheap, compute is expensive). For NL2SQL projects, star schemas are the most model-friendly structure: fact-versus-dimension choice is intuitive and same-named columns are less ambiguous. If the underlying schema is 3NF, pre-register the “business question to fact table plus join path” mapping in the semantic layer first.

How fresh the knowledge in a RAG base is: the ability of index content to stay consistent with its source documents over time. Four failure modes: stale content (source updated, index not), zombie chunks (document retired, vectors not deleted), coverage gaps (new documents never ingested), and source drift (document contradicts live business-system data). Freshness is a process metric, not a state — the base starts decaying the moment it launches, so it must be continuously measured with a Freshness SLA, contradiction detection, and a probe set, never once at acceptance.

The single authoritative origin of a fact (policies from the approval workflow, prices from the business system, processes from the document system). The first principle of RAG freshness management: there must be an automatic synchronization path between the knowledge base and its source of truth, not a one-time manual copy. A common error is treating “last edited time” as the point of effect — a document edited two weeks ago and still in review is not in effect; the point of effect is the approval or publish event. When the same fact exists in three versions (document, system, hallway), converge the source of truth before building a base at all.

The standard mode for RAG update pipelines: a changed document re-chunks only itself, deletes its old chunks by source_id, and upserts the new ones — cost drops from “hours, whole base” to “minutes, single document.” Precondition: every document has a stable source_id and every chunk a content_hash. The cost of the update pipeline determines knowledge-base freshness — if a full rebuild is too expensive, updates degrade into a quarterly project and everything between rebuilds is a blind spot. Reserve full rebuilds for exactly two cases: embedding-model upgrades and major chunking-strategy changes, and switch via dual indexes to avoid a half-new/half-old window.

Discarding the existing index and re-chunking and re-embedding every document. Costs hours to tens of hours, during which the base is unavailable or half-new/half-old — the anti-pattern of RAG update pipelines: the cost is too high for frequent updates, so the base inevitably rots. Use it only in two scenarios: embedding-model upgrades (dimension/version changes force full re-embedding) and major chunking-strategy changes. Execute with dual-index cutover — build the new index fully before switching traffic — eliminating the half-new/half-old window, in off-peak hours.

Re-ingesting the same document always yields the same result — the floor requirement for a RAG indexing pipeline. Implementation: chunk-level content_hash dedup with upsert rather than insert. A non-idempotent pipeline manufactures a batch of duplicate chunks on every retry: multiple vectors for the same content dilute similarity scores and make recall unpredictable — duplicates are the #1 killer of retrieval quality. Detection: periodically scan for highly similar chunk pairs under the same source_id.

A chunk whose source document has been retired or replaced, but whose vectors remain in the index store and can still be retrieved. More insidious than plain staleness: when old and new versions coexist and contradict each other, retrieval hits whichever by luck — answers that work “sometimes” and complaints arriving randomly. Detection: pairwise contradiction detection over highly similar chunks on the same topic; each contradiction pair is a ready-made deletion list. Cure: documents carry a status field plus an explicit retirement workflow (mark → delete vectors → probe-verify the deletion → audit trail). The verbs of RAG operations are add, change, delete — teams that only “add” accumulate zombie chunks fastest.

A regression set of 50–100 high-frequency real questions (from support tickets or search logs) plus pinned expected-answer fact points (2–3 facts that must be hit), run automatically weekly or after any source change, judging both “did the answer hit the fact points” and “is the cited document version the latest”; failures enter a fix queue. In essence an evaluation regression suite whose target is knowledge freshness rather than model quality. Two maintenance rules: when source documents change, the expected answers must be updated in lockstep (or the probes go stale before the base does); plot results as a weekly trend line — freshness degrades gradually, so one healthy data point proves nothing.

LLM pairwise contradiction judgment over chunk pairs of the same topic whose embedding similarity exceeds a threshold, producing a list of “highly similar but conflicting” chunks. The dedicated detector for zombie chunks: when old and new policies coexist, each contradiction pair is a deletion list. Cadence: full run weekly plus incremental runs after changes; findings cleared within 48 hours (delete, mark, or confirm-retention). It complements the Freshness SLA (watches the pipeline) and the probe set (verifies the outcome): contradiction detection audits the stock. Running only one detection instrument guarantees the others are missed — all three must be online.

The first stage of the RAG ingestion layer: extracting text and structure (heading hierarchy, tables, lists) from PDFs, Word files, and scans. Tools divide labor by document type rather than by “most accurate”: digital documents (with a text layer) use Unstructured or Docling; scanned documents need OCR plus layout analysis (MinerU / Docling); table-heavy documents should prioritize tools with strong table-structure preservation. The selection criterion is benchmarking 20 of your own documents on three numbers (parse completeness, structure preservation rate, parse failure rate) — never the public benchmark. Do not build your own parser; the legitimate place for custom code is a rule-patch layer (number recognition, running-header removal) built on top of the tool’s output, never a replacement for it.

The quality gate for scanned-document parsing: OCR output carries per-chunk confidence scores, and chunks below the threshold (typically 0.85) are flagged for human review instead of being ingested. Low-confidence OCR chunks in the base surface as “the AI occasionally answers a parameter with a shifted digit, or emits garbled characters,” and users cannot tell whether the model or the ingestion is wrong, so every complaint lands on the model. In knowledge bases where scans exceed 30% of documents, a confidence threshold plus a human-review queue is standard equipment, not polish — a 7% share of low-quality chunks expands at the user end into the overall impression that “this AI is not trustworthy.”

The parser’s recognition of a page’s visual structure (body region, headings, tables, captions, headers/footers) before extracting text, so output is organized by structure. The core capability for scanned and table-heavy documents: without it, tables become one-cell-per-chunk, page numbers bleed into body text, and multi-column text comes out in the wrong order. MinerU / Docling table-reconstruction quality represents the current state of the art; pure text-recognition layers (e.g. PaddleOCR) do no layout analysis and push structure handling downstream.

Keeping table row/column relations intact during parsing (header-to-row correspondence, merged cells, table titles), emitting Markdown tables or an equivalent structured representation instead of scattered text. In enterprise knowledge bases tables account for 40–60% of retrieval failures: a nameless cell retrieved alone cannot be interpreted by any model (“which column is unit price?”). Companion chunking rules: small tables (under ~15 rows) stay whole with header context; large tables chunk by row, but every row chunk must carry the table header; table chunks carry table name, section, and page number in metadata.

The first pass of the cleaning layer: removing running headers/footers (page numbers, document IDs, confidentiality marks), repeated boilerplate, and bookmarked table-of-contents entries — content that is structure in the source document and pure noise in the base. Rule requirements: regex-remove short lines appearing on more than 10% of pages that carry no business information; removal rules live in code with a sample archive, never “we think it looks clean” (not reproducible = not acceptable); delete only, never rewrite — content-level rewriting does not belong in the cleaning layer.

A Unicode character that exists in text but is invisible to the eye (zero-width space U+200B, etc.), extremely common in OCR output. Harm: it splits “the same passage” into near-duplicate chunks because of invisible-character differences, which after embedding becomes a hidden killer of retrieval quality — surfacing as “the same content keeps getting retrieved twice” and unstable recall. The cleaning layer must normalize these away; the detection move is a character-level diff of highly similar chunk pairs, where invisible-character differences are one of the most common sources.

The fundamental test for whether a chunk is fit: readable standalone, quotable directly as an answer fragment, without depending on the previous block’s context. Chunk units derive from it — the smallest self-contained semantic unit per document type (policies = heading hierarchy, contracts = clause number, manuals = task, FAQs = one Q&A) — never a fixed word count. The direct symptom of self-containedness failure is “answers missing half a sentence” (clause 4.2.1 split across two chunks; retrieval hits the first half, “Party A shall…”). It cannot be measured at the generation layer — given a half-clause the model completes it fluently, the answer reads fine, but the citation is wrong. Only inspecting the chunks themselves (number-integrity sampling) catches it.

Dual-channel retrieval for contract/regulation bases: “the penalty clause” goes through vector semantic recall, “clause 4.2.1” goes through exact number matching, and the two merge at the retrieval layer. The precondition is a `number` metadata field on every clause chunk, attached at ingestion — without it, number-based questions can only be guessed by embeddings, and embeddings have very poor discriminative power for identifiers like clause numbers and model codes. The `number` field has a second use: the version-comparison key for contradiction detection (the same clause number across contract versions makes diffs immediate). At acceptance, the number-integrity sampling (20 random clauses checked for single-chunk self-containment) tests the input quality of exactly this chain.