AI Engineering Interview Questions 2026RAG, LLMs, Agents and Projects

AI engineering is the discipline of building, deploying, and maintaining software applications that use AI models — especially large language models (LLMs) — to solve real problems. It combines software engineering (APIs, deployment, logging) with applied AI skills (prompt engineering, RAG, agents). The role is distinct from ML engineering, which focuses on training models, and from data science, which focuses on analysis and insights.

An LLM is a deep learning model trained on large text datasets to understand and generate human language. It predicts the next token (word fragment) given a context. LLMs like GPT-4, Claude, and Gemini are used via APIs — you send a prompt and receive a generated response. AI engineers do not typically train LLMs; they build applications on top of them via APIs.

Prompt engineering is the practice of designing inputs (prompts) to LLMs to reliably get the outputs you need. It involves designing system prompts, structuring context, using few-shot examples, and controlling output format. In production AI systems, prompt design directly affects output quality, reliability, and cost.

A token is the unit of text an LLM processes. Tokens are not exactly words — a common English word is roughly one token; longer words, code, and non-English text may be two or more tokens. LLMs have context window limits (e.g., 128K tokens) and APIs charge per token. Engineers must track and manage token usage for cost and context window reasons.

An API (Application Programming Interface) is a way for software to communicate with another service over the internet using structured requests and responses. LLM providers expose APIs — you send a JSON payload with your prompt and model parameters, and receive a JSON response with the generated text. AI engineers spend a large proportion of their time working with LLM APIs and building APIs of their own (using FastAPI or similar).

JSON (JavaScript Object Notation) is a lightweight text format for structured data. It is the standard format for API requests and responses, including LLM API calls. AI engineers need to read, parse, and write JSON constantly — from sending prompts to LLM APIs, to parsing structured outputs, to returning results from FastAPI endpoints. Structured outputs from LLMs (getting a model to return valid JSON) are a common production requirement.

An embedding is a numerical representation of text (or other data) as a high-dimensional vector. An embedding model converts a sentence or paragraph into a list of numbers such that semantically similar texts produce numerically close vectors. Embeddings are the foundation of semantic search and vector databases — and are used in every RAG pipeline to convert documents and queries into comparable numerical form.

RAG (Retrieval-Augmented Generation) is a pattern for building AI applications that can answer questions about private, current, or domain-specific information that the LLM was not trained on. It works by: (1) indexing your documents as embeddings in a vector database; (2) at query time, retrieving the most semantically relevant chunks; (3) passing those chunks as context to the LLM; (4) generating an answer grounded in the retrieved content.

A vector database stores text chunks as high-dimensional numerical embeddings and enables fast semantic similarity search — finding content by meaning rather than exact keywords. Common vector databases used in AI engineering include Pinecone, Chroma, FAISS, pgvector, and Weaviate. They are the retrieval engine of most RAG systems.

Hallucination is when an LLM generates content that is factually incorrect but stated with confidence. It happens because LLMs generate the statistically likely next token — they do not look up facts. Hallucination is mitigated in production AI systems through: RAG (grounding answers in retrieved documents), structured outputs, output evaluation with RAGAS, human review processes, and adding citations to generated responses.

RAG (Retrieval-Augmented Generation) is a pattern for grounding LLM answers in external knowledge. It is used because LLMs only know what they were trained on — they cannot answer questions about private documents, recent events, or domain-specific content without retrieval. RAG solves this by retrieving relevant context at query time and injecting it into the LLM prompt, enabling factual answers about any content that has been indexed.

The RAG pipeline has two phases. Indexing: load documents → split into chunks → embed each chunk with an embedding model → store vectors + metadata in a vector database. Querying: user submits question → embed the question → retrieve top-K similar chunks from the vector database → assemble chunks into a prompt template → call LLM → return grounded answer. Optional production additions: reranking, citation extraction, RAGAS evaluation, and caching.

Chunking is the process of splitting source documents into smaller segments before embedding. Chunk size is a critical design decision: chunks that are too small lose context (the chunk alone may not answer any question); chunks that are too large dilute the embedding (the vector averages across multiple topics). Common approaches: fixed-size with overlap (256–512 tokens, 10–20% overlap), recursive character splitting, and semantic chunking (split at topic boundaries). Poor chunking is the most common root cause of poor RAG retrieval quality.

Embeddings convert text chunks and user queries into numerical vectors. The core insight is that semantically similar texts produce similar vectors — allowing the vector database to find the chunks most relevant to the user's question by measuring geometric distance in vector space. The same embedding model must be used for indexing and querying; mixing models produces incomparable vectors.

A retriever is a LangChain abstraction that takes a query and returns relevant documents. It wraps the vector store similarity search and optionally adds metadata filtering, score thresholds, and ensemble combinations. Common retrievers: VectorStoreRetriever, MultiQueryRetriever (generates multiple query paraphrases), ContextualCompressionRetriever (compresses retrieved documents to keep only relevant passages), and BM25Retriever (keyword-based).

Reranking is a two-stage retrieval strategy: broad vector search retrieves top-20 candidates; a cross-encoder reranker then reorders them by relevance to the specific query, returning the top 3–5. Reranking improves retrieval precision at the cost of added latency (~50–200ms). Add reranking when your initial retrieval returns semantically close but topically irrelevant chunks, or when improving factual grounding is worth the latency trade-off. Cohere Rerank and FlashRank are common options.

Citations connect each part of the LLM's answer to the specific retrieved chunks it used. Implementation approaches: (1) prompt the LLM to include source references inline; (2) parse the response and map back to source metadata (source file, page number, URL); (3) use structured output mode to return citations as a separate field alongside the answer. Cohere's Command models have native citation support. Citations are important for enterprise RAG applications where users need to verify answers.

RAG reduces hallucination by giving the LLM specific, relevant, up-to-date context in the prompt. Instead of relying on potentially incorrect training memory, the model is instructed to answer only using the provided context and to say "I don't know" if the context doesn't contain enough information. The system prompt typically includes an explicit instruction like "Answer based only on the provided context. If the context doesn't contain the answer, say so."

RAGAS is an open-source framework for evaluating RAG systems. It measures four metrics: faithfulness (does the answer contradict the retrieved context?), context precision (are the retrieved chunks relevant?), context recall (were all necessary chunks retrieved?), and answer relevancy (does the answer address the question?). You use RAGAS by preparing a test dataset of questions with ground-truth answers and relevant document IDs, running your RAG pipeline, and computing the scores. RAGAS evaluation should be run as part of CI on every change to chunking, retrieval, or prompting logic.

Hybrid search combines vector similarity search (semantic) with keyword search (BM25) and merges results — typically using Reciprocal Rank Fusion (RRF). It improves retrieval when queries include specific identifiers, product codes, or exact phrases that benefit from keyword matching, while still handling semantic paraphrasing. Weaviate, Qdrant, and Azure AI Search support hybrid search natively. Adding hybrid search is a common improvement step for production RAG systems.

Production RAG challenges include: poor retrieval quality from bad chunking or wrong embedding model; duplicate or overlapping retrieved chunks; context window limits constraining how much retrieved text you can include; latency from embedding, vector search, reranking, and LLM call in sequence; cost management across embedding API + vector database + LLM; access control (preventing user A from retrieving user B's documents); index freshness (updating vectors when source documents change); and evaluation drift (retrieval quality degrading over time without an evaluation harness to detect it).

Choose fine-tuning when: you need the model to adopt a specific writing style or tone; the task requires domain-specific vocabulary the base model doesn't handle well; the information to be learned is stable and unlikely to change; response speed is critical and retrieval adds unacceptable latency. Choose RAG when: the information is dynamic, private, or must stay current; you need citations and traceability; the knowledge base is large or grows frequently; you cannot afford fine-tuning cost and compute. See the RAG vs Fine-Tuning guide for the full decision framework.

An AI agent is an LLM-powered system that autonomously decides which actions to take — calling tools, browsing the web, writing and executing code, querying databases — in order to complete a goal. Unlike a simple chatbot that only generates text, an agent has tools it can use and follows a loop of Reasoning → Acting → Observing the result → Reasoning again until the task is complete.

A chatbot takes a user message and generates one text response. An AI agent can take a user instruction and execute multiple steps autonomously: searching a database, calling an API, running code, reading a file, and writing output — all without requiring the user to direct each step. Agents use tool calling to interact with external systems and maintain state across multiple reasoning steps.

An agentic workflow is an AI pipeline where the LLM dynamically decides the sequence of steps — which tools to call, in what order, and when to stop — based on intermediate results. This contrasts with a fixed pipeline where steps are predefined. LangGraph is the primary framework for building structured agentic workflows with explicit state management and conditional routing.

Tool calling allows an LLM to invoke external functions you define — a database query, API call, calculator, code executor, or file reader. You register tools with their names, descriptions, and parameter schemas. When the LLM decides a tool should be called, it outputs a structured tool-call message. Your application executes the function and returns the result to the LLM, which continues reasoning. Tool calling is the mechanism that gives agents the ability to take actions in the world.

Memory in agents has two forms: short-term memory (the conversation history and intermediate results within a single session — stored in the context window); and long-term memory (persistent storage outside the context window — often a vector database or SQL database that stores summaries of past sessions, user preferences, or knowledge accumulated over time). Managing memory is a core design challenge in production agents: what to keep, what to summarise, and what to retrieve.

In the planner/executor pattern, one LLM call (the planner) decomposes the user goal into a sequence of sub-tasks; separate LLM calls (executors) or tools then carry out each sub-task. The planner also handles replanning when an executor fails or returns unexpected results. This pattern improves reliability for complex multi-step tasks by separating high-level reasoning from low-level execution.

A multi-agent system uses multiple specialised AI agents that collaborate — one agent might search the web, another analyses retrieved data, a third writes a report, and an orchestrator coordinates them. Multi-agent systems are more capable than single agents for complex tasks but introduce coordination overhead, communication costs, and failure cascade risks. LangGraph and CrewAI are the main frameworks for building multi-agent systems in Python.

LangGraph is a framework for building stateful, graph-based agent workflows. Unlike LangChain's original agent executor (which ran a generic ReAct loop), LangGraph gives you explicit control over state structure (via typed TypedDict nodes), conditional routing between nodes, cycles for retry/reflection, and human-in-the-loop checkpoints. LangGraph is preferred for production agents where you need predictable behaviour, debuggable state, and complex workflows with branching logic.

CrewAI is a framework for building role-based multi-agent systems. You define agents with specific roles (researcher, writer, analyst), assign tasks to agents, and define a process (sequential or hierarchical). CrewAI is higher-level than LangGraph — easier to get started with multi-agent systems, but less granular control over state and routing. Good for content creation pipelines, research summarisation, and agentic tasks that map naturally to human team structures.

Human-in-the-loop is a design pattern where the agent pauses at defined checkpoints to request human approval, clarification, or correction before proceeding. LangGraph supports this natively via breakpoints and persistence. HITL is important for high-stakes or irreversible agent actions (sending emails, modifying databases, making API calls with real-world effects). In production, HITL is often part of the QA and safety layer.

Guardrails are constraints that prevent an AI agent from taking harmful, unintended, or off-policy actions. They can be implemented as: input validation (rejecting out-of-scope queries); output validation (checking LLM responses before returning them to users); tool use restrictions (limiting which tools agents can invoke); rate limiting; and monitoring with alerting when anomalous behaviour is detected. NeMo Guardrails and Guardrails AI are Python libraries for systematic guardrail implementation.

Agent evaluation is harder than static LLM evaluation because agents take multiple steps. Approaches: trajectory evaluation (did the agent take the right sequence of steps?); final output evaluation (did the agent produce the correct result?); tool use accuracy (did it call the right tools with the right parameters?); latency and cost tracking (how many LLM calls and tokens per task?); and failure mode analysis (what percentage of tasks fail and why?). LangSmith and custom logging are the primary tools for agent tracing and evaluation.

LangChain is a Python (and JavaScript) framework for building applications on top of LLMs. It provides abstractions and integrations for: prompt templates, LLM wrappers, document loaders, text splitters, embedding models, vector stores, retrievers, chains, agents, and output parsers. Its main strength is accelerating RAG prototype development through its extensive integration ecosystem — connecting to over 100 LLM providers, vector databases, and data sources.

A chain in LangChain is a sequence of components connected together — for example, a prompt template → LLM → output parser. LangChain Expression Language (LCEL) uses a pipe operator syntax (prompt | llm | parser) to compose chains declaratively. Chains can be simple (prompt to LLM) or complex (retriever → prompt → LLM → parser → router → another chain).

A retriever is a LangChain interface that takes a query string and returns relevant document chunks. It wraps vector store similarity search but can also implement custom logic — multi-query retrieval (generating multiple paraphrases of the query), contextual compression (trimming irrelevant parts of retrieved chunks), or ensemble retrieval (combining vector search and BM25). Retrievers are the core component connecting vector stores to RAG chains.

LangChain tools are Python functions decorated or wrapped as tools that an agent can choose to invoke. Each tool has a name, description, and input schema. When a LangChain agent decides to use a tool, it generates the tool name and arguments as a structured output; LangChain invokes the function and passes the result back to the agent for the next reasoning step. Tool design — writing clear names and descriptions — directly impacts whether the agent chooses tools correctly.

A LangChain agent is an LLM given a set of tools and a goal. It follows a reasoning loop: the LLM reasons about the goal → decides whether to call a tool → LangChain executes the tool → observation is added to context → LLM reasons again. LangChain supports several agent types: ReAct, OpenAI Functions, and OpenAI Tools agents. For production stateful agents, LangGraph is preferred over the built-in agent executor.

LangChain provides components for building LLM applications and RAG systems. LangGraph is built on top of LangChain and provides a graph-based framework for stateful, cyclical agent workflows. LangChain agents run a generic loop with limited state control; LangGraph lets you define explicit nodes, edges, conditional routing, state schemas, and human-in-the-loop checkpoints. Use LangChain for straightforward RAG pipelines; use LangGraph for complex, stateful, or multi-agent systems.

Common LangChain production criticisms: abstraction layers make debugging harder when something goes wrong inside a chain; version updates frequently introduced breaking changes (a significant pain in production maintenance); the framework encourages rapid prototyping which can result in unmaintainable code in larger systems; and for very high-performance use cases, the abstraction overhead can be replaced with direct API calls. LangChain is excellent for prototyping; production teams often strip back to more direct code over time.

LangSmith is a LangChain-developed platform for tracing, evaluating, and monitoring LangChain applications. It records every step of a chain or agent run — inputs, outputs, latency, token counts — and makes them browsable and searchable. It supports creating test datasets, running evaluations, and comparing runs. LangSmith is the primary tool for debugging LangChain applications and setting up RAG evaluation pipelines in the LangChain ecosystem.

Build a FastAPI app with your LLM or RAG logic → write a Dockerfile that installs dependencies and starts uvicorn → build the image → push to a container registry (ECR, GCR, Docker Hub) → deploy to cloud (AWS ECS, Google Cloud Run, Azure Container Apps). Environment variables (API keys, database URLs) are injected via secrets management (AWS Secrets Manager, GCP Secret Manager). Health check endpoint (/health) is required for orchestration platforms to manage the container lifecycle.

Latency reduction strategies: response streaming (return tokens as generated rather than waiting for completion); caching (Redis cache for repeated queries — hash the query and return cached response if available); model selection (smaller, faster models for low-complexity queries); parallel LLM calls for multi-part responses; async request handling; and retrieval optimisation (reduce the number of chunks retrieved, tune chunk size to minimise LLM context). Measure p50, p95, and p99 latency per endpoint.

Use structured JSON logging with a consistent schema per log entry: request_id, timestamp, user_id (hashed), model, input_tokens, output_tokens, latency_ms, retrieved_chunks (IDs only, not full content), and response_status. Send logs to a centralised log aggregation system (CloudWatch, Datadog, ELK Stack). Never log raw user inputs or full LLM responses if they may contain PII — apply masking or only log hashed/truncated versions.

LLMOps is the practice of operating LLM-based applications in production — analogous to MLOps for traditional ML but adapted for the specific challenges of LLMs. It includes: prompt versioning and management; output evaluation pipelines; latency and cost monitoring; model version management (handling LLM provider model deprecations); A/B testing of prompts and retrieval strategies; and incident response when LLM output quality degrades.

Monitor at four levels: infrastructure (CPU, memory, container health — standard cloud monitoring); API (request rate, error rate, latency per endpoint — Datadog, CloudWatch); LLM (token counts per request, model error rates, cost per query — via LLM provider dashboards and structured logs); quality (sampled automated RAGAS evaluation, human review of flagged outputs, user feedback signals). Set alerts for: error rate > 1%, p95 latency > SLA, token cost anomalies, and quality metric degradation.

Cost control: choose the smallest model that meets quality requirements for each query type; cache frequent queries in Redis; limit context window by trimming low-relevance retrieved chunks; use streaming to detect early termination opportunities; track per-request token counts in structured logs; set hard token limits on API calls; implement cost budgets per customer or tenant; and review the cost-per-query dashboard weekly to detect usage anomalies.

AI applications use environment variables to store API keys, database connection strings, and configuration — never hardcoding secrets in source code. Locally, use .env files loaded by python-dotenv (never committed to git — add to .gitignore). In production, use a secrets management service (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault) to inject secrets at runtime. In Docker/Kubernetes, secrets are mounted as environment variables or files, not baked into the image.

Production AI applications should implement: retry with exponential backoff for transient rate limit and timeout errors; fallback to an alternative model or provider if the primary is unavailable (e.g., fallback from GPT-4o to Claude Sonnet); circuit breaker pattern to stop hammering a failing service; graceful degradation (return cached or default response rather than a 500 error); and alerting when error rates exceed thresholds. Structuring your LLM calls behind an abstraction layer makes provider switching easier.