AI Systems Engineering Cheatsheet

Designing production-grade AI systems — accuracy, cost, security, robustness, and evaluation

What to expect in AI engineering interviews: These interviews test how you'd design and build practical LLM-powered systems — RAG pipelines, customer support agents, evaluation workflows, and more. Expect questions about system design tradeoffs such as retrieval quality, latency, cost, memory, tool use, hallucination mitigation, and evals, as well as how you'd make the system reliable, safe, and useful in production.

How to use this cheatsheet:

🗺 Part 1 — Problem FrameworkUse this section to structure your answer from the ground up. Start with Scoping & Constraints to nail down goals and tradeoffs, then choose an Architectural Pattern (Prompt Engineering, RAG, or Agents), and finish by defining how you'll measure success with Testing & Evaluation.

⚙ Part 2 — Additional FunctionalityOnce the core design is set, layer on system-level optimizations. Walk through the four pillars — Accuracy & Grounding, Performance & Cost, Robustness & Fallbacks, Privacy & Security — and address the ones most relevant to your problem.

🗺

Problem Framework

Scoping + Constraints · Architectural Design · Testing & Evaluation

1. Scoping + Constraints

Category	Key Questions / Checkpoints
Business Goals	What is the core objective? (e.g., Automation, Revenue, Customer Support)
Budget	What is the cost-per-query limit? Total infrastructure budget?
Latency	Hard constraints (e.g., <2s for UI) vs. soft goals.
Accuracy	Is "hallucination" a dealbreaker (Legal/Medical) or an acceptable risk (Creative)?
Observability	How will we know if it breaks? (MTTD/MTTR requirements)

💡 Pro-Tips for Interviews:

Always start by defining Constraints. Ask about the Latency Budget (<2s?) and Accuracy Tolerance — this dictates whether you build a lightweight RAG or a complex multi-agent chain.
Never overcomplicate systems.
Namedrop tools that solve specific problems (e.g., Pydantic for JSON structuring) — shows hands-on experience.
Practice building a small project yourself!

2. Architectural Design

Choose the right architectural pattern for the problem before optimizing anything else.

Prompt Engineering

The baseline architecture — design the LLM's instructions to reliably produce the desired output. Best when the model already has the knowledge needed.

System Prompt: Define role, tone, output format, and hard constraints.
Few-Shot: Include 3–5 examples of ideal input/output pairs.
Chain-of-Thought: Ask the model to reason step-by-step before answering to reduce logic errors.

RAG (Retrieval-Augmented Generation)

Grounds the LLM in external or private knowledge. Best when the answer depends on documents the model wasn't trained on.

Chunk & Embed: Split documents into semantic pieces, embed into a vector DB (Pinecone, Milvus).
Retrieve: Use ANN search to find top-K relevant chunks at query time.
Re-rank & Inject: Cross-Encoder re-ranks chunks; inject into the prompt as context.

Agents

Multi-step, tool-using systems where the model plans and acts. Best for tasks that require external actions, dynamic decisions, or multi-turn workflows.

Tool Calling: Equip the model with APIs, search, code execution, or DB queries it can invoke.
State Management: Track multi-turn memory and loop state across tool calls.
Planning: Use ReAct or similar patterns to reason → act → observe → repeat.

3. Testing & Evaluation

Define upfront how you'll know if the system is working — before and after deployment.

Hierarchical Testing

Component Level: Test retrieval accuracy (Recall@k), tool-calling syntax, and guardrail triggers in isolation.
End-to-End (E2E): Compare full system responses against a Golden Dataset (human-verified ground truth).

Hybrid Judge

Deterministic: Use code/Pydantic to validate JSON schemas and structural requirements.
LLM-as-a-Judge: Use a high-tier model (GPT-4o, Claude Sonnet) with a strict rubric to score nuanced traits like tone or helpfulness.

Key Metrics

Latency P99Software perf

MAP / RRRetrieval quality

RAGAsFaithfulness + Relevancy

Closing the Production Loop

Slicing: Analyze failures by subgroup (e.g., "Does the model fail on math-heavy queries?").
Periodic Sampling: Pull 100 random production logs regularly to catch model drift early.

⚠️ All evaluation metrics require labeled ground truth data. Invest in high-quality golden datasets early.

⚙

Additional Functionality / Features to Optimize System

Accuracy & Knowledge Grounding · Performance & Cost · Robustness & Fallbacks · Privacy & Security

1. 🎯 Accuracy & Knowledge Grounding

Ensuring the model "knows" the right info and doesn't make things up.

RAG Pipeline

Use for dynamic/private data.

Chunking & Vector DB: Store in Pinecone/Milvus using semantic pieces.
Retrieval: Use ANN (Approximate Nearest Neighbor) for Top-K chunks.
Re-ranking: Cross-Encoder ranks most relevant docs before LLM sees them.

Fine-Tuning

For static, specialized domains or to bake in a specific voice/formatting style.

Prompt Engineering

Few-Shot: Provide 3–5 examples of the "Perfect Answer."
Chain-of-Thought (CoT): Force the model to "think step-by-step" to reduce logic errors.

Evaluation (RAGAs Framework)

Faithfulness: Is the answer derived only from the context?
Relevancy: Does the answer actually address the user's prompt?

2. ⚡ Performance & Cost Optimization

Reducing latency (P99) and keeping the token budget under control.

Model Routing (Gateway)

Simple tasks → "Small" models (GPT-4o-mini, Llama 8B).
Complex reasoning → "Large" models (o1, Claude Opus).
Distillation: Teacher model generates synthetic data to train a cheaper Student model.

Caching Strategy

Exact Cache: 1:1 match on prompt strings (0 tokens used).
Semantic Cache: Vector similarity to find "close enough" previous answers.

TTFT

Time to First Token — critical for UI snappiness.

TPS

Tokens Per Second — throughput measure.

3. 🛠️ Robustness & Fallbacks

Making the system "Production-Grade" so it doesn't break in the wild.

Logic Layer

State Management: Handling multi-turn memory and agentic tool-calling loops.
Structural Validation: Use Pydantic or JSON schema to force valid LLM output.

Failover Policies

Graceful Degradation: Route to an open-source fallback or "Safe" canned response if primary LLM is down.
Human-in-the-Loop: Route low-confidence scores to a human reviewer.

Observability

Golden Datasets: Human-verified ground truth to test before every deployment.
Periodic Sampling: Review 100 random production samples weekly to detect "Model Drift."

4. 🛡️ Privacy & Security

Protecting PII, preventing jailbreaks, and staying compliant.

Input Guardrails

PII Redaction: Scrub names/SSNs before they reach the LLM provider.
Prompt Injection Defense: Filter "Ignore all previous instructions" style attacks.

Output Guardrails

Toxicity Filters: Prevent harmful content generation.
Proprietary Leakage: Prevent the model from revealing internal system instructions.

Data Lifecycle

Rigorous Deduplication and Cleaning during acquisition to prevent the model from learning from sensitive or dirty data.

📚 Learning Resources

Resource	Source
AI Engineering: Building Applications with Foundation Models Comprehensive textbook covering the full spectrum of AI system design and production deployment	Chip Huyen (O'Reilly)
Retrieval-Augmented Generation Building RAG systems from basics to advanced techniques, including chunking strategies and evaluation.	DeepLearning.AI
Agentic AI Design patterns for autonomous AI agents that can plan, reason, and use tools effectively.	DeepLearning.AI
AI Agents in LangGraph Building complex multi-agent workflows with state management and tool orchestration.	DeepLearning.AI
Pydantic for LLM Workflows Structured data validation and JSON schema enforcement for reliable LLM outputs.	DeepLearning.AI
Building Effective Agents Anthropic's engineering guide to agent design patterns, prompt strategies, and production best practices.	Anthropic