ByteRun resources · Senior / Lead / Architect
AI system design interview questions
Six architecture scenarios for designing real LLM, RAG and agent systems. Practice explaining constraints, choosing trade-offs and defending what happens when things fail.
Illustrative interview prompts, not scored daily questions or a single prescribed architecture.
How to structure an architecture answer
Clarify the users, traffic assumptions, data sensitivity and success criteria first. Sketch the request and data paths, then explain the trade-offs, failure modes and evidence you would use to validate your design.
- Requirements and measurable constraints
- Components and data flow
- Security and failure handling
- Evaluation, observability and cost
01 / RAG architecture
Design a permission-aware enterprise knowledge assistant
Employees ask questions across internal documents with different access rights. Design ingestion, retrieval and grounded answers without exposing another team's data.
Questions to resolve
- Where do identity and document permissions enter retrieval?
- How do updates, deletions and permission changes reach the index?
- What happens when retrieved evidence is insufficient or contradictory?
Discussion points
Enforce access rules before unauthorized content enters the model context. Keep source identifiers and citations through the pipeline, and define an abstention path. Compare pre-filtering with post-filtering for retrieval quality and leakage risk; a vector index is not an authorization boundary.
Follow-up: An employee loses access to a document. How do you invalidate cached answers and indexed permissions?
02 / LLM serving
Design a low-latency AI autocomplete service
Your editor suggests completions while a developer types. Design the request path, model serving and fallbacks for variable traffic and a tight latency budget.
Questions to resolve
- Which metric matters: time to first token, full completion latency, or both?
- When is a smaller model sufficient?
- How will you cancel stale requests and isolate tenants in caches?
Discussion points
Start with a measured latency budget across networking, queuing and inference. Compare smaller models, shorter contexts and caching against suggestion quality. Streaming improves perceived responsiveness but does not by itself guarantee a faster full response. Cache reuse must preserve permissions and relevant request context.
Follow-up: Traffic spikes while the model provider slows down. Which requests do you reject, degrade or retry?
03 / AI security
Contain prompt injection in a RAG application
A retrieved page contains instructions to ignore the user and reveal secrets. Design defenses for an assistant that can also call internal tools.
Questions to resolve
- Which inputs are trusted, and which are only data?
- Who authorizes each tool call?
- Which actions require explicit human approval?
Discussion points
Treat retrieved content as untrusted data, not instructions. Limit tool privileges, validate arguments and authorize actions outside the model. Separate secrets from model context and test adversarial documents. Prompt wording alone is not a sufficient security boundary.
Follow-up: The model produces a valid-looking tool call influenced by a malicious document. What independently blocks it?
04 / Quality & evaluation
Evaluate a customer-support RAG system before launch
A support assistant must answer from product documentation, cite evidence and decline unsupported requests. Design an evaluation and release process.
Questions to resolve
- How do you distinguish retrieval failure from generation failure?
- What does the test set include beyond easy questions?
- How will you monitor quality after a model or index change?
Discussion points
Build representative test cases with expected evidence and acceptable responses, including unanswerable and permission-sensitive questions. Evaluate retrieval and answer quality separately. Use human review to calibrate automated grading; track groundedness, task success, latency and cost rather than relying on one aggregate score.
Follow-up: A model change improves fluency but reduces citation accuracy. What prevents that release from reaching everyone?
05 / Agent reliability
Design an agent that safely executes operational tasks
An assistant can inspect cloud incidents and propose remediation. Design a workflow that remains safe when a tool times out or the assistant repeats an action.
Questions to resolve
- Which operations are read-only, reversible or destructive?
- Where do you store workflow state and approval decisions?
- How do retries avoid duplicate side effects?
Discussion points
Use explicit workflow state, narrowly scoped tools and bounded execution. Make side-effecting operations idempotent where possible, and require approval for high-impact changes. Record action attempts and outcomes so a timeout does not silently become a duplicate deployment or deletion.
Follow-up: The tool times out after applying a change. How does the next attempt discover whether the action succeeded?
06 / Cloud architecture
Operate a multi-tenant LLM application under a cost budget
Several customers share an AI platform with different quotas and data requirements. Design routing, isolation, observability and graceful degradation.
Questions to resolve
- How do you enforce per-tenant quotas before expensive work?
- Which data can be cached, logged or sent to a provider?
- How do you preserve service when one provider fails?
Discussion points
Carry tenant identity through storage, retrieval, cache keys and telemetry. Set token and concurrency budgets, and measure cost per successful task. Define fallback behavior explicitly: a cheaper model or different provider may change quality, privacy guarantees and regional processing constraints.
Follow-up: One tenant floods the system with long-context requests. How do you protect the others without losing useful diagnostics?
Further reading
Use primary documentation to check assumptions. These references explain underlying techniques; they are not answer keys for the scenarios above.
Build the habit, not a memorized answer
Daily practice gives you repeated opportunities to compare architectural choices and identify gaps in your reasoning. ByteRun's AI and cloud challenges complement longer mock design interviews; timed questions alone do not replace drawing and defending a complete system.
Start with one free daily question and earn XP. Membership unlocks the full daily round and playing past rounds.