Anthropic interview prep

Use the STAR format: describe the context and the decision you disagreed with, the specific concern you raised and how you raised it (data, an experiment, or a clear argument), the outcome, and what you would do differently. Show that you can advocate for your position while remaining constructive, and that you can commit to a decision once it has been made even when you disagree.

What a strong answer covers: The goal is to show technical judgment and professional communication, not to relitigate old grievances. Avoid framing the story as you versus them. A strong answer includes evidence you used to make your case.

Follow-up: What would you do if the team made the same decision again and the outcome was worse than you predicted?

Last reviewed 2026-10-07

Use STAR: explain the project goal, what you built, what happened, and what you did next. Own your share of the failure rather than attributing it entirely to data, compute, or other constraints. Name a specific technical mistake — a leaky split, a flawed evaluation metric, a mislabeled dataset — not a vague sense that the model was wrong. Describe what you learned and how you applied it on a subsequent project.

What a strong answer covers: A strong answer names a specific technical root cause, not bad luck or insufficient time. The retrospective action matters as much as the failure itself.

Follow-up: What would be the first thing you would investigate if you had three days to debug the same project now?

Last reviewed 2026-10-07

Use STAR: describe the decision, what information was missing, what assumptions you made and why, how you documented those assumptions, and what you did to limit downside risk. Show that you can move forward without perfect information while keeping the team informed about your uncertainty and the conditions that would change your decision.

What a strong answer covers: This tests judgment and risk management. A strong answer distinguishes between uncertainty that is worth waiting to resolve and uncertainty you can safely assume away for now.

Follow-up: How did you decide what additional information was worth waiting for versus what you could assume away?

Last reviewed 2026-10-07

Use STAR: set the scene, describe the failure and how you found it, and explain how you communicated it. Lead with the business impact first, then explain the cause at a level the audience can act on. Show that you gave them what they needed to make a decision rather than a full technical post-mortem. Include what you proposed to fix and what the outcome was.

What a strong answer covers: The best answers show empathy for what the stakeholder needs to know, not just what you know. Avoid jargon without explanation and avoid false reassurance about a fix that is not yet validated.

Follow-up: How would you set up a review process to catch similar issues before they reach stakeholders?

Last reviewed 2026-10-07

Use STAR: describe the problem you noticed, why it was not clearly owned, what you chose to do, how you involved or informed the right people, and the outcome. Show judgment about when to escalate versus act, and demonstrate that you understood the constraints and worked within or around them thoughtfully rather than bypassing important processes.

What a strong answer covers: This tests initiative and judgment. Interviewers probe whether you involved the right people at the right time. A strong answer shows you didn't silently take over someone else's work.

Follow-up: How did you decide this was your problem to solve rather than someone else's?

Last reviewed 2026-10-07

Give the model a clear task description, a typed tool schema, and instructions to emit structured calls. Validate tool arguments before executing them and return structured results including errors. Limit the number of steps and total tokens consumed. Separate the loop controller from the model to enable retries, logging, and policy enforcement. Keep tools idempotent where possible and require explicit confirmation before irreversible actions.

What a strong answer covers: Reliability comes from the system around the model, not the model itself. Non-determinism makes it hard to guarantee a specific execution path, so monitoring and graceful failure matter more than forcing a fixed sequence.

Follow-up: How would you prevent an agent from entering an infinite loop when a tool keeps failing?

Last reviewed 2026-10-07

Define tasks with known correct outcomes or verifiable end states. Measure task completion rate, number of steps, cost, and latency. Where ground truth is hard to define, use a judge model or a rubric to rate intermediate steps and final outputs. Curate a diverse evaluation set including edge cases and adversarial examples. Run evaluations deterministically using fixed seeds and mocked external calls, and track regressions across versions.

What a strong answer covers: End-to-end task completion is the primary signal but can be noisy. Per-step quality metrics help locate where a version regressed. Mention that judge models have their own biases and should be calibrated against human ratings.

Follow-up: How would you build an evaluation set for a task where correct answers are inherently open-ended?

Last reviewed 2026-10-07

Decide which content is essential for the current query. For conversations, keep the system prompt, the most recent turns, and a compressed summary of older turns. For documents, retrieve relevant chunks rather than passing the full text. Always inform the model which information has been omitted. Monitor how often truncation occurs and whether query quality degrades — that signals the retrieval or summarisation step needs improvement.

What a strong answer covers: Different strategies suit different use cases: retrieval for document QA, summarisation for long conversations. Truncating silently is usually worse than explicit omission with a note to the model.

Follow-up: How would you decide how much conversation history to summarise versus pass verbatim?

Last reviewed 2026-10-07

Enforce tool permissions outside the model: the model requests an action, the controller decides whether to execute it. Apply the principle of least privilege to each tool. Validate and sanitise all arguments before execution. Require human confirmation for high-impact or irreversible actions. Log every tool call with its arguments and result. Rate-limit calls and apply a token budget. Treat retrieved content as untrusted data and test for prompt injection in tool outputs.

What a strong answer covers: The model cannot be trusted to self-enforce safety boundaries. The system architecture must assume the model can be manipulated and enforce constraints in code, not prompts.

Follow-up: How would you prevent a retrieved web page from hijacking the agent with an injected instruction?

Last reviewed 2026-10-07

Define harm categories relevant to the deployment context: misinformation, toxic content, dangerous instructions, privacy leakage. Use both manual red-teamers and automated adversarial prompts. Test jailbreak patterns such as role-playing, hypothetical framing, and continuation prompts; test multi-turn strategies and long-context attacks. Score responses against a rubric and track which categories the model fails most. Feed failures back into training or build a deployment-time classifier. Document what was tested and what was not.

What a strong answer covers: Red-teaming is not complete coverage; it is a structured way to discover the most exploitable weaknesses before release. Automated tools find breadth; manual testers find creative depth. A strong answer covers both.

Follow-up: How would you measure whether a safety fix for one vulnerability improved or regressed a related capability?

Last reviewed 2026-10-07

Constitutional AI uses a set of written principles to guide both supervised fine-tuning and reward modelling. In the SL-CAI stage, the model critiques and revises its own outputs according to the principles, generating training data without per-example human annotation. In the RLAIF stage, a model rather than human raters evaluates responses against the constitution to produce preference labels. This makes the process more scalable and the evaluation criteria more explicit than standard RLHF, whose human judgments remain implicit.

What a strong answer covers: The key distinction is that RLAIF replaces or supplements human raters with a model, and the criteria are made explicit in a document. Mention that this introduces a new source of bias: the model used for evaluation can have its own undesirable preferences.

Follow-up: What could go wrong if the model generating RLAIF preference labels has its own systematic biases?

Last reviewed 2026-10-07

A model card describes: the intended use cases and explicitly out-of-scope uses, training data sources and known biases, evaluation metrics broken down by relevant subgroups, known limitations and failure modes, and recommendations for monitoring. It matters because downstream users make deployment decisions based on it — an incomplete card leads to misuse in contexts the model was not designed for. Include both what the model does well and where it is known to struggle.

What a strong answer covers: Subgroup-disaggregated evaluation is often the most critical section because aggregate accuracy can hide serious performance gaps on important demographics. A strong answer explains why omitting this is not just an oversight but a risk.

Follow-up: How would you keep a model card accurate and current as you retrain the model on new data?

Last reviewed 2026-10-07

Common issues: benchmark contamination (training data contains benchmark examples), saturation (models already score near ceiling, leaving no signal), narrow coverage (a benchmark captures only a slice of real-world use), and metric mismatch (BLEU or accuracy does not reflect the quality users care about). Use task-specific evaluations on held-out data not in training, human preference studies for open-ended outputs, and red-teaming for safety properties. Track metrics that correspond to deployment objectives.

What a strong answer covers: Benchmark contamination is the most frequently underestimated problem: even accidental inclusion of benchmark data in pretraining can inflate scores significantly and makes fair comparison between models trained on different data very difficult.

Follow-up: How would you design an evaluation suite for a customer-support LLM that could not be gamed by training on public benchmarks?

Last reviewed 2026-10-07