Score the engineering decisions you can inspect: how the candidate defines the problem, judges the patch, chooses checks, and takes responsibility for the outcome. Use the same categories whether the candidate types every line or directs a coding tool. Record tool fluency separately when it matters to the role.
A clean diff cannot tell you who noticed a broken contract. A fast implementation cannot tell you whether the candidate checked an adverse case. The score needs a traceable connection to something the candidate wrote, reviewed, ran, or explained.
Separate tool access from assessment mode
Two decisions belong on the interview record. First, what resources were permitted? Second, was the exercise written analysis or execution?
A manual candidate can perform an execution exercise. A tool-assisted candidate can produce only a written review. Tool use does not determine which evidence standard applies.
In written analysis, assess the supplied artifact and the specificity of proposed checks. In execution, assess observed results and their limits. Do not compare totals from these modes. If an outage forces a written fallback, label that change before discussing scores.
Within a shared execution format, give candidates the same problem, time box, relevant materials, and disclosed resource policy. Record interruptions and access differences so the panel can identify gaps that require another observation.
Define categories before watching the candidate
Use categories that describe work rather than personality. A practical scorecard can include problem framing, brief quality, patch judgment, review depth, verification, communication, and ownership. Give each category a distinct question:
| Category | Question the evidence should answer |
|---|---|
| Problem framing | Did the candidate turn the report into a testable goal? |
| Brief quality | Could someone act on the scope, constraints, and acceptance cases? |
| Patch judgment | Did the candidate accept or reject changes for concrete reasons? |
| Review depth | Did they trace the behavior across the relevant boundaries? |
| Verification | Did their checks distinguish a correct repair from plausible mistakes? |
| Communication | Could another developer follow the decisions and limitations? |
| Ownership | Did they make a clear readiness decision and assign remaining work? |
Do not award communication points simply for confident delivery. An accurate explanation of an incomplete patch provides useful evidence. A fluent explanation that skips the changed authorization rule leaves that question unanswered.
Anchor scores to observable differences
Define what each level means before the round. Four dimensions offer a compact calibration exercise:
| Dimension | 0 | 1 | 2 | 3 |
|---|---|---|---|---|
| Framing | Wrong or undefined goal | Restates the request without an acceptance case | Testable goal and bounded scope | Also names dependencies, non-goals, and stop conditions |
| Risk | Ignores a material hazard | Names a hazard without a control | Defines the boundary and required proof | Specifies checks and recovery for material failure modes within scope |
| Evidence, written mode | No proof or a false success claim | Promises unspecified checks | Concrete main-path inputs and expected results | Discriminating main and adverse cases, with unverified work explicit |
| Ownership | Unexplained or unowned work | Vague handoff and unresolved assumptions | Owner, next check, and residual risk | Owner also responsible for rollout, measurable stop conditions, and recovery |
This compact table can guide calibration; a more detailed category scorecard still needs its own observable anchors. Avoid counting the same action repeatedly without explaining which separate capability it demonstrates.
Set role expectations in advance. If rollout judgment matters, supply enough deployment context to discuss it. An unstated production assumption cannot become a requirement only after the candidate leaves.
Keep proposed proof separate from observed proof
Suppose the task is to stop a profile save when a required phone number is missing. The candidate writes: “Test the empty field.” That leaves the input, route, and expected behavior unclear.
A stronger written answer specifies three cases: clearing the browser field prevents submission; a direct API request with the field missing is rejected without updating the profile; a valid profile still saves. It also names any account-type ambiguity and preserves existing format rules.
Those are useful planned checks. In an execution round, they do not earn credit as completed tests merely because they appear in a handoff note.
For execution evidence, level 2 requires relevant local results actually observed, including their limits. Level 3 requires observed main and adverse cases plus any integration proof required for the exact patch. A passing browser check leaves the direct API boundary unresolved until that boundary is checked.
Keep an evidence note such as: “Candidate proposed all three routes; only the browser case was observed.” This supports a precise discussion without inventing success or treating every environment limitation as a judgment failure.
Compare decisions across implementation paths
Consider two hypothetical candidates. One writes the patch manually and spots that the server still accepts an empty field. The other directs a tool, rejects a proposed change to phone-format rules, and requests a narrower patch.
Both actions can demonstrate technical judgment. Inspect their reasons and follow-through: what contract did they protect, what code did they inspect, and what check supported the decision?
Ask both candidates to explain a rejected option and identify the highest-risk changed line. Require both to own the final patch. Neither manual authorship nor tool use establishes that ownership by itself.
Treat an unexplained patch as an evidence gap even when it looks polished. Treat an incomplete patch according to the work actually shown, with the missing behavior recorded.
Calibrate against artifacts before making a recommendation
Have interviewers score independently before discussing the candidate. For each score, record a quotation, brief excerpt, review finding, or observed check. Then compare the evidence behind disagreements.
“They seemed senior” does not resolve a disputed verification score. “The candidate checked the browser but never exercised the direct server request” identifies the missing observation. If a follow-up is needed, name the question it must answer.
A total can summarize a rubric; it does not establish a hiring threshold or demonstrate that the interview predicts job performance. Keep the recommendation tied to role requirements, observed work, unresolved risks, and the limits of the exercise.
The Agentic Developer Interview includes a hiring-rubric card covering observable categories, tool fluency, rejected options, and final code ownership for tool-assisted and manual candidates.