Stage 3 · 9 min read read · Last reviewed 2026-05-23

Structured Interviewing with ATS: Scorecard Design and Bias Reduction

Structured interviewing is the practice of asking every candidate for a role the same questions and scoring them against the same criteria. The evidence base for this is substantial — structured interviews are 2x more predictive of job performance than unstructured “culture fit” conversations.

Most companies say they do structured interviews. Most companies have interviewers who ask different questions, score inconsistently, and make decisions based on gut feel confirmed post-hoc by a scorecard.

This guide covers how to implement structured interviewing properly using ATS scorecard tools.

The four components of a structured interview

1. Fixed questions per stage. Every candidate for the same role, at the same interview stage, is asked the same questions in the same order. There is no improvising.

2. Defined criteria per question. Each question is linked to a specific competency (problem-solving, communication, technical depth). The scorecard has a written definition of what a “strong,” “adequate,” and “weak” response looks like for that competency.

3. Independent scoring before discussion. Each interviewer submits their scorecard independently, before discussing with other interviewers. This is the hardest part to enforce — but it’s the part that eliminates anchoring bias.

4. Calibration after scoring. After independent scoring, the interview panel discusses discrepancies. If one interviewer gave “strong” and another gave “weak” on communication, that discrepancy is more informative than either individual score.

How to build scorecards in major ATS tools

Greenhouse

Greenhouse’s scorecard builder is the most mature in the SMB/mid-market category:

Define competencies for the role. Go to Job Setup → Scorecard. Add competencies relevant to the role (e.g., “Analytical thinking,” “Cross-functional communication,” “Technical depth in Python”).
Define what “strong” looks like. For each competency, write a “strong” descriptor — the observable behaviour that constitutes a top score. Example: “Strong: candidate demonstrates structured problem decomposition with explicit assumptions; weak: candidate jumps to solutions without articulating the problem.”
Assign competencies to interview stages. Not every competency should be assessed in every interview. Assign: recruiter screen → culture/motivation, first interview → problem-solving + communication, technical interview → technical depth, final → leadership/ownership.
Set required vs optional fields. In Greenhouse, you can mark scorecard fields as required before the interviewer can submit. Use this to ensure complete feedback.
Interview kit. Greenhouse’s “interview kit” combines the question bank, role background, and scorecard into one document that interviewers access before the interview.

Ashby

Ashby’s scorecard functionality is simpler but growing:

Create interview plans. Ashby uses “interview plans” that define the stages and the interviewers for each stage.
Add questions to each stage. Questions are attached to specific stages. Interviewers see their assigned questions in the Ashby interface.
Rating scales. Ashby uses numeric ratings (1–5 or configurable). Descriptors can be added but require more manual configuration than Greenhouse.
Analytics integration. Where Ashby adds value is linking scorecard ratings to post-hire performance data (when you feed that in). Ashby’s interviewer quality scoring identifies which interviewers’ assessments are most predictive of long-term performance.

Workable

Workable has a basic scorecard system:

Evaluation criteria. Workable allows you to add evaluation criteria to each stage. Interviewers rate each criterion on a star scale.
Interview kits. Similar to Greenhouse — interviewers see the criteria and questions before the interview.
Limitation. Workable’s scorecard doesn’t have Greenhouse’s competency-level definition capability. You can rate “problem-solving” but you can’t write a detailed descriptor of what “strong” looks like in a way that’s shown to interviewers at scoring time.

Bias reduction techniques

Separate application review from video/profile

Before reviewers see a candidate’s video, LinkedIn profile, or photo, have them score the resume anonymously (name and photo removed) against job criteria. ATS tools that support anonymised review: Greenhouse (has anonymisation module), Lever (partial).

Enforce independent scoring

The single most important anti-bias practice: require interviewers to submit scorecard ratings before the debrief discussion. When interviewer A’s opinion influences interviewer B before B has scored, B’s rating is measuring A’s opinion, not the candidate.

Configure your ATS to hide other interviewers’ scores until all scores are submitted. Greenhouse supports this natively (hide scorecard responses until submitted). Ashby has this as a settings option.

Calibration sessions

Schedule a 30-minute calibration after each round of interviews. The agenda:

Each interviewer states their overall rating and top concern — without knowing others’ ratings
Reveal all ratings simultaneously
Discuss discrepancies — not to converge on a number, but to understand what different interviewers observed
Make the advance/reject decision with discrepancies documented

Discrepancies between experienced interviewers on the same candidate are information. Two experienced interviewers giving very different scores should prompt a third interview, not an averaging.

Use the same questions across underrepresented and majority groups

Interviewers naturally ask harder questions to candidates they see as “not the typical profile.” Fixing interview questions eliminates this — the same question, asked the same way, to everyone. This is the core of structured interviewing and the hardest norm to maintain.

What the data should tell you

If you’ve been running structured interviews for 6+ months, your ATS data should be able to answer:

Interviewer consistency: Do interviewer X’s ratings correlate with other interviewers’ ratings for the same candidates? If interviewer X always rates much higher or lower than others, their scoring may be uncalibrated.

Predictive validity: Among candidates hired, do first-interview scorecard ratings correlate with 6-month performance reviews? This requires a data link between your ATS and your HRIS/performance tool. Ashby makes this easier; Greenhouse supports it via Harvest API integration.

Demographic parity: Are candidates from different demographic groups passing each stage at similar rates, controlling for the scorecard scores? A large discrepancy at a specific stage (e.g., women pass the recruiter screen at the same rate as men, but at 30% lower rates at the first interview) is a signal to investigate that stage’s structure.

Practical starting point

If you’re starting from unstructured interviews:

Week 1: Define 3–5 competencies for one high-volume role. Write one “strong” descriptor per competency.

Week 2: Create the scorecard in your ATS. Require all interviewers to submit scores before debrief.

Week 3: Run the first calibration call with the interview panel. Note discrepancies.

Month 2–3: Review whether your scorecard identifies candidates who then perform well. Adjust descriptors where they don’t.

Scale to other roles only after the first one is working. Structured interviewing at half your roles is more valuable than a broken system at all of them.