Skip to content

Instantly share code, notes, and snippets.

@aseba

aseba/ats-scorecard-benchmarking-2026-02-03.md

Created February 3, 2026 13:05
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save aseba/4bf08194cbfc77bf20f9baf214b31c7f to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save aseba/4bf08194cbfc77bf20f9baf214b31c7f to your computer and use it in GitHub Desktop.
Download ZIP
ATS Scorecard Benchmarking Analysis - Greenhouse, Ashby, Lever, SmartRecruiters
Raw
ats-scorecard-benchmarking-2026-02-03.md

ATS Scorecard Benchmarking Analysis

Date: February 3, 2026
Prepared by: Ava Chen
Requested by: Jenny Rector

Executive Summary

This report analyzes how leading ATS platforms handle interview scorecards — from creation and customization to usage in interview pipelines and feedback compilation. The analysis covers Greenhouse, Ashby, Lever, and SmartRecruiters, with observations on emerging trends and implications for Remotely.

Feature Greenhouse Ashby Lever SmartRecruiters
Scorecard Setup During job setup; templates + custom Part of interview plans; highly flexible Feedback forms at company/dept/job level Integrated with job requisitions
Customization Attributes, focus attributes, categories Custom forms, briefing views, AI summaries Multiple response formats, scorecard format Competency-based, five-star scale
Rating Scale 5-point (Definitely Not → Strong Yes) 1–4 scale with pass/fail icons 4-point (Strong No → Strong Hire) 5-star or custom scales
Feedback Blinding Configurable by permission policy Built-in blinding controls Siloed before collective review Configurable
AI Features AI-summarized notes AI-powered feedback summaries, candidate assistant CRM automation Interview intelligence integrations
Mobile Experience Yes Optimized Yes Strong mobile support

1. Greenhouse

Scorecard Creation & Setup

Who creates scorecards:

  • Site Admins and Job Admins configure scorecards during job setup
  • Scorecards are required by default (can be made optional)
  • Templates can be created at the organization level and applied to jobs

Setup workflow:

  1. During job creation, admin defines attributes (skills, traits, qualifications)
  2. Attributes are categorized (e.g., Technical Skills, Interpersonal Skills, Qualifications)
  3. Admin selects focus attributes for each interview stage
  4. Focus attributes can be linked to specific interview questions

Customization options:

  • Add custom attributes per job
  • Categorize attributes by type
  • Set focus attributes per interview stage
  • Make scorecards optional per stage
  • Bulk edit scorecards across multiple jobs

How Scorecards Are Used in the Interview Pipeline

Interviewer experience:

  1. Scorecard automatically assigned when interview is scheduled
  2. Can also be manually assigned (e.g., for group interviews or informal feedback)
  3. After interview, interviewer fills out scorecard with:
    • Key Takeaways: Conclusions, pros, cons, follow-ups
    • Attribute Ratings: 5-point scale with icons
    • Overall Recommendation: Definitely Not / No / Yes / Strong Yes
  4. Private notes can be added (visible only to Site Admins with specific permissions)
  5. Can @ mention other users for follow-up

Rating system:

  • 5-point scale using icons:
    • ✗✗ Definitely Not
    • ✗ No
    • ~ Mixed or Neutral
    • ✓ Yes
    • ✓✓ Strong Yes

Feedback Compilation & Reporting

Visibility controls:

  • Interviewers can be prevented from seeing others' scorecards until they submit their own (configurable)
  • Job Admins' access to scorecards is configurable by permission level
  • Site Admins can view all scorecards for all candidates

Reporting features:

  • Scorecard Feedback Report: Generates aggregate view of all feedback
  • Default view: past week's scorecards
  • Filter by job, stage, interviewer
  • Data Connector integration for custom reporting (Google Sheets, etc.)

Candidate Roundup:

  • Dedicated feature for reviewing submitted scorecards as a group
  • Compares focus attribute ratings across interviewers
  • Facilitates collective hiring decisions

Key Strengths

  • Mature, well-documented system
  • Strong DE&I features (anonymized reviews, structured scoring)
  • Granular permission controls
  • Focus attributes drive interview consistency

Key Weaknesses

  • Can feel rigid for some teams
  • Editing interview stages requires support ticket
  • No native AI summarization (relies on integrations)

2. Ashby

Scorecard Creation & Setup

Who creates scorecards:

  • Recruiters and admins during job setup
  • Part of the Interview Plan configuration
  • Templates + per-job customization

Setup workflow:

  1. Create structured interview plans with defined stages
  2. Configure custom feedback forms per stage
  3. Add interviewer instructions and candidate context
  4. Define scoring criteria and scales

Customization options:

  • Fully customizable feedback forms
  • Per-stage, per-role flexibility
  • Custom fields on candidate profiles, jobs, offers
  • Briefing views with interviewer instructions

How Scorecards Are Used in the Interview Pipeline

Interviewer experience:

  1. Receives notification with one-click access to review
  2. Sees briefing view with:
    • Custom feedback form
    • Interviewer instructions
    • Candidate details/context
  3. Fills out structured assessment
  4. Real-time interview notes with automatic timestamps
  5. Scores on 1–4 scale (pass/fail indicated by green/red icons)

Candidate Reviews (new feature):

  • Bulk request reviews from hiring managers on sourced candidates
  • 1–4 scoring scale
  • Pass (3–4) = green thumbs-up; Fail (1–2) = red thumbs-down
  • AI-powered summaries highlight patterns and suggest next steps

Feedback Blinding:

  • Built-in controls: interviewers can only see feedback once they've submitted their own
  • Prevents bias and influence

Feedback Compilation & Reporting

AI-powered features:

  • AI Feedback Summaries: Automatically highlights strengths and areas to review
  • AI Candidate Assistant: Chat interface for getting up to speed on candidates
  • Summaries cite original feedback for context

Reporting:

  • Visual pipeline view across jobs
  • Scorecard submission rates tracked (84–89% typical, majority within 2 hours)
  • Quality of Hire surveys with customizable forms
  • Central view of all Candidate Reviews across organization

Debrief support:

  • AI summarizes interview feedback for debriefs
  • Cites original sources for verification

Key Strengths

  • Modern UX, loved by hiring managers
  • Native AI summaries (no third-party needed)
  • Flexible enough for any process
  • Strong collaboration features
  • Seat-based pricing (predictable costs)

Key Weaknesses

  • Newer platform, less enterprise adoption
  • CRM features less mature than Lever
  • Smaller ecosystem of integrations

3. Lever

Scorecard Creation & Setup

Who creates scorecards:

  • Admins create feedback forms (Lever's term for scorecards)
  • Can be defined at:
    • Company level (default for all jobs)
    • Department level
    • Job posting level

Setup workflow:

  1. Work with HR and hiring managers to define standardized questions
  2. Build questions into feedback forms
  3. Select response formats per question:
    • Single line text
    • Text/code boxes (long-form)
    • Multiple choice
    • Yes/No
    • Checkbox
    • Scorecard format (quick multi-criteria rating)
  4. Assign forms to interview stages

Customization options:

  • Multiple response format types
  • Can mix scorecard + free-text questions
  • Templates shareable across roles
  • Interview kits include feedback forms

How Scorecards Are Used in the Interview Pipeline

Interviewer experience:

  1. Receives interview kit with feedback form
  2. After interview, fills out form with ratings and notes
  3. 4-point rating scale:
    • 1 = Strong No Hire
    • 2 = No Hire
    • 3 = Hire
    • 4 = Strong Hire
  4. Must provide hire/no-hire recommendation at every stage
  5. Notes and feedback visible to entire hiring team

Structured interviewing emphasis:

  • "Stick to the script" philosophy
  • Standardized questions across all candidates
  • Siloed scoring before collective discussion

Feedback Compilation & Reporting

Collaboration features:

  • Shared scorecards visible to hiring team
  • Notes and feedback centralized in candidate profile
  • Interviewers submit independently before discussing

Reporting:

  • Robust analytics: time-to-fill, diversity tracking
  • Compare average scores across candidates
  • Assess interviewer leniency/strictness over time
  • Track correlation between scores and future job performance

Best practices emphasized:

  • Collect feedback promptly (within hours)
  • No inter-interviewer discussion before submission
  • Use scores as primary decision factor

Key Strengths

  • Strong CRM + nurture features
  • Easy to implement (<24 hours)
  • Collaborative by design
  • Good analytics for process improvement

Key Weaknesses

  • Less flexible than Ashby
  • Editing interview stages requires support ticket
  • UI/UX not as modern as competitors

4. SmartRecruiters

Scorecard Creation & Setup

Who creates scorecards:

  • Admins configure scorecards as part of job requisition
  • Aligned with organizational competency frameworks

Setup workflow:

  1. Define job-specific competencies
  2. Create measurement scale (typically 5-star)
  3. Add culture fit assessment criteria
  4. Include hire/not-hire recommendation

Typical scorecard components:

  • Job-specific competencies
  • 5-star (or custom) measurement scale
  • Culture fit assessment
  • Notes on candidate responses
  • Areas of concern
  • Hire / Not Hire recommendation

How Scorecards Are Used in the Interview Pipeline

Interviewer experience:

  1. Structured feedback forms on web and mobile
  2. Rate competencies on defined scale
  3. Add notes and concerns
  4. Provide overall recommendation
  5. Strong mobile experience for on-the-go feedback

Integration capabilities:

  • Integrates with interview intelligence tools (recording, AI search)
  • Assessment platform integrations (TestGorilla, etc.)
  • Results flow into candidate profile

Feedback Compilation & Reporting

Analysis features:

  • Compare scorecard results to future job performance
  • Identify interviewer calibration issues
  • Measure process effectiveness over time

Candidate experience:

  • Candidate experience surveys integrated
  • Compare feedback across roles, locations, recruiters

Key Strengths

  • Enterprise-grade scalability
  • Strong integration ecosystem
  • Interview intelligence partnerships
  • Good mobile experience

Key Weaknesses

  • Less startup-friendly pricing
  • Can feel heavy for smaller teams
  • Less flexibility than Ashby

5. Other Notable Implementations

Workday Recruiting

  • Enterprise-focused, integrated with Workday HCM
  • Competency-based scorecards tied to job profiles
  • Strong for large organizations with existing Workday investment

BambooHR

  • SMB-focused, simpler scorecard functionality
  • Basic rating scales and notes
  • Good for teams with straightforward processes

Workable

  • Balanced mid-market offering
  • Customizable scorecards
  • Good collaboration features

6. Comparison Matrix

Dimension Greenhouse Ashby Lever SmartRecruiters
Ease of Setup ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐
Customization Depth ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐
Interviewer UX ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐⭐
Feedback Blinding ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐
AI/Automation ⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐⭐
Reporting ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐
Mobile Experience ⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐⭐
Enterprise Ready ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐

7. Key Trends & Best Practices

Emerging Trends

  1. AI-Powered Summaries

    • Ashby leads with native AI summarization
    • Reduces time to synthesize feedback across interviewers
    • Surfaces patterns and suggests next steps

  2. Feedback Blinding

    • Standard practice to prevent groupthink
    • Interviewers can't see others' feedback until they submit
    • Critical for reducing bias

  3. Interview Intelligence Integration

    • Recording interviews for later search/analysis
    • AI-powered natural language queries ("Who showed leadership?")
    • Reduces reliance on real-time note-taking

  4. Candidate Reviews (beyond interviews)

    • Ashby's new feature for sourcing alignment
    • Hiring managers rate profiles before interviews
    • Faster feedback loops on sourcing strategy

  5. Quality of Hire Tracking

    • Correlating scorecard ratings with post-hire performance
    • Identifying which criteria actually predict success
    • Continuous process improvement

Best Practices

  1. Standardize questions — same core Qs for all candidates in a role
  2. Require recommendations — every interviewer must say hire/no-hire
  3. Submit promptly — within hours, not days
  4. Blind feedback — no discussion before everyone submits
  5. Focus attributes — each stage assesses specific criteria
  6. Track effectiveness — correlate scores to job performance
  7. Train interviewers — ensure consistent calibration

8. Implications for Remotely

What to Adopt

  1. Focus attributes per stage (Greenhouse model)

    • Each interview stage assesses specific criteria
    • Reduces redundancy, improves coverage

  2. AI-powered summaries (Ashby model)

    • Auto-generate debrief summaries from raw feedback
    • Cite original sources for context

  3. Feedback blinding (all platforms)

    • Prevent interviewers from seeing others' feedback until submission
    • Table stakes for reducing bias

  4. Candidate Reviews (Ashby model)

    • Allow hiring managers to rate profiles before interviews
    • Accelerates sourcing alignment

  5. Mobile-first experience (Ashby/SmartRecruiters)

    • Hiring managers often review on-the-go
    • Quick feedback submission critical for velocity

What to Avoid

  1. Overly rigid structures

    • Allow some flexibility in question format
    • Don't over-constrain interviewers

  2. Paper-style scorecards

    • Static forms that don't leverage data
    • No AI assistance or pattern recognition

  3. Opaque feedback compilation

    • Make it clear how feedback is aggregated
    • Show interviewers how their input is used

Differentiation Opportunities

  1. AI-native from the start

    • Build AI summarization as core feature, not add-on
    • Use LLMs to detect patterns across interviews

  2. Talent marketplace integration

    • Unique to Remotely: scorecards tied to talent profiles
    • Portable feedback that travels with talent

  3. Continuous calibration

    • Real-time alerts when interviewer ratings diverge from norms
    • Built-in interviewer training nudges

  4. Outcome tracking by default

    • Automatically correlate interview scores to job performance
    • Surface insights on which criteria predict success

Sources

Greenhouse

Ashby

Lever

SmartRecruiters

Comparisons

9. Visual Resources (Demos, Videos, Screenshots)

Greenhouse

Videos:

Documentation with screenshots:

Ashby

Videos:

Documentation/Product pages:

Lever

Videos:

Documentation:

SmartRecruiters

Videos:

Documentation:

10. Remotely-Specific Recommendations: Calibration Use Case

Based on Jenny's input that the primary use case is calibration (knowing what didn't work about candidates to improve future match quality), here are targeted recommendations:

Key Features to Prioritize

  1. Structured rejection reasons

    • When a candidate doesn't move forward, capture why in structured fields
    • Categories: skills gap, culture fit, experience level, communication, etc.
    • This data feeds back into sourcing quality

  2. Attribute-level feedback (not just overall scores)

    • Score candidates on specific criteria (technical skills, soft skills, role fit)
    • Aggregate this data across candidates to identify patterns
    • "We keep rejecting candidates for X" → adjust sourcing criteria

  3. Feedback tied to talent profiles

    • Unique to Remotely: feedback can travel with talent across placements
    • "This candidate was a 4/5 on technical but 2/5 on communication at Company A"
    • Helps match to roles where their strengths align

  4. Closed-loop reporting

    • Dashboard showing: what reasons are candidates being rejected for?
    • Filter by: role, customer, time period
    • Identify systemic issues in sourcing or vetting

Competitive Benchmarks for Calibration

Platform Calibration Features
Greenhouse Quality of Hire surveys; focus attributes per stage; scorecard feedback reports
Ashby Candidate Reviews (pre-interview alignment); AI pattern detection; pass rate analytics
Lever Score correlation to performance; interviewer calibration tracking
SmartRecruiters Scorecard-to-performance comparison; interviewer effectiveness analysis

Differentiation Opportunity

None of these platforms are built for talent marketplaces where:

  • The same talent may be placed at multiple customers over time
  • Feedback from one placement improves matching for the next
  • Remotely has visibility across the full talent lifecycle

This is a significant opportunity to build something unique.

Report complete. Happy to dive deeper into any section or explore specific implementation approaches for Remotely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment