I've written before about the cost breakdown of running a 400-application funnel. This piece is different. This is about what I noticed happening to my own judgment over three weeks of screening, the mistakes I know I made and the ones I probably don't know I made, and what a different process would have looked like.
The role was an operations coordinator position at a consumer goods company. The job spec was clear. The candidate pool was large and active. By most definitions, this should have been a tractable recruiting problem. It wasn't, for reasons that had nothing to do with the candidates and everything to do with the format.
Week One: The Volume Lands
Four hundred and twelve applications came in over the first week. I read every resume. I took that seriously at the time and still believe it was the right call for that role, because keyword filters would have missed several candidates who turned out to be strong. But 27 hours of resume review is not a sustainable input for a single role, and I knew that while I was doing it.
By Thursday of week one I had a list of 61 candidates who seemed worth a call. I noted at the time that my early resume reviews were more careful than my later ones. The first 150 resumes got real attention. The back half of the stack got faster processing, because I was already thinking about the scheduling problem ahead and I was tired from the reading.
That meant some candidates in the back half of the alphabetically organized stack probably received less consideration than candidates in the front half. Not intentionally. Just structurally: the later resumes landed on a more fatigued brain with a tighter threshold.
The Pattern Problem in Phone Screens
I ran 18 phone screens over four days in week two. I had a list of questions I planned to ask. What I found is that the questions stayed consistent but my follow-up did not. Early screens got more probing. When a candidate said something interesting but incomplete, I asked the follow-up question. By screen 14 or 15, I was accepting surface answers more readily. Not because I had decided to lower my standard. Because I was tired of asking the same follow-up question eighteen times, and the answer I was going to get felt predictable before I asked.
The technical term for this is interviewer fatigue, but experiencing it feels different from reading about it. It doesn't feel like lowering your standard. It feels like you are getting faster and more efficient. The efficiency is real. The accuracy degradation is hidden.
I also noticed a pattern I am less comfortable describing, but I think it matters. Candidates who sounded confident and fluent on the phone scored better in my notes than candidates who sounded uncertain but said accurate, specific things. I caught some of these in review. I didn't catch all of them. There were eight candidates who advanced to the second round. Two or three of them probably had the fluency advantage working in their favor more than the content of their answers warranted.
The Feedback Loop I Didn't Have
I found out, eventually, that two of the eight second-round candidates didn't perform well in the panel interview. One withdrew before the panel. Two were hired; one worked out well, one left after five months. I never learned what happened to the 41 candidates I screened by phone but didn't advance. I don't know if anyone I passed on would have outperformed the people I advanced.
This is the feedback loop problem in recruiting. The cost of a miss, meaning a strong candidate you eliminated, is almost entirely invisible. The cost of a false positive, meaning a weak candidate you advanced who didn't get the job, is visible at the panel stage but tends to be attributed to the second round rather than the first. The first round rarely gets the credit for producing good candidates or the blame for failing to produce them.
Without that feedback, it is very hard to know whether your first-round screening is actually identifying the right people or just the people who are good at sounding right on a phone call.
What I Would Have Done Differently
I have thought about this particular funnel a lot since building Intervieux. Here is what a different version of it would have looked like.
The rubric would have been written before the first application arrived, not assembled informally in my head over the course of reading 412 resumes. The dimensions I actually used to make decisions, competency with competing priorities, clarity in cross-functional communication, process adherence versus exception judgment, would have been defined with score-level anchors before any candidate was in front of me.
Every candidate who passed resume review would have answered the same five questions, in writing or by voice, before I evaluated any of them. My evaluation would have happened against the rubric, not in real time during a phone call while managing the social dynamics of a live conversation. The order of evaluation wouldn't have followed the order of the original applications.
I would not have read candidate 301's responses after spending all afternoon on calls. I would have reviewed them in batches, when I was fresh, after all responses were in.
And the candidates I passed on would still have received nothing, because the signal problem is structural. But at least I would have been more confident that the eight I advanced had actually demonstrated the dimensions I cared about, rather than just having the advantage of arriving early in the queue or sounding fluid on the phone.
What This Doesn't Fix
I want to be honest about the limits. Structured async screening would not have made week one's resume review faster or easier. The initial triage problem, deciding which 60 to 80 candidates from a pool of 412 warrant further evaluation, is hard regardless of the format for the next step. Resume quality is a weak signal, and nothing I have described here changes that.
It also wouldn't have given me the candidates I missed entirely. If someone strong applied and I passed them at the resume stage, structured first-round questions wouldn't have helped them. The filter happens in sequence, and each filter is imperfect.
What it would have changed is the part where I spent 10-plus hours on phone screens with inconsistent follow-up, advancing candidates who benefited from my early-day attention or phone presence, and rejecting candidates whose written or async answers might have been clearly stronger than the ones I advanced. That part of the process was broken. A structured async format is the direct fix for that specific break.
See also: The Real Cost of Running a 400-Resume Funnel for One Role and Async vs. Live Phone Screens: What Changes and What Stays the Same.