A picture is worth a thousand words or, perhaps, a hundred thousand dollars in extra salary. Academics claim that personality traits inferred using AI photo analysis can predict how depicted individuals will fare in the labor market.
They emphasize that they don’t advocate doing so because personality extraction from facial images is fundamentally discriminatory.
Even so, they say, personality screening is already commonplace among admissions and HR committees, and AI tools that offer personality assessment are seeing rapid adoption. So they argue that an academic evaluation of the technology is necessary.
In a paper titled “AI Personality Extraction from Faces: Labor Market Implications,” the authors describe how they used the LinkedIn facial images of over 96,000 MBA graduates to extract subjects’ Big Five personality traits – Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.
The machine learning algorithm used was originally described in a 2020 Scientific Reports paper titled “Assessing the Big Five personality traits using real-life static facial images,” one of about two dozen papers cited as “ML-laundered junk science” in a 2024 paper titled “The reanimation of pseudoscience in machine learning and its ethical repercussions.”
The algorithm “uses facial features to predict self-reported personality, rather than others’ perceptions of personality based on visual appearance,” according to the authors of the new paper.
By applying this algorithm, the authors found “that personality traits inferred from facial features provide substantial incremental predictive power for labor market outcomes.”
The researchers determined that applying machine learning to infer personality traits from facial images produced accurate predictions for the rank of undergraduate and MBA programs attended by the depicted individual, initial compensation, salary trajectory, and job transitions.
Were an HR department to use a similar technique to assess the personality of managerial applicants, the result could serve as a forecast of the job applicant’s future performance in the labor market – biased though it may be. And that appears to be happening.
Co-author Marina Niessner, assistant professor of finance at Indiana University, told The Register in a phone interview that companies like banks already use personality surveys in hiring and promotion decisions and that AI hiring companies are starting to use technology like Big Five personality trait analysis on video interviews.
“The regulatory environment, as you probably know, is very uncertain,” said Niessner. “And so we don’t think this is necessarily a valid way to do it [or] that companies should be doing it. But I think it’s really important to have an academic evaluation of these methodologies if there’s even going to be a regulatory discussion around this.”
The paper argues that the AI-based screening needs to be considered in conjunction with the alternative, which is human decisions based on physical appearance that may also be inconsistent or biased.
The other atuthors were authors Marius Guenzel (Wharton), Shimon Kogan (Reichman University), and Kelly Shue (Yale University). ®