– A nerdy guide that will ensure you the highest possible job performance predictability in assessment and recruitment – which unfortunately is still not high.
To begin with the bad news, the highest possible validity in personal assessment, predicting job performance is 40%.
This was recently presented to me at a test certification in General Mental Ability (GMA aka Intelligence testing). Held by CfL, Occupational Psychologist Erik Blatt Lyon.
40% is to me shockingly low, especially when you consider how big the recruitment/assessment industry is and how recruitment process is conducted.
For one thing, if the highest possible validity one can obtain in a recruitment process is 40%, most of the recruitment industry is selling hot air when guaranteeing 6, 12 or even 24 months… Not saying that companies are not getting the +6 months, but simply saying that the consultancy has very low predictable security when offering it and that it is primarily up to the employing company and the employee to succeed on this.
To me, the 40% screams that we need to change the industries ways of working. To stop selling false security and instead opening up for a more agile process with higher company and candidate involvement and interactions.
For instance, most recruitment processes are a matter for the hiring manager, consultant and/or HR and the candidate(s), while it is no secret that the Opportunity for performance is a much more complex matter, involving the team and internal and external stakeholders among others.
40% is better than nothing
Being a positive spirit this lead me to dive into the research in the area, to find the way to the highest possible validity and the conclusions are clear: Use of hiring methods with increased predictive validity leads to substantial increases in employee performance as measured in increases in output, increased monetary value of output, and increased learning of job-related skills.
The most important of a personnel assessment method is predictive validity, thereby the ability to predict future job performance and job-related learning.
The following is based on the research conducted by Frank L. Schmidt, In-Sue Oh & Jonathan A. Shaffer, an update on Schmidt and Hunters all in all including more than 100 years of research. They have analysed the validity of 31 one procedures for predicting job performance.
The study quantitatively studies and the conclusion that I am extracting from it in the following will be made from generalised assumptions, not specific employee groups or organisational levels. Another important disclaimer is that I in this piece only look at the assessment part of a recruitment and other parts of the process does also play a key impact on the outcome.
In my opinion, an assessment can never be greater that the job analysis before it.
The overall best predictors: GMA combined with Integrity test and/or Structured interviews
GMA – The highest validity, lowest application cost and the single most researched method.
The reason why is that it says the most about the ability within job-related learning. When an employer uses GMA to select employees who will have a high level of performance on the job, that employer is also selecting those who will learn the most from job training programs and will acquire job knowledge faster from experience on the job.
In fact, most selection procedures shown to be valid for training performance are also valid for predicting job performance.
On this matter, it is my experience that you always need to take the role to be filled into consideration and match the needed GMA to the role.
There is some truth to the old saying ‘what you don’t have in your head you must have in your legs’.
And it is my experience that a high GMA can be a disadvantage as in some cases a high GMA comes with a motivation towards solving difficult analytical problem and that time from “Getting things done”.
GMA primarily speaks to the candidates’ ability to succeed, not motivation and opportunity, which leads to the following extract from the Bath Model:
Ability*Motivation*Opportunity=Performance
Integrity test
A not often used tool, that tests probability of counterproductive work behaviours on the job, such as fighting, drinking or taking drugs, stealing from the employer, equipment sabotage, or excessive absenteeism. Integrity tests do predict these behaviours, but surprisingly they also predict overall job performance.
The explanation to why Integrity test has a high predictive factor on overall job performance could be that it measures on the personality traits of Conscientiousness, Agreeableness, and Emotional Stability, this scale measures tendencies towards being sincere, modest, fair, and honest.
Some test houses have specific Integrity test (f.ex. Cut-E), while others have parts of it included in their personal assessment test (f.ex. Hogan).
Needed to be added to this, that why this is a strong supplement to GMA testing is due to the fact that integrity tests correlate nearly zero with GMA.
What is also clear from the studies that not all personality traits have a high predictability factor, more on that in the following.
Structured interviews
The obvious opposition to an unstructured interview, structured interviews have a fixed format and a set of questions to be answered. It has a fixed procedure for scoring responses and a planned evaluation (or rating) format. The questions to be asked are usually determined by a careful analysis of the job in question and cannot be deviated from by the interviewer. Structured interviews are costlier to construct and use.
To my experience, a structured interview gives a better foundation for evaluation, by giving less power to stomach feeling, ‘daily fitness’ and first impression and more attentions to the ‘actual matters’.
The structured interview can have great value especially in the early stages of a recruitment process, but that it is important to note that it is often neither motivating for the interviewer or the person interview – we all like to feel unique and be able to bring our stories and experiences into play.
a semi-structured interview is often the best way to go and something that many companies could gain great value from implementing. With a semi-structured interview, you get most of the positive effects from a structured interview while still giving room for individuals and for the conversation to “grow”.
What NOT to look for
Job experience (years)
One of the procedures in the other end of the scale of predictability is Job experience, there simply is a very small predictable validity in years of experience in a similar job and performance.
This is something that we should be aware of in assessments, it is my experience that evaluation of past performance instead of potential plays way too big a role in today’s assessments.
The research has been based on job experience ranging from 6 months to 30 years, and the highest predictable factor was found where the experience did not exceed 5 years.
Apparently, during the first 5 years on these (mid-level, medium complexity) jobs, employees were continually acquiring additional job knowledge and skills that improved their job performance. But by the end of 5 years, this process was nearly complete.
Likewise, age has very low predictive validity.
Years of Education
It is important to note that this finding does not imply that education is irrelevant to occupational success; education is clearly an important determinant of the level of the job the individual can obtain. What this finding shows is that among those who apply to get a particular job, years of education does not predict future performance on that job very well.
Again, to me, it is one of the typical misunderstandings that leads to evaluating past performance rather than future potential.
Extraversion – personality trait
The glorification of extroversion is to me one of the most common mistakes when working with personal assessment, luckily this is also backed up by the studies. One of the areas where I most often meet this is when assessing sales profiles. Some of the best sales profiles I have met are in fact introverts.
To elaborate a bit and to put on the line, extroverts will often find it easier/more comfortable to approach new people and find it natural to get a conversation going, but will often struggle with focusing on the goal of the conversation (sales etc.) and can find it harder to listen and ask relevant questions.
The same example can be used in a job interview where extroverts often will shine more than introverts in a first (unstructured) interview.
Honestly, the discussion on extroverts vs. introverts are not that interestingto me, I have only given it some space here as I often repeated mistake here.
Graphology – The analysis of handwriting.
Graphologists claim that people express their personalities through their handwriting and that one’s handwriting, therefore, reveals personality traits and tendencies that graphologists can use to predict future job performance.
Apparently, Graphology is a quite popular method in some parts of the world (f.ex. France and Israel). Not surprisingly there is almost no validity in the method and the research shows, that a trained graphologist predict just as well (which is not high) as an untrained person.
Emotional Intelligence
Something that is getting quite a bit of attention at the moment, but unfortunately according to the studies the validity is very low. In fact, most psychologists, do not accept that emotional intelligence is a trait or construct. According to the “believers” in Emotional Intelligence, there are two types of emotional intelligence measures: personality-based measures and ability based measures.
I recommend:
Even though that 40% is not much it is still something and if you convert it to the possible outcome that working professionally with recruitment and assessment can have on your organisation’s performance.
To me, the conclusion from the studies on the specific methods are a great inspiration but all recruitment/assessments are unique and it is important to choose the right methods for the specific process.
One thing that I do see many recruitment processes lacking is structure, a clear mythology and reasoning for why we do what we do.
In the world of HR there is a tendency for recruitment being the lowest in the hierarchy of working areas but at the same time being one of the work areas that deliver the most value, with clear and specific initiatives and results. There is a tendency of firefighting in recruitment, always being late and never connecting the business strategy with the recruitment/succession/resource strategy.
At the same time, even though the recruitment industry is huge measured in turnover, unfortunately, I do not experience that the consultancies in markets do much to lift the bar.
When I look into my crystal ball I see a future, where recruitment will play a much bigger role as candidates are shifting jobs more often and a bigger part of the workforce are working as freelancers (jumping from assignment to assignment and thereby being recruited rapidly to new assignments)
It is no secret I am on a mission to place recruitment higher on the HR/organisational development agenda, work smarter and harder and implement better structures as well as to help organizations to connect the dots from an organisational pain or potential to realising the potential (recruitment to me is somewhere in the middle of that value string).
I hereby hope that this post will inspire You to give thoughts to how you choose the right methods for your assessments in order for your organisation perform optimally.
Please don’t hesitate to get in touch, share or comment on my post, I would love to get a little debate going
Further readings:
Links:
Schmidt & Hunter (1998):
http://mavweb.mnsu.edu/
Schmidt, Oh & Shaffer (2006)
http://mavweb.mnsu.edu/
HBR, 21st-Century Talent Spotting
https://hbr.org/2014/06/21st-century-talent-spotting
Egon Zehnder, Look for past performance to see potential
http://www.egonzehnder.com/leadership-insights/look-past-performance-to-see-potential.html
Youtube High Potential: A Discussion With Dr. Robert Hogan
https://www.youtube.com/watch?v=peCMaMOZKXw
Books:
The Bath Model: Purcell, J, Kinnie, N & Hutchinson, S (2003a) Open minded. Inside the black box: overview.People Management, 9 (10).
