The generalizability of a construct driven approach to SJTs Eugene Burke, Chief Scientist Carly Vaughan, Managing Consultant April 2011 ©2011 SHL Group Limited What we are going to cover The largest multinational, multi-language and multicultural SJT programme that we are aware of Two projects that provide evidence of proof of concept Highlights in terms of the evidence from those projects A tentative answer to the question what do SJTs measure ©2011 SHL Group Limited 2 Part I: Development of SJTs for The European Union ©2011 SHL Group Limited 3 Imagine you are asked to: Design 1,625 SJT items over 3 years 125 items to be delivered every 6 weeks That map to five competencies And have to operate in 3 languages To produce 40 equivalent tests That operate at two different job levels ©2011 SHL Group Limited 4 And the items and tests: Have to be calibrated and pre-trialled Shown to be effective psychometrically Shown to be effective linguistically Fair by gender, age and nationality ©2011 SHL Group Limited 5 An example item You work as part of a technical support team that produces work internally for an organisation. You have noticed that often work is not performed correctly or a step has been omitted from a procedure. You are aware that some individuals are more at fault than others as they do not make the effort to produce high quality results and they work in a disorganised way. a. Explain to your team why these procedures are important and what the consequences are of not performing these correctly. b. Try to arrange for your team to observe another team in the organisation who produce high quality work. c. Check your own work and that of everyone else in the team to make sure any errors are found. d. Suggest that the team tries many different ways to approach their work to see if they can find a method where fewer mistakes are made. ©2011 SHL Group Limited 6 Competencies covered • Analysis and problem solving: Identifies the critical facts in complex issues and develops creative and practical solutions • Prioritising and organising: Prioritises the most important tasks, works flexibly and organises own workload effectively • Delivering quality and results: Takes personal responsibility and initiative for delivering work to a high standard of quality within set procedures • Resilience: Remains effective under heavy workload, handles organisational frustrations positively and adapts to a changing working environment • Working with others: Works co-operatively with others in teams and across organisational boundaries, and respects differences between people ©2011 SHL Group Limited 7 Building blocks for the solution Leading & Deciding Deciding & Initiating Action Leading & Supervising Supporting & Cooperating Working with People Adhering to Principles & Values Interacting & Presenting Relating & Networking Persuading & Influencing Presenting & Communicating Analysing & Interpreting Writing & Reporting Applying Expertise & Technology Analysing Creating & Conceptualising Learning & Researching Creating & Innovating Formulating Concepts & Strategies Organising & Executing Planning & Organising Delivering & Meeting Expectations Following Instructions & Procedures Adapting & Coping Adapting & Responding to Change Coping with Pressure Enterprising & Performing Achieving Goals & Objectives Entrepreneurial & Commercial Thinking Universal Competency Framework ©2011 SHL Group Limited 8 Mapping to client’s competencies Leading & Deciding Supporting & Cooperating Working With Others Interacting & Presenting Analysing & Interpreting Analysing & Problem Solving Creating & Conceptualising Organising & Executing Prioritising & Organising Adapting & Coping Resilience Enterprising & Performing ©2011 SHL Group Limited Delivering Quality & Results Delivering Quality & Results 9 Behavioural blueprint for items Behaviour matrix Analysing & Problem Solving EPSO Competency Making Judgements UCF Behavioural Component Positive indicator Negative indicator Takes all available information into account when making judgements Makes quick judgements based upon incomplete information Passage with statements ©2011 SHL Group Limited 10 Delivering quality and results You work as part of a technical support team that produces work internally for an organisation. You have noticed that often work is not performed correctly or a step has been omitted from a procedure. You are aware that some individuals are more at fault than others as they do not make the effort to produce high quality results and they work in a disorganised way. a. Explain to your team why these procedures are important and what the consequences are of not performing these correctly. b. Try to arrange for your team to observe another team in the organisation who produce high quality work. c. Check your own work and that of everyone else in the team to make sure any errors are found. d. Suggest that the team tries many different ways to approach their work to see if they can find a method where fewer mistakes are made. ©2011 SHL Group Limited 11 Language equivalence ©2011 SHL Group Limited 12 Evidence of the process • Trialled on a total of 70,000 • This includes a competition with real 38,000 applicants in 2010 • Full range of p-values • Ranging from 0.4 to 0.8 with a mean of 0.6 • Internal consistency reliabilities averaging at around 0.8 • Multiple test forms are equivalent in terms of reliabilities, means and SDs as well as subscore correlations • Competency domain scores that are coherent (see next slide) • Correlations with other constructs that are consistent with published research (see nest slide) ©2011 SHL Group Limited 13 Correlations between test scores Test Verbal Numerical Abstract Reasoning Reasoning Reasoning SJT 0.36 0.23 Average 0.17 Prioritising & Organising 0.25 Score Delivering Quality Results Working With Others Prioritising & Organising 0.27 Working With Others 0.28 0.31 Resilience 0.24 0.29 0.26 Analysis & Problem Solving 0.27 0.33 0.30 Resilience 0.22 Source EPSO 01 ©2011 SHL Group Limited 14 Part II: Development of SJTs for a Global Retailer ©2011 SHL Group Limited 15 Imagine you are asked to: Design an SJT for a global retailer Who wants to start with a pilot in the UK Then evaluate the test in Turkey and the US And then roll it out globally in local languages ©2011 SHL Group Limited 16 Some choices for that client • Use the classic inductive approach – One option is a local instrument per country and language – In addition to time and cost, how would you know that the instrument is assessing common qualities across populations? – How would you ensure a consistent talent acquisition strategy across geographies? • Use a forced etic approach – Develop in UK English and then “localise” into other languages – How does this ensure that the instrument is effective across all cultures and languages • Use the combined emic-etic approach combined with the criterion-centric approach – Which is the strategy that we used ©2011 SHL Group Limited 17 Deductive approach to SJTs The criterion-centric approach is founded on a theory of the job That uses a behavioural taxonomy to define the critical behaviours that underpin effective performance By using the links between behaviours and attributes of people the constructs for the SJT are defined Combine this with the combined emic-etic approach to evaluate the generalisability of these constructs across cultures and languages ©2011 SHL Group Limited 18 Of these competency dimensions Leading & Deciding Deciding & Initiating Action Leading & Supervising Supporting & Cooperating Working with People Adhering to Principles & Values Interacting & Presenting Relating & Networking Persuading & Influencing Presenting & Communicating Analysing & Interpreting Writing & Reporting Applying Expertise & Technology Analysing Creating & Conceptualising Learning & Researching Creating & Innovating Formulating Concepts & Strategies Organising & Executing Planning & Organising Delivering & Meeting Expectations Following Instructions & Procedures Adapting & Coping Adapting & Responding to Change Coping with Pressure Enterprising & Performing Achieving Goals & Objectives Entrepreneurial & Commercial Thinking ©2011 SHL Group Limited 19 These were identified Supporting & Cooperating Working with People Interacting & Presenting Relating & Networking Delivering & Meeting Expectations Organising & Executing Adapting & Coping ©2011 SHL Group Limited Adhering to Principles & Values Adapting & Responding to Change Coping with Pressure 20 An example item A customer has accidentally broken a bottle of oil. There is broken glass and the oil is quickly spreading across the floor. One of your colleagues is helping the customer. You still have to re-fill a lot of shelves before your break. Watch your colleague for a moment to see if your help is needed. Help your colleague deal with the situation by calling the cleaning staff. Stay focused on completing your current task of re-filling the shelves in time. 21 Overlap between SJTs forms 65% UK Turkey 65% 88% 71% US ©2011 SHL Group Limited 22 An example change to an item A customer has accidentally broken a bottle of oil. There is broken glass and the oil is quickly spreading across the floor. One of your colleagues is helping the customer. You still have to re-fill a lot of shelves before your break. Watch your colleague for a moment to see if your help is needed. Help your colleague deal with the situation by calling the cleaning staff. Stay focused on completing your current task of re-filling the shelves in time. 23 Cross validation UK Predictive (N=70) Working with People 0.35 0.38 0.31 0.25 0.32 Adhering to Principles & Values 0.28 0.18 0.55 0.21 0.32 Relating and Networking 0.37 0.52 0.36 0.29 0.38 Meeting Customer Expectations 0.46 0.28 0.22 0.26 0.30 Adapting to Change 0.38 0.19 0.26 0.32 0.29 Coping with Pressure 0.38 0.18 0.43 0.10 0.28 Overall Rating 0.47 0.40 0.36 0.28 0.37 Criterion Measure Turkey US Concurrent Concurrent (N=93) (N=101) Sample Weighted Average (N=342) UK Concurrent (N=78) All correlations are corrected for range restriction but not attenuation in the criterion measures ©2011 SHL Group Limited 24 Differences in performance 70% 63% 60% Lowest scorers on the SJT 5 : 1 being lowest performers on the job 50% 40% 30% 20% 46% Lowest Quartile Mgr Ratings 13% 4% 10% 0% Lowest Quartile SJT Scores ©2011 SHL Group Limited Highest scorers on the SJT 12 : 1 being highest performers on the job Highest Quartile Mgr Ratings Highest Quartile SJT Scores 25 Key messages from our presentation ©2011 SHL Group Limited 26 Common problems with SJTs No clear logic in design and construction of content Empirical rather than construct driven scoring Not designed to anticipate localisation and generalizability needs These are the problems we have tried to address through a criterion-centric and combined emic-etic approach ©2011 SHL Group Limited 27 Thank you eugene.burke@shlgroup.com carly.vaughan@shlgroup.com You will also find us on ©2011 SHL Group Limited 28