Quantitative Measurement of Project Manager Competencies Research Team 306 Implementation Session 2014 CII Annual Conference July 21–23 • Indianapolis, Indiana Research Team 306 Quantitative Assessment of Project Manager Competencies Academic Team Members Jose A Buitrago, Jr, CSA Group Awad S. Hanna, University of Wisconsin- Madison Richard D. Carrithers, Jr. , DTE Energy Jeffrey S. Russell, University of WisconsinMadison George Carter, Anheuser-Busch InBev Carol C. Menassa, University of Michigan Alexandra E. Beletic, University of WisconsinMadison Industry Team Members Chair: Richard D. Bradford, Bechtel Group, Inc Vice Chair: Jeffery L. Moore, Ameren Missouri Dale C. Allen, Shell Global Solutions Robert Bryant, BP Refining & Logistics Technology Jeff Drinkard, Hargrove Engineers + Constructors Ian D. Etzkin, URS Corporation Roque Francisquez, SABIC - Saudi Basic Industries Corporation Bryan Hoerner, Audubon Engineering Company, LP Richard A. Krout, Walbridge Rita M. Norat, Air Products and Chemicals, Inc. Nancy Skinkle, Architect of the Capitol Frank Williamson, Wood Group Mustang Agenda • Research Summary – Literature Review – Data Collection and Analysis – Tool Development and Validation • Tool Walk Through • Panel Discussion • Questions/Answers Research Objectives The primary purpose of RT306’s research was to develop a consistent methodology to define and quantitatively measure the competencies of Project Managers (PMs) in order to: – Identify PM competency gaps – Focus PM training needs and development – Develop and validate an implementable and useable Project Manager Competency Assessment Tool (PM CAT) PM CAT (Project Manager Competency Assessment Tool) A user-friendly tool designed to quantitatively assess and score PMs’ competencies. 1. Provides a thorough list of competencies and definitions that a PM should possess to successfully lead, manage and execute a project 2. Assess the competencies of a PM and compares him/her to the industry 3. Improves PM effectiveness by targeting specific development needs to enhance their competencies in a personalized and more efficient manner 4. Enhance the selection process of qualified or potential PMs Background • PMs must have the technical and leadership expertise, training, and characteristics to bring about a successful realization of project goals • Should continuously improve their skills through development and training • Historically been evaluated by qualitative means • CII commissioned RT-281 – Project Management Skills for the Future in 2010 – Anticipate the PM competencies to respond to industry changes and future challenges – Served a s a platform for RT-306’s research Research Methodology Literature Review • Industry-wide need for a simple and quantitative PM Competency Assessment Tool • Existing PM competency models • Discover, include, define and properly weight all critical PM competencies • Organized into four competency groups: – Knowledge/Experience (altered from RT-281’s Technical/Virtual), – Management – Cognitive (& Attributes) – Leadership Existing Competency Model examples…. NASA Development Framework Leadership Model: Management (NASA 2008) Corporate Education Group Project Management Competency Model (CEG 2008) RT-306 List of PM Competencies Data Collection: Two Surveys Preliminary Survey Detailed Survey • Objective of survey: • Objective of survey: – Eliminate unimportant competencies – Test competencies on industry PMs – Assign competency weights – Determine industry performance levels and benchmarks • Administered to RT-306 Members • 5-point scale – Ensure each competency is statistically significant • Administered to PM Supervisors – asked to answer about an average PM and a “stellar” excellent PM • Questions were developed with the team and professionally edited to pinpoint the meaning of each competency Competency Weights (from first survey) Knowledge/Experience (KE) Awareness of and knowledge to use of state of the art technology Business/Financial Acumen Certification/Training Continuously Monitors and is Aware of Similar EPC projects Disciplinary Understanding of all Phases of EPC Projects and their Interrelationship Disciplinary Understanding of a PM Job Management (M) Ability to Look Ahead and Plan Build Knowledge Network Business Developments/Ability to Sell Communications Management Focus on Client's Needs Human Resource Management Integration Management Internal and External relations Issues Management/Conflict Resolution Knowledge and Management of Legal Issues Organizational Savvy Organize Project Staff and Process Expertise Project Controls Leadership in Safety Risk Management Quality Management 22.1% 13.8% 18.6% 14.0% 13.5% 20.1% 20.1% 25.0% 6.4% 6.0% 6.3% 6.5% 6.9% 5.3% 5.7% 6.3% 6.5% 5.5% 6.3% 6.9% 7.0% 6.8% 5.9% 5.7% Cognitive (CO) Achievement and Action Analytical Thinking Courage Impact and Influence Initiative Personal Effectiveness Self-Awarness Strategic Thinking Vision Attributes (CO-A) Assertive/Aggressive/ Result Driven/ Decisive Adaptable/Approachable/Attentively Listens to Others/Easily Relates to Others Detail Oriented/ Organized Energetic and Enthusiastic Credible/ Honesty/ Ethics/ Integrity/ Trustworthy/ Loyalty/ Empathetic Mature/ Possesses Self-Control Skills/Professional Positive Attitude/ Selflessness Accountable/ Responsible/ Reliable Leadership (L) Build Coalitions within Project Team Build Consensus Build Trust Cultural and Ethnic Sensitivity/ Leverages Diverse Thinking Develop and Mentor Others Directness/Influential Engages Others Innovation Possess Strategic Insight Team Builder 23.6% 11.2% 10.8% 10.7% 11.2% 12.0% 10.7% 11.0% 11.5% 11.2% 11.4% 12.4% 12.5% 12.6% 13.4% 12.8% 12.4% 12.5% 29.3% 10.5% 9.2% 11.1% 9.8% 9.2% 9.7% 10.1% 9.3% 10.0% 11.1% Data Collected Respondent and Company Information • 77 PM Supervisors from (at minimum) 29 companies – 154 PMs were analyzed Company Type Company Projects Architect 12% Engineer 15% Contractor 31% Owner 42% Private 47% Public 53% • Company Size Company Locations: • United States (92%) • International (8%) [28%] < 250 Million 8% 28% 28% [12%] 250 Million - 1 Billion [24%] 1 Billion - 25 billion 12% 24% [28%] 25 billion-50 billion [8%] > 50 billion Excellent versus Average Project Manager Excellent PM Total Number of Years working as PM Number of Years Working as a PM for this Company 13.1 Average PM 11.3 6.8 6.6 Total number of Years of Relevant Previous Experience 15.2 16.8 Number of Projects completed as a PM 42.8 21.8 Age 46.6 46.8 Data Analysis • Paired t-test – Tested if excellent PM scores were significantly different then the average PM scores • All competencies (except for one*) were statistically different and therefore necessary to quantifiably measure a successful PM • PM scores were calculated with weights from first survey – to make the industry distribution – to create benchmarks for the PM CAT • Formula to calculate PM scores: • Used in the PM CAT – where j denotes Project Manger – wi is the weight of the category – sij is the composite score of competency j for category i (this is the individual question score times the competency weight) *The one exception, where there was no statistical difference between average and excellent PMs, was the attribute “Credible/Honesty/Ethics/Integrity/Trustworthy/ Loyalty/ Empathetic”. Formula Further Broken Down 1. Overall Formula Scores were normalized at each step! 2. Summed Categories 3. Summed Competencies within the Category (only Knowledge and Experience shown) Tool Description • Reliably and consistently measure PM competencies • Excel 2007 (v 12) or later versions • Rates PM’s competence on an industry-based scale • Identifies training and development opportunities (competency gaps) • Helps in the selection or placement of PMs • Should be completed by the PM's Supervisor • Questions are job specific and relate to any phase of EPC projects • PM CAT can be administered at any time during PM’s career – can be administered multiple times on one person over time to see progression • Should not be confused with or substituted for a performance rating system Tool Validation: Beta Testing • 14 PM supervisor participants • 93% accuracy – tool is suitable for use – of the limited overseas sample (3 respondents): 100% accuracy • Respondents said – the tool overall was useful to somewhat useful • 2.7 on a scale of 5 – user friendly to very user friendly • 3.35 on a scale of 5 • Asked if the research objectives of the tool were met: identity competency gaps, focus training and development, and develop and validate implementable/useable tool. – Yes: 10 – Maybe: 4 – No: 0 • Asked if they would use the tool in their company – Yes: 9 – Maybe: 2 – No: 3 • Tool enhancements done based on feedback received The deployment of the PM CAT will help enhance overall project performance by improving the process by which Project Managers are developed and selected. This, in turn, will contribute to CII’s mission to improve overall industry performance Tool Input Questions focus on the competencies that a PM should possess to successfully lead, manage & execute a project Tool Output • PM Competence Percentile – Rates PM’s competence on an industry-based scale – Illustrated on a graph against industry benchmarks • Detailed Competency Report – Highlights PM’s strong and weak competencies – Opportunities for development and training (competency gaps) Percentile Score and Graph Detailed Competency Results Tool Walk Through… Panel Discussion Research Objectives The primary purpose of RT306’s research was to develop a consistent methodology to define and quantitatively measure the competencies of PMs in order to: – Identify PM competency gaps – Focus PM selection, training needs and development – Develop and validate an implementable and useable PM CAT Question and Answers