By Joel Bertrand, Trevor Sparrow, and Sujatha Jagdeep
Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Organizational Performance and Workplace Learning Department.
Spacely Sprockets (a pseudonym) specializes in the manufacture of aerospace products. The corporate mission of Spacely Sprockets is simply “client success,” which drives the expectation that every product must work the first time, every time. Spacely Sprockets relies on accurate application of torque when tightening mechanical fasteners during the manufacturing process. Torque is a twisting force; too little or too much torque applied in the manufacturing process can lead to unreliable and unsafe products and failure to meet its mission.
Several years ago, a calibration technician observed a wide variation in employee performance while using torque tools. One of the actions in response to the observation was the 2013 implementation of revised torque certification training for all individuals who use a torque tool, with mandatory recertification every 24 months.
Re-certification training began in 2015 and included a pre-training evaluation of individual accuracy with five torque tools commonly used during manufacturing. To receive recertification, individuals were required to use all five tools accurately on a torque analyzer that measures the actual amount of torque applied.
The Performance Problem
In 2013, trainees demonstrated proficiency (within 5% of the required torque value) with all five torque tools. However, prior to re-certification training, trainees demonstrated proficiency with an average of only three out of five tools, and there were failure rates of up to 69 percent on certain types of tools. Obviously, at some point between initial training and re-certification 24 months later, individuals were losing proficiency in this critical skill required in manufacturing.
During fall 2015, we, the co-authors of this case study, formed a team to pursue our team project for OPWL529, the needs assessment course in the Master of Science program in organizational performance and workplace learning at Boise State University, taught by Dr. Don Winiecki. Our needs assessment for Spacely Sprockets was to determine causal factors contributing to the observed decline in torque accuracy and provide recommendations for interventions to close the performance gap.
The Performance Gap
We identified a measurable performance gap, using data obtained from the pre-training evaluation results of individual performance with torque tools:
- The current level of performance: Individuals demonstrate proficiency in applying torque (within 5 percent of the applicable torque requirement) with only three out of five torque tools (60 percent) on the recertification pre-training evaluation.
- The optimal level of performance: Individuals must demonstrate proficiency in applying torque (within 5percent of the applicable torque requirement) with all five torque tools (100%) on the recertification pre-training evaluation.
At the outset, we recognized that a continued performance gap could lead to unreliable and unsafe products. The challenge for us was to develop a clear picture of the potential root causes and credible recommendations from the sources of data available.
Needs Assessment Framework
We planned a systematic needs assessment, in which each stage of data collection and analysis informed and provided input for successive stages in the process (Rossett, 2009, p. 42). Schensul and LeCompte (2013) describe the value of a formative evaluation model that guides the perceptions of the researchers (p. 63). We chose two frameworks, Gilbert’s (2007) behavior engineering model (BEM) and Langdon’s (2000) language of work model (LOW), to guide the design of data collection instruments and the analysis of collected information at each stage.
Gilbert’s (2007) BEM provided us with an accessible framework for systematically examining the factors related to not only the individual worker’s qualities such as knowledge, capacity, and motives, but also the environmental supports such as data, instruments, and incentives.
Langdon’s (2000) LOW model lays out the fundamental elements of performance, as they exist in the interactions of individuals with each other, their equipment, and organizational factors (p. 15):
- The proforma (input, conditions, process element, outputs, consequences, and feedback)
- Layers (behavior, standards, support, and human consonance)
- Levels (business unit, core processes, individuals, and work groups)
We used a multistage process of data collection.
Stage 1: We reviewed existing data such as pre-training evaluation data and training documents obtained from the client organization, which assisted us to identify trends related to individual performance with torque tools. Then we conducted open-ended interviews with key informants (instructor, manufacturing engineers, manufacturing technicians, and supervisors) to obtain data regarding what individuals actually accomplish while operating a torque tool, and other factors that influence their performance.
Stage 2: Data collected during Stage 1 allowed us to identify what parts of the work process and environment should be the focus of our observation of technicians, which included posing structured follow-up questions based on a number of key issues we identified in Stage 1:
- Are individuals utilizing the technique (hand position and form) taught in training?
- What are the environmental factors that influence performance?
- What interpersonal interactions are involved?
- How and where do employees receive feedback on torque performance?
- Are there observable challenges or barriers to performance? If so, to which factors do they relate?
- What job aids or performance supports are available?
One of our team members observed manufacturing technicians at one of several final assembly production areas located at the client site.
Stage 3: After we identified the ways performance factors varied in actual work processes, we conducted a survey with employees returning for torque recertification training. The survey enabled us to both verify the existence of, and quantify, potentially performance-affecting variations in the workplace.
We applied the following techniques to analyze the data collected during all three stages:
|Level / Domain / Factor
| 1. PEWSTA -
Performance, Expectations and Work Standards regarding Torque
|| 1.1 MVSO - Mission and Vision Statements of the
1.2 PEE - Performance Evaluation of Employees Address
1.3 CSE -
Clear Standards and Expectations
AFTA - Adequate Feedback regarding Torque Accuracy
|| 2.1 IFTA -R -
Immediate Feedback -- Reinforcing Desired Behavior
2.2 IFTA -C - Immediate Feedback
-- Correcting Torquing Behavior
2.3 DFTA - Delayed Feedback
*** Addendum to each of the
feedback codes originally assigned (T / V / H) Feedback delivered-- T, in
Training; V, during Verification; H, on HardwareFeedback on Torquing
JATA - Job Aids regarding Torque Accuracy
|| 3.1 JAA -S - Job Aid is Available
and Sufficient to Support Performance
3.2 JAA -I - Job Aid is
Available and Insufficient to Support Performance
3.3 JANA - Job Aid Not
Available 3.4 JANA - Job
Aid Not Required
- Applied the codes to the data obtained during existing data and document review, open-ended interviews, and performance observations. This provided us with a systematic and quantifiable view of the occurrence (or absence) of different behaviors or conditions. The grounding of the codes in the organizational performance models provided insight into which of the observed factors contributed to desired performance and which likely fueled the identified performance gap.
- Conducted a preliminary analysis between each phase of data collection, which guided the design of the activities in the following stages--focusing more deeply and narrowly on the trends identified. The analysis also spurred adjustments to the codebook to ensure that the codes enabled us to describe themes and trends in our observations accurately.
- Compiled and analyzed survey data quantitatively to confirm or rule out possible trends in observations from the first two stages, to investigate more deeply into issues that we had identified, and to facilitate determination of the variables that had the most influence within each factor.
- Triangulated responses from all stages to reinforce our analyses and aid in identifying potential root causes.
- Organized the coded data on visual representations of both the BEM (Figure 1) and LOW (Figure 2) to make it easier to identify trends and potential root causes.
Figure 1 illustrates how we used the fishbone diagram to represent our data analysis with each “rib” dedicated to one of the six elements of the BEM. We found this sort of blending of tools (in this case, the fishbone diagram and BEM) was nearly essential in enabling us to model the data in a way that maintained a close connection to the tools we were using to guide our process. We chose a simple coding motif to help us illustrate the relationship among the factors affecting performance:
- Green represents well-aligned factors that contribute to effective performance.
- Yellow identifies factors where our observations did not provide a clear link to the performance gap; either because of mixed behaviors or because we did not yet have sufficient data. A number of these blocks informed our recommendations for further study.
- Red identifies factors that were misaligned, therefore they linked directly to the performance gap identified. The red blocks on the diagram contributed directly to our identification of root causes.
Figure 1: Fisbone diagram demonstrating data trends in BEM framework.
Figure 2 illustrates the visualization of our analysis using the LOW model. We started by distributing observations in terms of the proforma in flowchart form, with this diagram. We reflected observations linked to causal factors in bold text and tagged the four potential causal factors of the performance gap.
We indicated factors linked to the layers and levels of performance with the round shapes that include an indicator for the specific layer or level, to capture the multidimensional nature of the LOW model.
Like the BEM, this representation shows the concentration of data related to feedback (CF1), which occupies a central position in the proforma. Additionally, the diagram clearly calls out the lack of clarity of the performance standard (CF2), challenges with tools (CF3), and gaps between work conditions and training (CF4).
Figure 2: Proforma of LOW model with linkages to layers and levels.
Potential Root Causes
The primary cause that e found was: There is insufficient performance feedback at the point of performance for individuals to maintain required torque performance.
The contributing causes included:
- Unclear or mixed standards for torque performance
- Variety or performance of tools
- Variable performance conditions not reflected in training or verification
Each of the causes was broken down into sub-factors to identify possible interventions to address and close the performance gap. Table 2 shows how we broke down the primary cause.
Table 2. Breakdown of Primary Cause
Performers receive feedback about torque performance on hardware less than once a month.
- Informants associate feedback with results rather than technique.
- It is hard to determine accuracy of applied torque by visual inspection.
- Introduce more frequent feedback in the performance environment
- Communicate observation guidelines to manufacturing engineers and working leads
Performers receive feedback primarily in response to problems.
- Manufacturing engineers and working lead providing support with problems.
- Technique not observed consistently or proactively to ensure consistent performance.
- Positive feedback is described as “praise” by informants.
- Relationship between technique and accuracy is well documented.
- Training for manufacturing engineers and working leads on the value of regular positive and negative feedback on performance
- Reinforce relationship between technique and accuracy
Feedback concentrated on verification.
- Focus is on verifying tools.
- Inconsistent reports of coaching/corrective feedback.
- Process recently introduced.
- Torque verification in ideal conditions--technique on hardware affected by environment.
- New process--did not exist during the period in which performance gap developed and was measured.
- Clarify expectations for manufacturing engineers and working leads regarding regular feedback
- Monitor verification data results to measure changes in individual torque accuracy
- Focus on-position observations on adaptation of technique
- Modify training presentation material to emphasize the importance of torque verification and the application of torque on hardware
We considered a number of performance improvement interventions to eliminate the potential root causes. Then, we used multi-criteria analysis (Watkins et al., 2012, p. 171) to prioritize the following interventions for recommendation to the client:
- Develop guidelines for performance feedback at the operational level, including key points of technique that manufacturing engineers and working leads could utilize to reinforce proper technique. Consistent feedback on proper technique in addition to daily verification results (which were not available during the period in which the gap developed) will further support improved performance.
- Provide effective communication to all affected employees to reinforce the relationship between proper technique, feedback from the tool, and accurate application of torque. Many survey respondents indicated that sensory feedback is their primary guide when self-assessing application of torque during the manufacturing process. However, incorrect technique would result in a skewed sensory feedback. This communication will be most effective if delivered in conjunction with feedback indicated in the first intervention stated above.
- Monitor daily verification data of torque operators to gauge the improvement in torque accuracy at the operational level, and to provide a consistent way to measure the effectiveness of interventions.
We determined that a combination of the three interventions would more effectively enhance torque performance than isolated interventions (Watkins et al., 2012, p. 176). As these three interventions can be associated with what Gilbert (2007) calls “data” (p. 87), they are perhaps the most economical type of intervention possible. These interventions can be implemented informally and efficiently by supervisors and between technicians themselves.
Challenges and Lessons Learned
We faced a number of challenges, but the most significant was access to information. A number of sources of data were deemed too competitively or strategically sensitive to be shared with us (due in part to the fact only one member of the project team is an employee of Spacely Sprockets). This made it difficult to pursue a number of aspects of this project. Despite the challenges with access to information, we were still able to isolate performance-impacting factors and recommend valuable interventions by:
- faithfully applying appropriate frameworks and combining them in ways that enhanced our ability to visualize and communicate performance issues, and
- triangulating observations to identify the causal factors contributing to the performance gap.
The focus of the recommendations on the operational level reflects the kinds of information to which we had access. However, by systematically working with the data within organizationally imposed constraints, the needs assessment pointed out a number of improvement opportunities for the client that reach to the tactical and strategic levels and that may have otherwise gone undiscovered without the findings of this project.
Gilbert, T. (2007). Human competence: Engineering worthy performance (tribute edition). [K2] [Y2] San Francisco, CA: Pfeiffer.
Langdon, D. (2000). Aligning performance: Improving people, systems, and organizations [K3] [Y4]. San Francisco, CA: Jossey Bass/Pfeiffer.
Rossett, A. (2009). First things fast: A handbook for performance analysis (second edition). San Francisco, CA: Jossey-Bass/Pfeiffer.
Schensul, J., & LeCompte, M. (2013). Essential ethnographic methods: A mixed methods approach (2nd edition). Lanham, MD: AltaMira Press.
Watkins, R., West Meiers, M., & Visser, Y. (2012). A guide to assessing needs: Essential tools for collecting information, making decisions and achieving development results. Washington,
D.C.: The World Bank. Retrieved from www.needsassessment.org
Joel Bertrand has 22 years of performance support experience in the aerospace industry, specializing in e-learning content development. Joel’s educational background includes a graduate certificate in workplace e-learning and performance support, a Bachelor of Science degree in management–human resources, and an Associate in Applied Science degree in education and training management. You may reach Joel at NMnative67@yahoo.com.
Trevor Sparrow has more than 20 years of experience in service management, training, and performance improvement in the telecommunications and government sector. He holds a Master of Science degree in organizational performance and workplace learning from Boise State University (December 2015). He may be contacted at email@example.com.
Sujatha Jagdeep has over eight years of experience as a high school biology teacher and lecturer of educational psychology, in both India and the United States. She has taught in both in-person and online environments. She is currently pursuing her Master of Science degree in organizational performance and workplace learning, and the workplace e-learning and performance support certificate at Boise State University. Sujatha may be contacted at firstname.lastname@example.org.