Case Study: Steering "ECxplorers" Toward STEM Higher Education and Careers

By Carla Denton, Alia Eisele, and Ieva Swanson

Tales from the Field, a monthly column, consists of reports of evidence-based performance improvement practice and advice, presented by graduate students, alumni, and faculty of Boise State University’s Organizational Performance and Workplace Learning Department.

Background and Evaluation Request

In recent years, an increasing skill gap in America’s workforce has developed in industries that require a degree in science, technology, engineering, or math (STEM). The availability of STEM jobs is growing faster than the number of STEM graduates (The White House, Office of the Press Secretary, 2009).

The Engineering College (EC) at State University (SU) established a summer outreach program for exiting 8th and 9th grade students in 1997. In 2009, SU’s EC updated its week-long program to a three-day on-campus residential experience filled with STEM-based workshops designed to show students that interesting and rewarding STEM careers are within their capability. In honor of career exploration and the Engineering College itself, SU named the shortened program “ECxploration” and kept the focus on enrichment activities rather than academics.

ECxploration hosts 40 to 50 “ECxplorers” each summer in June, mostly drawing from the surrounding cities within an easy round-trip drive to campus. The ECxplorers complete eight workshops that showcase equipment and activities that the EC uses in STEM education, and that might be found in the workplace. Volunteer EC professors and paid student mentors (chiefly EC undergraduate and graduate students) staff the ECxploration program. When not participating in activities, ECxplorers and their student mentors stay in the dorms to experience typical college campus life. The ultimate goal of outreach programs like ECxploration is to increase enrollment and retention in SU’s STEM-related degree programs, as well as to generate STEM talent for the workforce.

The ECxploration program director requested this evaluation to determine how effective ECxploration is at connecting ECxplorers to a STEM career path, particularly via higher education through SU’s EC. The program director plans to continue ECxploration indefinitely, so our team designed a formative evaluation to explain what adjustments (if any) the ECxploration program might make to increase support of the program’s goals. Due to the changes in program delivery, our evaluation covers the “updated” years (2009-2015) rather than reaching all the way back to the program’s beginning in 1997.

The overarching question that we chose to guide this evaluation was: How well does ECxploration influence and steer kids toward STEM and SU’s EC, and what could be changed to increase support of the program goals?

Evaluation Questions and Dimensions

To address this question, we defined four program dimensions, gaining the client’s input to weight the relative importance of the dimensions (Table 1).

Table 1

A Breakdown of Evaluation Questions into Weighted Dimensions

Category

Dimension

Importance Weighting

Process

1. Workshop Design:

How well are ECxploration workshops designed to motivate ECxplorer interest in STEM careers?

Extremely Important

2. Mentor Support System:

How well is ECxploration’s mentor support system designed to contribute to ECxplorers’ campus life experiences?

Very Important

Outcome

3. Continuing Contact:

How many former ECxplorers return to SU’s EC for events and camps?

Important

4. College Enrollment Rate:

How many former ECxplorers go to college for further education in STEM fields?

Extremely Important

 

The first two dimensions looked at how the ECxploration program steers ECxplorers toward STEM fields, and the second two dimensions measured how well it is doing so. Because we were conducting a formative evaluation, we emphasized the process dimensions in our data collection and analysis.

Methodology

Our team used Chyung’s evaluation project guidelines (2015) as a framework to design and conduct this evaluation. Our evaluation was a blended goal-based evaluation (GBE) and goal-free evaluation (GFE) (Scriven, 1991), meaning that our data collection deliberately went slightly beyond the focus questions (evaluating the program’s goals) in case we had missed anything crucial in our data collection design. We kept the scope of exploratory questions small, so as not to exceed the resources we would expend for our GBE.

We built a program logic model (PLM), based on the logic model structure described by the W.K. Kellogg Foundation (2004), to illustrate how the overall ECxploration program draws on resources to produce services with lasting impacts on young people (Figure 1). Creating a program logic model helped us define ECxploration’s performance system and systematically identify its processes.

When addressing our two process dimensions, we selected some additional tools to help us systematically design our data collection questions with respect to the greater ECxploration performance system. We used Keller’s (1987) ARCS (attention, relevance, confidence, satisfaction) model to organize our data as we evaluated motivation and engagement levels for each of ECxploration’s workshops. Additionally, we used Chevalier’s (2008) performance analysis job aid to define ECxploration’s mentor support system and to help us sort the SU environment (beyond the program director’s control) from the mentor environment (within the program director’s control).

With the aid of those models, we designed data collection instruments for multiple methods of data collection (critical multiplism), including semi-structured interviews, web-based surveys, and document review. We collected data from numerous sources (triangulation)--including former ECxplorers (via their parents or guardians), former mentors, and the program director. We also reviewed extant data in the form of composite exit surveys for ECxplorers from 2009-2015, as well as the tracking records maintained by the program director and SU registrar.

 

 Figure 1. A program logic model of SU’s EC ECxploration summer enrichment program.


Overall Results

Based on our collected data, we evaluated the quality of each dimension using a three-level rubric:

  1. Meets/exceeds expectations
  2. Needs minor improvement
  3. Needs major improvements

Dimension 1: Workshop Design--Needs Minor Improvement

According to former ECxplorers, the majority of workshops (nine of 15) have adequate support in each of the four motivation elements of ARCS: attention, relevance, confidence, and satisfaction. We evaluated workshops independently, and for those workshops needing some improvement made specific recommendations based on input from former ECxplorers and mentors.

Dimension 2: Mentor Support System-- Needs Minor Improvement

Although the mentor support system currently addresses all areas of Chevalier’s (2008) performance analysis job aid, the program could make small improvements. The quality of individual hires is excellent. The most significant environmental improvements include providing more feedback to mentors on their performance, both during and after ECxploration.

Dimension 3: Continuing Contact-- Undetermined

At least 29% of ECxplorers have returned for one or more EC recruitment or STEM events. Our research at SU and other similar institutions did not produce any usable standards; data were too scant to derive standards, so we were not able to evaluate the return rate against a meaningful rubric.

Dimension 4: College Enrollment Rate-- Meets/Exceeds Expectations

The National Science Foundation (n.d.) reported that the state average for students pursuing STEM degrees and careers is 26.6% for SU’s state. Of the former ECxplorers that we were able to locate, 65.2% have declared a STEM-focused degree plan or are working in STEM fields; and 45.3% of former ECxplorers have enrolled at SU. It is still too early for many from the study population to have declared a major, but early numbers show over 50% of those choosing STEM at SU are in EC, which is well above the 10.4% national average for freshman intending to pursue engineering.

Recommendations

Our team recommends that EC continue to track outcomes and impacts of ECxploration after improving data collection methods (such as exit survey design and techniques). College enrollment data for ECxplorers who attended during the summers of 2009-2011 are already beginning to build a powerful picture of ECxploration’s success. Without better context, it is unclear if tracking former ECxplorers’ continuing contacts with SU EC is an informative practice. However, continuing to collect these data may prove helpful if EC develops evidence-based standards.

Formative evaluations are corroborated by program outcomes, but focus primarily on program processes. Our data collection for ECxploration’s outcomes indicates success, which aligns with our data-driven findings that only minor process improvements are needed (tweaks, rather than drastic change). We summarized the strengths of the ECxploration program in Table 2.

Table 2

ECxploration Program Strengths

Strengths

Importance Weighting

Reasoning

Workshop Design: ECxploration offers a variety of workshops that cover a range of STEM fields, as well as a range of activity types.

Extremely Important

ECxploration advertises a broad range of topics and delivers unique experiences that proactively engage kids from all socio-economic backgrounds using a range of methods to engage a broad spectrum of curiosity levels.

Workshop Design: ECxploration makes use of facilities unique to SU and to EC.

Extremely Important

To create forward momentum toward higher education, ECxploration goes beyond what exiting 8th and 9th graders can expect to encounter in their high school classrooms.

Mentor Support System: ECxploration attracts mentor staff with technical expertise, experience, engagement, communication skills, and an authentic passion for sharing STEM with younger people.

Very Important

EC begins hiring mentors by invitation and recommendation, focusing on candidates that have established relationships with the department. ECxplorers have a short time to make the most of the mentor relationship, so generating authentic rapport quickly is of great value in the program.

Mentor Support System: ECxploration establishes a culture where relationships and teamwork are paramount; ECxploration fosters autonomy and creativity.

Very Important

Mentors play a significant role in meeting the individual needs of ECxplorers; ECxplorers experience the benefits of role-modeling as well as active mentoring.

Continuing Contact: SU EC offers other outreach programs to maintain contact during the long period between ECxploration and undergraduate enrollment.

Important

Many ECxplorers return to SU EC to participate in more outreach programs. Former ECxplorers attending these events is a reflection of their continued interest and shows that ECxploration has been effective in steering kids toward SU EC education and STEM careers.

College enrollment rate: SU EC offers low-threshold admissions at a relatively low cost.

Extremely Important

The United States needs STEM-educated talent to fill jobs. Former ECxplorers enroll in SU EC at a substantially higher rate than do other students in this locale.

 

Due to lack of data, we made cautious recommendations about the nature of minor improvements to the following areas:

  • Ensure each workshop is planned with Keller’s ARCS elements in mind to keep ECxplorers engaged
  • Redistribute activity time to better align with program goals, such as steering ECxplorers (and mentors) toward careers in STEM
  • Drive clear communication to meet program goals
  • Adjust activities within some workshops to increase ECxplorer engagement
  • Improve exit survey design
  • Develop and reward mentors
  • Provide performance measures and feedback that clearly indicate how well a mentor is doing

Our evaluation of ECxploration’s workshop design and mentor support system also leads us to recommend tapping the mentors as another source of data to complete ECxploration’s annual surveys. The culture of ECxploration is very valuable. We feel it can best be preserved throughout any changes for improvement by continuing to involve the mentors’ input as much, or more, as in the past.

 

REFERENCES

Chevalier, R. (2008, November/December). The evolution of a performance analysis job aid. Performance Improvement, 47(10), 9-18. doi: 10.1002/pfi.20034

Chyung, Y. (2015). Evaluation project guidelines. Unpublished manuscript, Department of Organizational Performance and Workplace Learning, Boise State University, Boise, Idaho.

Keller, J. M. (1987, October). Strategies for stimulating the motivation to learn. Performance & Instruction, 1-7.

National Science Foundation. (n.d.). About NSF: NSF at a glance. Retrieved from http://www.nsf.gov/about/glance.jsp

Scriven, M. (1991). Pros[e] and cons about goal-free evaluation. American Journal of Evaluation, 12(1), 55-76. doi: 10.1177/109821409101200108

The White House, Office of the Press Secretary. (2009). President Obama launches “Educate to Innovate” campaign for excellence in science, technology, engineering & math (STEM) education [Press release]. Retrieved from https://www.whitehouse.gov/the-press-office/president-obama-launches-educate-innovate-campaign-excellence-science-technology-en

W.K. Kellogg Foundation. (2004). Logic model development guide. Battle Creek, MI: Author.

 

Author Bios


Ieva Swanson has 10 years of experience in the retail, food and beverage, and nonprofit fields applying her knowledge of adult learning to inspire, improve working conditions, and drive results. She holds a BA in cultural anthropology and linguistics from Rice University, and a graduate certificate in Workplace Performance Improvement from the Organizational Performance and Workplace Learning program at Boise State University. She is currently completing a master’s degree with an expected graduation date of May 2016.  She may be reached at ievaswanson@u.boisestate.edu.

 


Alia Eisele graduated from Boise State University’s Master of Science in Organizational Performance and Workplace Learning in December 2015, additionally earning a graduate certificate in Workplace Performance Improvement. She is a medically retired Army veteran with a strong interest in peer support of other disabled veterans, and founded and runs an informal support group for veterans dealing with chronic pain. Alia can be reached at alia.eisele@gmail.com.

 


Carla Denton currently lives in the Denver, CO, area. She worked as a marine biologist and served as an officer in the U.S. Coast Guard before entering the learning and performance field. She is working toward earning her MS in Organizational Performance and Workplace Learning through Boise State University and will complete her degree in May 2016. She can be reached at carlajdenton@gmail.com