Outstanding Research/Student Research

 

Submission Deadline: September 15, 2016

Sponsored by
International Society for Performance Improvement
P.O. Box 13035
Silver Spring MD 20910
301.587.8570

awards@ispi.org

Overview: ISPI Awards of Excellence Program

ISPI's Awards of Excellence (AOE) program is designed to showcase the people, products, innovations, and organizations that represent excellence in the field of performance improvement. If you would like ISPI to consider you for an Award of Excellence, review this document and complete the appropriate submission forms. 

Outstanding Research/ Student Research

This award recognizes outstanding research in the field of Human Performance Technology (HPT) or in a related field such as Adult Education, Human Technology, Behavioral Psychology, and Vocational Education

  • The research can be conducted by a professional in the field of HPT or a related field
  • This category is also open to all students (master’s and doctoral) enrolled in a related graduate program. Dissertations, theses, or studies completed as part of an independent study or external contract to the institution will be considered

Scope

  • How we define “research” for ISPI awards
  • The categories of research that qualify for awards
  • The criteria for excellent research, worthy of an award
This document does not attempt to teach the research process, nor does it pretend to describe how the methods of scientific research can be applied to various situations.  For this reason, the research category relies upon the extensive research background of its Evaluators

Basic characteristics of HPT Research
Human Performance Technology is an applied field, more of an engineering discipline than a basic science, so when we speak of research in this context, we mean scientific research as it is applied to an engineering discipline.  The purpose of such research is to produce new applications, technology, or insights that improve our practice of HPT.  It is grounded in solid scientific research technique, but has definite implications for the practice of the discipline. The applied research studies we wish to recognize will all fulfill a few basic goals. They:

  • Systematically compare plausible alternatives relevant to the development or application of HPT methods (i.e., there must be two or more experimental conditions)
  • Draw conclusions that are generalizable to situations other than the context of the original study
  • Have practical implications and immediate consequences (e.g., a definitive study of the motivational value of spraying whipped cream from an aerosol can on top of  participants’ heads, is a non-starter because in actual practice, only Thiagi could get way with doing it!)
Award evaluators are asked to evaluate award applications in light of these criteria, the research questions being asked, and the appropriateness of the research methodology employed in answering them.

Categories of Research
ISPI wishes to recognize excellence in HPT research without unduly constraining the questions being asked or the methods employed to answer them.  This section describes the major categories of research, which we believe are clearly relevant to advances in HPT.  The intention is not to endorse one method over another, but rather to open the awards process to a variety of research techniques.
 
The description of these categories should make the point that there are many avenues to rigor, generalizability, and significance.  Excellence comes from choosing the right path, and following it thoughtfully.
 
Research in HPT may come in many forms, have more than one purpose, and employ different conceptual models of the research process.  This variety would make our awards procedures unmanageable if we had not established some categories and corresponding criteria.  These categories may appear arbitrary, and their boundaries may be fuzzy (a given entry may well overlap several categories) and the categories themselves are far from exhaustive.  Even so, they summarize years of experience with the Research Award program, and historical patterns of HPT research.
 
In your submission, you will be asked to apply these categories (as we have defined them) to your research project, so that your submission will be that much easier for our evaluators to comprehend.

Applied Research Categories - Purpose
Applied research may be categorized by its purposes, which we list to distinguish research studies from studies that use the same methods of systematic inquiry, but are not research. 
 
The most frequent confusion occurs with evaluation.  Evaluations test the hypothesis that the outcomes are as the design intended (summative evaluation) or that the tactics employed achieve the intended result (formative evaluation).  Rarely do they produce a result that is generalizable to other situations, although the method of evaluation, or its template, may well be of broader use. For the purpose of ISPI Awards, we view evaluations as components of the projects they evaluate, not as independent research projects.
 
The research categories are:
  • Design-oriented Research
  • Outcomes-oriented Research 
  • Technology-oriented Research 
  • Meta-analysis Research 

A brief description of each type of research is included below.

Design- Oriented Research
Design-oriented Research seeks to determine the effects of certain design parameters of HPT interventions, the independent variables, on the dependant variable, human behavior or performance.
 
For example, a researcher may examine the effects of various forms of feedback on task performance to decide how much feedback, and what types of feedback, are necessary to achieve a given level of performance.  Providing that the forms of feedback are adaptable to other contents and subjects, the results will be generalizable to a variety of settings.

Outcomes-Oriented Research

Outcomes-oriented Research seeks to compare the results produced by programs or interventions that have the same nominal goals.  The purpose of this research is to provide an empirical basis for decision-making in selecting the best possible outcome (Geis & Smith, 1992; Rossi & Freeman, 1993).
 
For example, three companies have developed fire safety courses for their physical plant departments.  Now that the companies have merged, the combined company wants to standardize on a single program.  An outcomes-oriented study would gather the information to make that decision.  If those three programs had been purchased from outside vendors and were generally available, or if they were representative of three recognized approaches to teaching fire safety, the research and the decision would be of interest to other companies similarly situated.

Technology- Oriented Research

Technology-oriented Research concerns the impact of the tools and techniques we use to develop and implement HPT interventions.
 
An example would be comparing the same training delivered in a classroom with training delivered in a virtual classroom over the Internet.  To isolate the effects of the technology, all of the design features of the two courses (e.g., exposition, practice, and feedback) would have to be as comparable as possible. Technology may encompass hardware, software, or systematic methods of developing performance improvement solutions. 
 
Technology-oriented research is treated as a separately because its purpose often straddles the boundary between Design-oriented and Outcomes-oriented Research.

Meta-Analysis Research

Meta-Analysis Research combines the results of a number of comparable studies to increase the sensitivity and power of the inferences that can be justified by the data. It is especially useful in areas where a number of ambiguous or non-significant results have been obtained in a collection of experiments run by different people at different times and places.

Applied Research Categories –Measurements

A second method of categorizing research considers the method of measurement used, and the nature of the data collected.  Broadly speaking, measurement in the social sciences can be defined as “the process of linking abstract concepts to empirical indicants” (Carmines & Zeller, 1979).  But, there are many methods of linkage: 

  • Quantitative Research Measures
  • Qualitative Research Measures
  • Statistical Measures

Applied Research Categories – Design
A third method of categorization is based on the designs of the experiments.  Over time, certain designs have become associated with certain types of study; but, in principle these designs are not bound to a particular question or data type. 

Starting with the most rigorous statistical designs, we list the types in order, from those capable of the strongest, most objective inferences to the most subjective ones. Whether a design delivers to its full capability depends largely upon the details of its application, so a powerful design inappropriately chosen, or ineptly implemented, may be significantly worse than a less assuming design that is appropriate to existing experimental constraints.
Be assured that your submission will not be judged by your choice of experimental design in isolation from the context of the entire experimental situation, and the conclusions you draw. Types of Experimental Designs for Applied HPT Research include:

  • Classic balanced randomized block designs
  • Quasi‑experimental designs
  • Single-subject designs
  • Case studies
  • Opinion research
  • Modeling
  • Phenomenological Studies

Research Phases
Research phases follow a common pattern. There are frequent variations, but the basic procedure is:

  1. Identify the question/issue/hypothesis
  2. Plan/design an appropriate study
  3. Collect and code data
  4. Analyze data
  5. Interpret results, draw conclusions
  6. Communicate the results
  7. Use the results to improve human performance theory or practice

We grant that the seventh phase is not always under the control of the researcher.  However, utilization of the results is necessary if the research can be said to be “applied.”  Furthermore, successful application validates the claims of utility to HPT as nothing else can.

Submitting Award

Proposal files:

  • Must be submitted in either Word or PDF; no other formats will be accepted
  • Cannot be password protected

Shall be named as:

  • Contact’s First Initial Last Name Entry Form.xxx (where xxx is .doc or .pdf)
  • Contact’s First Initial Last Name Overview.xxx
  • Contact’s First Initial Last Name Submission Proposal.xxx
  • Contact's First Initial Last Name Supplemental Material.xxx (if applicable) 

Examples:

  • T Gilbert Entry Form.doc
  • T Gilbert Overview.doc
  • T Gilbert Submission Proposal.pdf
  • T Gilbert Appendix.pdf (if applicable)

Submission Qualifications
To qualify, submissions must:

  • Have been completed using the appropriate award submission packet
  • Have addressed all outlined criteria
  • Have been substantially conducted by the individual or organization/team submitting. Students must provide a letter from a faculty member attesting to their primary role in conducting the research
  • Have been completed within the three-year period preceding the nomination deadline and include documentation of this fact
  • Possess the characteristics of excellent research, including these phases:
     -- Identify the questions/issue/hypothesis
     -- Plan/design an appropriate study to answer the question
     -- Collect and code the data
     --  Analyze data
     -- Draw conclusions
     -- Communicate the results
     -- Use the results to improve human performance theory or practice

Qualifications of Submitter
To qualify, a submitter must be willing to:

  • Submit full-length copies of the research and development effort
  • Submit a description and documentation providing evidence of the existence of all relevant criteria related to the systematic approach used to conduct the research
  • Describe and/or provide verifiable evidence of the application of the results or the influence of the research on later research or theory
  • Pay for shipping, special equipment rental, or translation, if needed, for evaluation

Display the actual full-length research (or a mock-up/alternative if confidentiality is a concern), if required, at ISPI's Annual Conference
 
Evaluation Criteria
Each submission will be evaluated for:

  • The internal rigor and quality of the research
  • The extent to which the findings can be generalized to new settings and populations
  • The impact the work has had or will have on the field of Human Performance Technology