There was a long-held assumption that intelligence tests will always result in race differences to the detriment of certain groups such as African Americans and Latinos and in some circumstances women. Siena researchers feel this outcome does not reflect reality and is instead based on traditional ways of measuring intelligence that don’t properly capture the construct and are contaminated by factors that have nothing to do with intelligence. In particular, these traditional tests use items that depend on a test taker’s past experience and education. Siena takes a modern approach to test design that does not use traditional test items to measure intelligence. Siena’s modern approach focuses on having test takers respond to novel, unfamiliar problems so an individual's prior education, work, and life experience are not used to solve these problems. Siena researchers believe that the best leaders and employees are the ones who can solve unique problems; not familiar problems that rely more on book smarts and memorization. By taking this approach, Siena’s tests dramatically reduce race and ethnic differences (as well as the gender differences that sometimes are observed) thus providing a test that is valid and fair when assessing a diverse, global population.
The Siena team are experts and have PhD’s in the field of Industrial and Organizational Psychology (also called I/O Psychology). This specialty area of psychology is characterized by the scientific study of human behavior in organizations and the workplace. I/O psychology focuses on deriving principles of individual, group, and organizational behavior and applying this knowledge to the solution of problems at work. In fact, the field of I/O Psychology is responsible for most of the research and work in measuring intelligence and personality to predict performance in the workplace over the past 4 decades.
Siena Consulting uniquely leverages research from I/O psychology and blends it with the research and findings from other critical fields including Developmental Psychology, Neuropsychology, Cognition, and Social Psychology. This interdisciplinary strategy has led to the development of Siena’s innovative ways to measure intelligence, personality, and motivation.
Across a wide range of sectors and industries, Siena Consulting develops world-class assessment tools that identify talented, diverse candidates for selection and manager & executive development. They develop innovative ways of measuring the intelligence and personality capabilities that drive performance in the workplace. The award-winning Siena Reasoning Test (SRT) has been developed as the optimal way to measure the multi-dimensional elements of intelligence as used in a work environment. The SRT can be customized to best fit those elements of intelligence that are situationally more relevant to a wide range of jobs, levels, and organizations. Versions of the SRT are used today in Fortune 500 companies and a wide variety of government agencies.
The Player Assessment Test (PAT) is a customized test of intelligence, personality, and motivation that has been administered to all participants at the NFL Combine since 2013. Results from this Player Assessment Test (PAT) are distributed to all NFL clubs for use in player selection and/or coaching individual athletes. Alternative tailored versions of the PAT are used in other professional sports as well as with college athletes.
The short answer is “it depends”; but our tests range from anywhere from 15 to 60 minutes. Ultimately, different businesses and sports teams choose different options customized to their existing evaluation process and the breadth and depth of the insights they seek.
Siena’s tests can be administered on a range of devices including computers, tablets, and cell phones. Typically, test results are available immediately for use by the organization.
We agree that different sports require different contributions from athletes. Fluid sports, like basketball and soccer, allow less time for coaching interaction and play calling to optimize a team’s chance for offensive or defensive success vs. less fluid sports like football and baseball.
Fortunately, the simple answer is (for the most common sports like basketball, soccer, baseball, and football) there is a core set of dimensions – that is a great universal model and a great starting point. However, within each of these dimensions, we have “tweaked” scoring and evaluation based on judgment and prioritization. For example, you might choose to measure aspects of intelligence for sports like Basketball and Soccer (where decision making is on-the-fly and there are fewer coaching interventions) with a greater emphasis on visual processing and the ability to think ahead as play evolves. In each case, we fully evaluate the requirements and desired culture of an organization to best meet the needs of that organization.
The short is “it depends”; but the range is anywhere from 15 minutes up to an hour. Ultimately, different teams choose different options customized to how it best fits into their existing evaluation process and the breadth and depth of the insights they seek. Here are some of the biggest variables:
The scores provided are based on the percentile achieved by the individual. The percentile is calculated by comparing the individual to the sample of athletes who have taken the test to date.
Scores range from low (1) to high (10) based on the percentile achieved. The scores are generated based on the responses of the individual relative to the relevant sample of athletes who have taken the test to date. Scores of 1 to 3 are considered low, scores of 4 to 7 are considered moderate, and scores of 8 to 10 are considered high.
An overall score for each athlete is provided, as well as summary scores for Intelligence and Personality. As described above, these scores are all on a 10-point scale with 10 being the highest and one the lowest. Each dimension also has a 1-10 score. For each 1-10 score, it compares the individual athlete relative to the sample of athletes who have taken the test.
To get a complete picture of the athlete, it is recommended that the information provided by the PAT be used in combination with other data that teams collect on players via other sources – like the combine and scouting operations. The information from this assessment should be seen as an additional indicator of the potential capabilities of a player. Using this information in combination with other valid indicators can help a team gain insight into a player’s capabilities and how that individual could perform in their specific sport (like the NFL).
Long standing intelligence and personality tests like the Wonderlic and the Myers Briggs have rich data that goes back many years and across many cultures. They can share a wide range of benchmarking data to allow for comparing and contrasting and have large enough samples to examine subsets of the populations vs. new test takers. Most of this work has been outside of sports but is more broadly familiar and understood by mainstream society and especially in US business.
Newer tests (like the PAT) use different ways to measure and assess some of the same attributes (arguably) with more precision and (inarguably) with less bias. Because of new ways of asking questions and new methods for item development, the PAT has the advantage of removing bias as well as being more closely focused on sports-related behaviors and competencies.This is especially critical with measuring intelligence which has been a challenge across several decades with intelligence testing. It also has the benefit of more data and statistical evidence for being highly correlated to performance and outcome measures in sports.
Based on our experience, we believe that when you add the PAT to player evaluation, the assessment becomes the best friend of the scout or recruiter. We don’t recommend the PAT be the sole solution. But it does provide an infrastructure for scouts and recruiters to compare across candidates. In our experience, a single scout or recruiter does not have the bandwidth or expert psychological assessment capabilities to dive deep into each individual player – they instead need to compare notes across scouts and across coaches. The PAT allows them to do this. And the PAT provide the predictive variables whereas scouts and recruiters need to connect these assessments to output data – like on-field performance, leadership behaviors, team behaviors, and classroom performance. The PAT also directs conversations towards what parts of a player's character needs to be most closely examined and thus focuses conversation to decisions can be made based on facts.
The PAT is not a magic wand. But it does provide material information to explain player on-the-field and off-the-field performance. It allows teams to understand if one-off incidents are red flags or if they are just that; one-off exceptions.
Based on interviews with coaches and former players, we believe tests like the PAT are becoming more important. To quote Teddy Bruschi, “Your players are only as good as their ability to execute your game plan.” Most would argue that sports strategy and game plans are becoming more and more complicated. If you agree with this, the PAT gives teams a running start into fully evaluating the individual beyond their athletic ability.
The other factor that has emerged in interviews is being a good player “off-the-field”. The emergence of social media has amplified mistakes being made by young adults. In college, it’s going to class, staying eligible, and staying out of negative social media. Professional athletes need to worry about social media and also carry the added burden of the media and the paparazzi waiting to catch them with an ill-advised quote or a regretted tweet.
In either or both cases, getting a comprehensive picture of an individual is predictive of outcome variables both on and off-the-field; above and beyond athletic ability. You can watch more about how the PAT predicts beyond athletic ability when Ken presents on this topic at the MIT Sports Analytics Conference.