The company described conducted multiple validation studies on jobs with a large number of incumbents. The authors describe the extensive job analysis process that the company underwent in order to construct the performance criterion. This was obviously very costly and time intensive. In today's rapidly evolving economic situation, many jobs are in a continual state of flux. How does this impact validation studies when one piece of the validation process is dynamic rather than static? Are the validation studies fairly rapidly obsolete?
In line with Shay's comment this project was a massive undertaking. What are some of the benefits of doing an entire organization's job analyses at once? What are some some of the drawbacks?
My question was similar to Shay's question. This article demonstrates the usefulness of synthetic validation. Yet, given the nature of change within individual jobs over time, it is really beneficial to an organization to dedicate so much time to determining job families and test batteries when it is very plausible that the jobs in these families will change enough over time and no longer appropriately fit into these categories? Although validation is necessary, is the amount of work required to continue validation worth it for an organization?
Although change happens, (especially with technological innovations), it seems the 11 job families identified in the study could be fairly stable (see p. 979) and many of the categories of tests used (e.g. computation, FIT assembly, space visualization, coding, language skills) could still be updated and used to draw valid inferences at a later time. What do others think?
Roni Qs: This gives me a sense of the magnitude of this kind of study. How much time (and $) would a study like this take? What sorts of companies do this kind of thorough work? Is the Position Analysis Questionaire (PAQ) still considered the best-researched worker-oriented job analysis method?
Maybe I do not understand their arguments against the Uniform Guidelines of 1978, but aren't Hoffman, Holden, & Gale (2000), Landy (1986), and Binning & Barrett (1989) simply adding more “boxes” to the Guidelines? It is not like these authors would argue against the fact that we shouldn’t be concerned with criterion-related validity or construct validity. They are simply taking a more modern view of validity, in that one should use multiple sources of evidence to have a defensible validity argument for selection purposes. I do like to see the concept of synthetic validation being applied in an actual organization. This really gave a great example of what Johnson and Carter (2010) were talking about and covered many job families. Does it matter that they are all in the same organization? Could their validation extend to similar job families in other organizations?
How does the economy influence the selection process and validation steps? Are companies more reluctant to restrict funding to selection processes in a bad economy like today? This would see to affect the possibility to conduct various validity measures within the selection process. Also, organizational change due to uncontrollable factors (e.g., economy), may change the demand, organizational goal, and job tasks in that this type of large validation and selection process may not hold accurate predictions/inferences as once did.
The company described conducted multiple validation studies on jobs with a large number of incumbents. The authors describe the extensive job analysis process that the company underwent in order to construct the performance criterion. This was obviously very costly and time intensive. In today's rapidly evolving economic situation, many jobs are in a continual state of flux. How does this impact validation studies when one piece of the validation process is dynamic rather than static? Are the validation studies fairly rapidly obsolete?
ReplyDeleteIn line with Shay's comment this project was a massive undertaking. What are some of the benefits of doing an entire organization's job analyses at once? What are some some of the drawbacks?
ReplyDeleteMy question was similar to Shay's question.
ReplyDeleteThis article demonstrates the usefulness of synthetic validation. Yet, given the nature of change within individual jobs over time, it is really beneficial to an organization to dedicate so much time to determining job families and test batteries when it is very plausible that the jobs in these families will change enough over time and no longer appropriately fit into these categories? Although validation is necessary, is the amount of work required to continue validation worth it for an organization?
Although change happens, (especially with technological innovations), it seems the 11 job families identified in the study could be fairly stable (see p. 979) and many of the categories of tests used (e.g. computation, FIT assembly, space visualization, coding, language skills) could still be updated and used to draw valid inferences at a later time.
ReplyDeleteWhat do others think?
Roni Qs: This gives me a sense of the magnitude of this kind of study. How much time (and $) would a study like this take? What sorts of companies do this kind of thorough work?
Is the Position Analysis Questionaire (PAQ) still considered the best-researched worker-oriented job analysis method?
The authors focused cognitive-ability predictors. How do you think such a validation procedure would apply to the use of personality-based predictors?
ReplyDeleteMaybe I do not understand their arguments against the Uniform Guidelines of 1978, but aren't Hoffman, Holden, & Gale (2000), Landy (1986), and Binning & Barrett (1989) simply adding more “boxes” to the Guidelines? It is not like these authors would argue against the fact that we shouldn’t be concerned with criterion-related validity or construct validity. They are simply taking a more modern view of validity, in that one should use multiple sources of evidence to have a defensible validity argument for selection purposes.
ReplyDeleteI do like to see the concept of synthetic validation being applied in an actual organization. This really gave a great example of what Johnson and Carter (2010) were talking about and covered many job families. Does it matter that they are all in the same organization? Could their validation extend to similar job families in other organizations?
How does the economy influence the selection process and validation steps? Are companies more reluctant to restrict funding to selection processes in a bad economy like today? This would see to affect the possibility to conduct various validity measures within the selection process.
ReplyDeleteAlso, organizational change due to uncontrollable factors (e.g., economy), may change the demand, organizational goal, and job tasks in that this type of large validation and selection process may not hold accurate predictions/inferences as once did.