The unitarian perspective on validation makes a lot of sense to me, especially after examining the complex models of inferences made in selection by Binning and Barrett (1989). In selection in particular, it seems very important to support any inference made with predictor measures. With the difficulty we have in explaining what validity is to employees and organizations, can you think of different ways of explaining the importance of many types of validation, and the principles of hypothesis testing, to an organization?
According to Ployhart, Schneider, and Schmitt (2006, p. 301), "The 1999 version of the "Standards" describes eight types of evidence for validity and the need to integrate this evidence in a meaningful way as the bases on which to support the inferences we draw from test scores." This approach seems more one of amounting a body of evidence in support of the validity of a selection tool than the approach Landy opines about. Would this new perspective on validation have satisfied Landy's concerns about the categorical approach he addressed?
Similar ti Vicki's question, do you think the Unitarian perspective would make things easier to explain to organizations or do you think it is still far to complicated for most people to understand?
Landy 1986 OK – good to read and be exposed to this article that so many later refer to.
It’s starting to get through to me (after the two big theoretical articles – Binning & Barrett and this one) that when we are talking about validity - we need to keep clear in our minds at least that we are talking about inferences (the difference between “test validity and the validity of inferences based on tests” (p.1186)).
Is it helpful to explain to decision-makers that selection is a kind of R&D? (e.g. Selection specialists who do validity analysis are scientists testing the hypothesis: “People who do well on test X will do well on activity Y” (p.1186)
Landy's (1986) criticisms on the work in selection validity mainly focuses on the "Uniform Guidelines" ("UG") revised in 1978 and its lack of discussion about constructs in validation. This really comes clear with the stamp collecting references. However, in line with Shay's comment, I don't think the "UG" really advocated for a recipe-type approach for validation, rather, a set of requirements or situations in which practitioners might use to build support for their selection measure's validity. I'm not sure if it is "stamp collecting vs. science" as Landy suggests but more of "specific guidelines" vs. saying "it depends on multiple job factors". I think the field in 1976 needed to go somewhere with selection validation issues and the "UG" tried to answer that call. It might have been more beneficial for Landy to work on another revision of the "UG" than a completely criticizing it.
I think Landy would be happy with the current view of validity (as I understand it) and would agree with the permeability of different aspects of validity (i.e., construct, content, criterion-related). It seems that talking about validity is not as helpful as discussing the specific evidence and inferences drawn from that evidence. Similarly, I would assume talking about constructs with people in industry might also be like making a steak for a vegan (i.e., pointless). With such confusion about validity in the literature, how do we talk about validity with people who REALLY don't understand validity?
Cool article. I wonder if the status quo, in terms of validation, has warmed up to the notion of a Unitarian approach to validation? Also thought that it was relieving to simply of validation efforts as a well designed study (somewhat a reprieve from all the details that surround the procedure).
This article hones in on the main purpose of validation. I am baffled by the expectations from legal regulations on selection procedures specifically on our professional use of validity. Should Judges be required to understand scientifically the validation process in selection? Or is this already taken into account today? I feel the scientific aspect may be lost in a law suite which I find disturbing. How do we account for this dilemma?
The unitarian perspective on validation makes a lot of sense to me, especially after examining the complex models of inferences made in selection by Binning and Barrett (1989). In selection in particular, it seems very important to support any inference made with predictor measures. With the difficulty we have in explaining what validity is to employees and organizations, can you think of different ways of explaining the importance of many types of validation, and the principles of hypothesis testing, to an organization?
ReplyDeleteAccording to Ployhart, Schneider, and Schmitt (2006, p. 301), "The 1999 version of the "Standards" describes eight types of evidence for validity and the need to integrate this evidence in a meaningful way as the bases on which to support the inferences we draw from test scores." This approach seems more one of amounting a body of evidence in support of the validity of a selection tool than the approach Landy opines about. Would this new perspective on validation have satisfied Landy's concerns about the categorical approach he addressed?
ReplyDeleteSimilar ti Vicki's question, do you think the Unitarian perspective would make things easier to explain to organizations or do you think it is still far to complicated for most people to understand?
ReplyDeleteLandy 1986
ReplyDeleteOK – good to read and be exposed to this article that so many later refer to.
It’s starting to get through to me (after the two big theoretical articles – Binning & Barrett and this one) that when we are talking about validity - we need to keep clear in our minds at least that we are talking about inferences (the difference between “test validity and the validity of inferences based on tests” (p.1186)).
Is it helpful to explain to decision-makers that selection is a kind of R&D? (e.g. Selection specialists who do validity analysis are scientists testing the hypothesis: “People who do well on test X will do well on activity Y” (p.1186)
Landy's (1986) criticisms on the work in selection validity mainly focuses on the "Uniform Guidelines" ("UG") revised in 1978 and its lack of discussion about constructs in validation. This really comes clear with the stamp collecting references. However, in line with Shay's comment, I don't think the "UG" really advocated for a recipe-type approach for validation, rather, a set of requirements or situations in which practitioners might use to build support for their selection measure's validity. I'm not sure if it is "stamp collecting vs. science" as Landy suggests but more of "specific guidelines" vs. saying "it depends on multiple job factors". I think the field in 1976 needed to go somewhere with selection validation issues and the "UG" tried to answer that call. It might have been more beneficial for Landy to work on another revision of the "UG" than a completely criticizing it.
ReplyDeleteI think Landy would be happy with the current view of validity (as I understand it) and would agree with the permeability of different aspects of validity (i.e., construct, content, criterion-related). It seems that talking about validity is not as helpful as discussing the specific evidence and inferences drawn from that evidence. Similarly, I would assume talking about constructs with people in industry might also be like making a steak for a vegan (i.e., pointless). With such confusion about validity in the literature, how do we talk about validity with people who REALLY don't understand validity?
Cool article. I wonder if the status quo, in terms of validation, has warmed up to the notion of a Unitarian approach to validation? Also thought that it was relieving to simply of validation efforts as a well designed study (somewhat a reprieve from all the details that surround the procedure).
ReplyDeleteThis article hones in on the main purpose of validation. I am baffled by the expectations from legal regulations on selection procedures specifically on our professional use of validity. Should Judges be required to understand scientifically the validation process in selection? Or is this already taken into account today? I feel the scientific aspect may be lost in a law suite which I find disturbing. How do we account for this dilemma?
ReplyDelete