The principle of validity generalization rests on the idea of a minimally acceptable value of validity. This reminds me of the idea that statistical significance only exists if p < .05, such that anything p > .05 is not “significant.” Setting a minimally acceptable value or cutoff for validity to me seems to fall into the same problems we now deal with in the arbitrary determination of statistical significance such that anything below this lower bound or cutoff (e.g. .10, or whatever is chosen) is “not valid.” Who sets the minimally acceptable value for validity? The legal system? Researchers? And what is the rationale behind the selection of such a value?
The legal system may prefer the cutoffs because they are simple, black & white, and not dependent on various factors (e.g., situational). The decision rule of 75% seems to be not a good fit in legal sense b/c conservativeness is valued more highly. Is there likely to be changes to the decision rule and maybe a joining of situational factors in the VG procedure? How would this be reflected in a legal sense?
Is the moral of the VG analysis study: Know thy variance? In other words, validity generalization is only really as good as the studies it samples and their relevance to the current application of the test. It seems to me, that like anything else in statistical analysis, indiscriminate application of any analysis will not yield meaningful results.
In response to Vicki's question, do you think it would be helpful to have a sort of confidence interval for a test being generalizable? Or do you think this would be too much of a gray area for courts and organizations?
The appeal of validity generalization is strong... that we not have to validate a test in every situation, that tests (like cognitive tests) can be valid across situations. Is the thinking now (a-la Murphy 2000) that this still may be true, but more testing is needed considering a strong situational variable of restrictiveness of climate? Are there other meaningful sources of difference between organizations that might also be included in future research? (e.g. for-profit/non)
It seems like some of these readings are missing a huge piece of the puzzle. Before we assume that measures generalize across certain jobs, shouldn't we have a good explanation as to why we would think that the measure would generalize? I am thinking this would might be better at the more general level. However, I would think organizations would want a customized selection process that DOES take into account the organization's specific situational context.
Perhaps, a discussion of the benefits of a proper selection system is the first step to designing one. That is, even if I put in the most state-of-the-art selection system using multiple valid predictors and taking into account the situational factors of the specific organization, but we only see a 4-5% rise in production. Is it even worth it to go through the effort? It seems part of the challenge for I/O specialists is to realistically balance between what can be accomplished and what is financially feasible to accomplish. Perhaps part of the gap between scientist and practitioner is that the latter takes the cost/benefit consideration more seriously (perhaps they are more realistic about what can be accomplished by a selection system) and the former is more concerned with ideal measurements and 'experimental' control. What can we realistically expect from a selection system?
VG is clearly a powerful tool, but it’s subject to so many problems (e.g., the file drawer problem). What is your personal comfort level making recommendations with this tool?
The principle of validity generalization rests on the idea of a minimally acceptable value of validity. This reminds me of the idea that statistical significance only exists if p < .05, such that anything p > .05 is not “significant.” Setting a minimally acceptable value or cutoff for validity to me seems to fall into the same problems we now deal with in the arbitrary determination of statistical significance such that anything below this lower bound or cutoff (e.g. .10, or whatever is chosen) is “not valid.” Who sets the minimally acceptable value for validity? The legal system? Researchers? And what is the rationale behind the selection of such a value?
ReplyDeleteThe legal system may prefer the cutoffs because they are simple, black & white, and not dependent on various factors (e.g., situational). The decision rule of 75% seems to be not a good fit in legal sense b/c conservativeness is valued more highly. Is there likely to be changes to the decision rule and maybe a joining of situational factors in the VG procedure? How would this be reflected in a legal sense?
ReplyDeleteIs the moral of the VG analysis study: Know thy variance? In other words, validity generalization is only really as good as the studies it samples and their relevance to the current application of the test. It seems to me, that like anything else in statistical analysis, indiscriminate application of any analysis will not yield meaningful results.
ReplyDeleteIn response to Vicki's question, do you think it would be helpful to have a sort of confidence interval for a test being generalizable? Or do you think this would be too much of a gray area for courts and organizations?
ReplyDeleteThe appeal of validity generalization is strong... that we not have to validate a test in every situation, that tests (like cognitive tests) can be valid across situations. Is the thinking now (a-la Murphy 2000) that this still may be true, but more testing is needed considering a strong situational variable of restrictiveness of climate? Are there other meaningful sources of difference between organizations that might also be included in future research? (e.g. for-profit/non)
ReplyDeleteIt seems like some of these readings are missing a huge piece of the puzzle. Before we assume that measures generalize across certain jobs, shouldn't we have a good explanation as to why we would think that the measure would generalize? I am thinking this would might be better at the more general level. However, I would think organizations would want a customized selection process that DOES take into account the organization's specific situational context.
ReplyDeleteRepost from my James (1986) comment:
ReplyDeletePerhaps, a discussion of the benefits of a proper selection system is the first step to designing one. That is, even if I put in the most state-of-the-art selection system using multiple valid predictors and taking into account the situational factors of the specific organization, but we only see a 4-5% rise in production. Is it even worth it to go through the effort? It seems part of the challenge for I/O specialists is to realistically balance between what can be accomplished and what is financially feasible to accomplish. Perhaps part of the gap between scientist and practitioner is that the latter takes the cost/benefit consideration more seriously (perhaps they are more realistic about what can be accomplished by a selection system) and the former is more concerned with ideal measurements and 'experimental' control. What can we realistically expect from a selection system?
VG is clearly a powerful tool, but it’s subject to so many problems (e.g., the file drawer problem). What is your personal comfort level making recommendations with this tool?
ReplyDelete