It seems crucial to have specific content domains in mind when administering an SJT for scoring purposes, although it seems tempting to include a whole spectrum of situations. Keeping in mind the argument from last class about not adding constructs we already have predictors for (i.e., cognitive testing), what would be the best content to test with SJTs?
I think the selection of a scoring key would be the most challenging aspect of developing an SJT. Have any SJTs been subjected to legal scrutiny for adverse impact claims? How do these tests stand up against cognitive ability, personality, etc. in terms of discrimination?
I was thinking the same thing as Vicki on SJT's comparison to cognitive ability tests regarding discrimination and adverse impact? I would think that since SJTs are more specific to the job or a construct that they may have less discrimination within the test.
What a clearly written article and an important, well, described study. This helped me understand what goes into a well-developed SJT reflecting clear content domains. I learned: about different kinds of keying for SJT; that the validity of a particular SJT depends in part on its scoring; the importance of validating the keys and having them meet standards of high criterion validity, incremental validity, minimizing subgroup differences, and construct validity. It helped me appreciate the cost and complexity that can go into developing an SJT e.g that is computer delivered; and the complexity of scoring when consensus has not been reached (e.g. in a theory).
For Roni, anyway you could give an example of N-fold cross validation? A little confused by the description given in the reading. Also, not clear why this creates a scoring method “that does not capitalize on chance.”
It seems crucial to have specific content domains in mind when administering an SJT for scoring purposes, although it seems tempting to include a whole spectrum of situations. Keeping in mind the argument from last class about not adding constructs we already have predictors for (i.e., cognitive testing), what would be the best content to test with SJTs?
ReplyDeleteI think the selection of a scoring key would be the most challenging aspect of developing an SJT. Have any SJTs been subjected to legal scrutiny for adverse impact claims? How do these tests stand up against cognitive ability, personality, etc. in terms of discrimination?
ReplyDeleteI was thinking the same thing as Vicki on SJT's comparison to cognitive ability tests regarding discrimination and adverse impact? I would think that since SJTs are more specific to the job or a construct that they may have less discrimination within the test.
ReplyDeleteWhat a clearly written article and an important, well, described study. This helped me understand what goes into a well-developed SJT reflecting clear content domains. I learned: about different kinds
ReplyDeleteof keying for SJT; that the validity of a particular SJT depends in part on its scoring; the importance of validating the keys and having them meet standards of high criterion validity, incremental validity, minimizing subgroup differences, and construct validity. It helped me appreciate the cost and complexity that can go into developing an SJT e.g that is computer delivered; and the complexity of scoring when consensus has not been reached (e.g. in a theory).
Do you think I/O psychologists need to determine what the best scoring method is each time they administer an SJT?
ReplyDeleteWhat type of scoring do you think organizations would endorse the most?
For Roni, anyway you could give an example of N-fold cross validation? A little confused by the description given in the reading. Also, not clear why this creates a scoring method “that does not capitalize on chance.”
ReplyDelete