Perhaps I missed it, but I don't think these authors looked at differences between applied research and empirical research. I would think that is an important distinction to make when looking at SJT's, because empirical studies may not include a frame of reference whereas in applied studies, the participants may be using their own organization for a frame of reference, sort of like an organizational desirability artifact. I wonder if organizations even think about what construct the SJT is representing? I feel like they use the face validity technique.
If organizations actually determined the constructs that they were measuring in an SJT, might they find it easier to just use a paper-and-pencil assessment of each construct individually? Not that I think this is the right approach or answer, but it seems like most organizations might not define the constructs being measured in an SJT because it is time consuming to develop and determine the constructs.
How useful are the SJTs if the constructs in the applied setting are not accurately defined? This may likely lead to inaccurate selections. I like Shane's point where organizations are likely not interested or aware to be know the construct being measured by SJTs. It was interesting to find all the different constructs SJTs do measure, like leadership.
Although content validity of an SJT can be increased by carefully defining the construct and gathering convergent and divergent validity evidence, I wonder if ecological validity suffers due to narrowing the focus of the measure? For example, if the intended use of a particular SJT is to measure leadership ability, narrowly defining this to one valid measure may create a simulated situation and solution options that are much more restricted than would actully be the case.
If we had a set of critical incidents for managers, how long would it take a group of us to develop dilemmas to test managerial performance domains in SJTs for applicants? Are their instruments already developed that can easily be implemented?
I regret reading this article first because while reading the previous three articles, I could not stop thinking “where the heck is the construct?” I can definitely see why it is difficult to narrow in on the construct when using SJTs. To me, I cannot help but think that SJTs are mostly measuring/intended to measure job experience and tacit knowledge which is why the construct often gets ignored. I pretty much think researchers are operating under the assumption that SJTs are not meant to measure just one construct… What do you think?
Perhaps I missed it, but I don't think these authors looked at differences between applied research and empirical research. I would think that is an important distinction to make when looking at SJT's, because empirical studies may not include a frame of reference whereas in applied studies, the participants may be using their own organization for a frame of reference, sort of like an organizational desirability artifact. I wonder if organizations even think about what construct the SJT is representing? I feel like they use the face validity technique.
ReplyDeleteIf organizations actually determined the constructs that they were measuring in an SJT, might they find it easier to just use a paper-and-pencil assessment of each construct individually? Not that I think this is the right approach or answer, but it seems like most organizations might not define the constructs being measured in an SJT because it is time consuming to develop and determine the constructs.
ReplyDeleteHow useful are the SJTs if the constructs in the applied setting are not accurately defined? This may likely lead to inaccurate selections. I like Shane's point where organizations are likely not interested or aware to be know the construct being measured by SJTs. It was interesting to find all the different constructs SJTs do measure, like leadership.
ReplyDeleteAlthough content validity of an SJT can be increased by carefully defining the construct and gathering convergent and divergent validity evidence, I wonder if ecological validity suffers due to narrowing the focus of the measure? For example, if the intended use of a particular SJT is to measure leadership ability, narrowly defining this to one valid measure may create a simulated situation and solution options that are much more restricted than would actully be the case.
ReplyDeleteIf we had a set of critical incidents for managers, how long would it take a group of us to develop dilemmas to test managerial performance domains in SJTs for applicants? Are their instruments already developed that can easily be implemented?
ReplyDeleteHow might identifying the construct that is being assessed help I/O psychologists develop SJTs for companies?
ReplyDeleteI regret reading this article first because while reading the previous three articles, I could not stop thinking “where the heck is the construct?” I can definitely see why it is difficult to narrow in on the construct when using SJTs. To me, I cannot help but think that SJTs are mostly measuring/intended to measure job experience and tacit knowledge which is why the construct often gets ignored. I pretty much think researchers are operating under the assumption that SJTs are not meant to measure just one construct… What do you think?
ReplyDelete