HOW TO: designing case studies best suited for tracking cognitive processes - part 1
Updated: Feb 4, 2022
Unlike traditional aptitude analysis tools cassudy uses case studies in combination with a process tracking methodology to determine personality traits and cognitive abilities of a test subject. The approach intuitively seems straightforward and plausible. But in order to generate meaningful and reliable results a lot of design criteria for such case studies have to be considered and respected accordingly. This blog and its successor (part 2) will describe these design criteria and provide evidence of their importance.
The big picture set up
A cassudy case study deals with business scenarios that most likely will be relevant for (or even part of) the day job of the test subject that is processing the case study.
To name a few business scenarios cassudy offers (taken from the list of cassudy´s pre-configured case studies):
1. a strategic buyer cleaning up his supplier portfolio based on the strategic guidance of his company
2. a warehouse manager selecting one out of several improvement projects to automate operation
3. a recruiter or hiring manager selecting one out of several candidates for a given role
4. a manager dealing with several (conflicting) leadership guidelines at the same time
Looking at these examples it becomes obvious that each case study relates to a specific role (which should be close to the role of the test subject) and a scenario, that is realistic and ambiguous. Ambiguity as a characteristics of the case study is extremely important. If the scenario is to choose between several alternatives, the „best“ solution should not be that obvious. In the example of the portfolio cleansing mentioned above, the different suppliers should have different strength and weaknesses. The decision taken by the person doing the test will then deliver insights into the preference profile of the testee.
To summarize – a typical cassudy case study relates to a specific role, to a specific (realistics) business scenario and there is no obvious „best outcome“.
The mental strain for the test subject
Looking at the typical use cases for cassudy case studies (screening applicants for a job posting, identifying areas of development for managers, etc.) the processing time of the case studies should not exceed 30 minutes. Within this time limit, the mental strain should be kept on a high to very high level. That´s why cassudy case studies follow a specific pattern. They start with a description of the strategic context of the company followed by the description of the case scenario itself. Afterwards the test candidates are confronted with different alternatives to choose from. To understand and evaluate the alternatives supporting information is provided that needs to be explicitly accessed. Finally the test subject has to take a decision and should also provide evidence for it.
This approach keeps the mental strain high. The test subject will have to either remember a lot of details or needs to check the information several times. In parallel the test subject has to keep track of the remaining processing time. And as the alternatives show different strengths and weaknesses, the test person has to come to intermediate, consolidated judgements regarding the different alternatives, which in turn will be used for the final decision or judgement.
Mental strain and the description of the alternatives to judge on
With the above in mind I would like to mention a few more detaiIs on how alternatives should be described best. Each alternative (also often called option) in a given case study is described using the same set of criteria. (Example: if the task is to choose a business lunch from a set of three different options, each option would be described using the same set of nutrition information (fat, sugar, calories, etc., maybe a special hint from the chef and whether the ingredients are from regional suppliers or not). So the test subject would look at the criteria that interest him/her most for the option under consideration, come to an intermediate overall assessment for this option, repeat this process for all options, and then finally decide which lunch menu to take.
If each option has more than 15 criteria that characterize it, the test subject will not be able to memorize all of them and will look at those criteria that are of higher interest to him/her, a second and maybe third time. Such criteria will than have higher attention than the other ones. The attention distribution is one of the key data sets that enter into the final evaluation of the test subject.
The impact of empirical knowledge of the test candidate on the design of the case study
A cassudy case study is designed to track the mental steps the test subject is going through when processing the case study. To do this, these steps have to become somehow „visible“. A good indication for the steps of the mental process is the information acquisition path the test subject chooses which in turn is tracked by cassudy.
Any mental or cognitive processes that are based on the test subject´s empirical knowledge cannot be observed directly by cassudy. Therefore cassudy case studies are designed to avoid the use of empirical knowledge. The food choice example as shown above would therefore need a special design that „forces“ the test subject through an information acquisition process rather than using empirical knowledge. The demo example we use to demonstrate the basic methodology of cassudy uses exactly such a design.
To summarize – a typical cassudy case study consists of a set of general descriptions and a set of options to choose from. Each option is described by the same set of criteria with different instances of these criteria. Each option should be sufficiently complex (consist of >15 criteria) so that the mental strain level is kept high during processing the case study. The use of empirical knowledge of the test subject has to be avoided as best as possible.
In part 2 of the design instructions we will go deeper into the layout of the options, the workflow within the case study and how the criteria relate to the test subject´s traits to be analyzed. All these aspects are important to achieve a high level of validity of the evaluation results.
Overwhelmed?
This blog (consisting of part 1 and 2) is meant to give you an idea of a „good case study design“. In fact there are a lot more details to be considered and traps to be avoided. The cassudy team has more than a decade of experience in developing these kind of case studies. You can be sure, we also ran into many traps and dead ends during the development process. So if you feel overwhelmed at first don´t give up - to discuss the scenarios or issues you have in mind!