Procedures

A procedure defines the manner in which Trial templates are presented in a study. Both participant grouping and conditional branching are specified in the procedure.

Basic form

The procedure of a study is defined in the following form:

{
  "type": "NAME_OF_PROCEDURE",
  "PROCEDURE_SPECIFIC_PROPERTY1": SOME_VALUE
  "PROCEDURE_SPECIFIC_PROPERTY2": SOME_VALUE
  ...
}

For example, we can define a simple blocking procedure with the following code:

{
  "type": "blocking",
  "blocks": {
        "BL1": {
          "trial_templates": ["TSA", "TSB"],
          "pattern": {"order": "random", "repeat": 2},
          "end_trials": ["BlockEnd"]
        },
        "BL2": {
          "trial_templates": ["TS2", "TS3", "ATS1", "CommentTS"],
          "pattern": {"order": "fixed", "repeat": 1},
          "catch_trial": {"template": "CatchTS", "frequency": 3}
        }
  },
  "block_sequence": ["BL1", "BL2"]
}

Here, we use the procedure type "blocking", with two blocks named "BL1" and "BL2". In "BL1", two trial templates will be presented, in a random order so that realized trials of TSA and TSB are randomly mixed together. In addition, each of the realizable trials of those two templates will be presented twice (the effect of "repeat": 2). Finally, the "BL1" block ends with a trial template BlockEnd, which can be used to display some instructional text.

The properties in "BL2" have similar interpretations. A main difference in "BL2" is that the trial templates will be presented in the order listed under the trial_templates property (in this example, ["TS2", "TS3", "ATS1", "CommentTS"]). In addition, participants will experience a “catch trial” (most likely unknown to them) drawn from the trial template CatchTS every 3 normal trials (that is, for every 4 trials, the last trial is a catch trial).

Note

Unlike other top-level properties, procedures do not have names, since they don’t need to be referenced in other parts of the specification

The “blocking” Procedure

The most common procedure in which trials are organized into one or more blocks. To use blocking, specify "blocking" for the "type" property of the top-level property "procedure":

{
   "type": "blocking",
}

Procedure-specific properties

blocks

A dictionary of property-value pairs, each corresponding to the definition of a block. The order in which block definitions are listed here is inconsequential. The presentation order of individual blocks during a study is determined by block_sequence instead.

block_sequence

A list of items that define the order in which individual blocks are presented in a study, possibly varying between participants on a fixed (i.e., participant groups) and/or dynamic (i.e., conditional branching) basis.

The items in block_sequence can be (1) block names, (2) participant group dictionaries, (3) conditional branch dictionaries, or (4) dedicated keyword instructions. These four types of items can be combined (and in some cases, nested within one another) to achieve complex between-subject manipulation:

  • All participants experience the same fixed-order block sequence: When the "block_sequence" is a list of block names, all participants experience the same list of blocks. There is no between-subject manipulation. For example, the following procedure definition presents "block_2" before "block_1" for all participants.

    {
        "type": "blocking",
        "blocks": {
           "block_1": {
               // block_1 definition here
           },
           "block_2": {
               // block_2 definition here
           }
        },
        "block_sequence": ["block_2", "block_1"]
    }
    
  • All participants experience the same blocks, but the presentation order of those blocks is randomized per participant: FindingFive will automatically randomize the presentation order of a sequence of blocks if they appear in a dictionary containing the "$randomize" keyword.

    For example, if we define the procedure as:

    {
        "type": "blocking",
        "blocks": {
            "block_1": {
                // block_1 definition here
            },
            "block_2": {
                // block_2 definition here
            },
            "block_3": {
                // block_3 definition here
            }
         },
         "block_sequence": [{"$randomize": ["block_1", "block_2", "block_3"]}]
     }
    

    During a session of this study, each participant will be presented with a randomized ordering of 3 blocks (e.g., "block_2", "block_1", "block_3").

    It is also possible to randomize only part of the complete block sequence per participant:

    {
        "type": "blocking",
        "blocks": {
            // block definitions here
        },
        "block_sequence": ["block_1", {"$randomize": ["block_2", "block_3"]}]
    }
    

    During a session of this study, each participant will be presented with first "block_1", then the remaining two blocks in random order.

    Note

    The order of blocks contained within the "$randomize" dictionary will always be truly random, making it possible some of the randomizations will never be presented to any participants. This can be avoided using the "$factorialize" keyword (see below).

  • Counter-balanced participant groups, each with a different block sequence: FindingFive will automatically create counter-balanced participant groups, if the "block_sequence" consists of dictionaries that meet the following criteria: (1) the keys of that dictionary are user-defined group names (i.e., they are not dedicated keywords, such as "$randomize" or "$factorialize", and they are not branch names, see below); and (2) the values of that dictionary are block names. Participants of each group will experience their own sequence of blocks as specified by the researcher.

    For example, if we define the procedure as:

    {
        "type": "blocking",
        "blocks": {
           "block_1": {
               // block_1 definition here
           },
           "block_2": {
               // block_2 definition here
           },
           "block_3": {
               // block_3 definition here
           },
           "block_4": {
               // block_4 definition here
           }
        },
        "block_sequence": [{"GroupA": ["block_1", "block_2", "block_4"],
                            "GroupB": ["block_1", "block_3", "block_4"]
                           }]
    }
    

    During a session of this study, half of all participants will be assigned to “GroupA”, who will be presented "block_1", "block_2", and then "block_4". The other half of participants will be assigned to “GroupB”, who will be presented "block_1", "block_3", and then "block_4". The assignment of participants to different groups is automatically handled by FindingFive. Researchers only need to make sure that the number of participants requested for a session is a multiple of the number of participant groups defined here.

    It is possible to simplify the above definition by putting the common blocks outside the participant group dictionary:

    {
        "type": "blocking",
        "blocks": {
            // block definitions here
        },
        "block_sequence": ["block_1",
                           {"GroupA": ["block_2"],
                            "GroupB": ["block_3"]},
                           "block_4"]
    }
    

    This has the exact same effect as the code above, but highlights the between-subject manipulation. In general, blocks outside participant group dictionaries will be presented to all participants, and each participant group will experience their own sequence of blocks as specified in the dictionaries. Researchers only need to define participant groups for the part of a study that actually differs among groups.

    It is also possible to apply the "$randomize" keyword to a group-specific block sequence. For example:

    {
        "type": "blocking",
        "blocks": {
            // block definitions here
        },
        "block_sequence": ["block_1",
                           {"GroupA": ["block_2", "block_3"],
                            "GroupB": {"$randomize": ["block_2", "block_3"]}},
                           "block_4"]
    }
    

    In this case, the between-subject manipulation is whether "block_2" and "block_3" are presented in a fixed order ("GroupA") or a random order ("GroupB").

  • Create an exhaustive set of counter-balanced participant groups corresponding to the permutations of a block sequence: If you would like to present participants with blocks in a random order, but wish to assign an equal number of participants to each block sequence order, this can be achieved using the "$factorialize" keyword.

    For example, if we define the procedure as:

    {
        "type": "blocking",
        "blocks": {
            // block definitions here
        },
        "block_sequence": [{"$factorialize": {
                               "my_group": ["block_1", "block_2", "block_3"]
                           }}]
    }
    

    FindingFive will automatically create a group for each of the 3! = 6 permutations of the blocks defined after "my_group". The groups will be named "my_group1", "my_group2""my_group6", where the user-defined base group name (in this case "my_group") is appended with a number corresponding to which permutation of the block sequence the group received.

    Note

    The algorithm used to assign the permutations into groups always uses the same ordering to assist in identifying which group recieved which order. In our example above, "my_group1" will always get the order ["block_1", "block_2", "block_3"]; "my_group2" will always get the order ["block_1", "block_3", "block_2"]; "my_group3" will always get the order ["block_2", "block_1", "block_3"]; and so on.

    As with participant grouping and the "$randomize" keyword, the "$factorialize" keyword can be applied to a subset of the block sequence:

    {
        "type": "blocking",
        "blocks": {
            //block definitions here
        },
        "block_sequence": ["block_1",
                           {"$factorialize": {
                               "my_group": ["block_2", "block_3"]
                            }}]
    }
    

    This setup is equivalent to the following, where the participant groups are manually defined:

    {
        "type": "blocking",
        "blocks": {
            //block definitions here
        },
        "block_sequence": ["block_1",
                           {"my_group1": ["block_2", "block_3"],
                            "my_group2": ["block_3", "block_2"]}]
    }
    
  • Participants dynamically assigned to different branches based on the same evaluation: FindingFive implements conditional branching when the block_sequence contains both a branching block and then a branch dictionary. A dictionary is interpreted as a branch dictionary (as opposed to a participant group dictionary) if the keys of that dictionary have already been declared as conditional branches in a preceding branching block.

    For example, the following procedure defines a branching action:

    {
        "type": "blocking",
        "blocks": {
           "block_1": {
               // block_1 definition here
           },
           "block_2": {
               // block_2 definition here
           },
           "branching_block": {
               // we only show the branching instructions here for illustration purposes
               // one still needs to define all the other components of a block
               "branching": {
                   "method": "accuracy",
                   "triggers": [{"trial_template": "TRIAL_TEMPLATE_NAME",
                                "response": "RESPONSE_NAME"}],
                   "min_score": 0.8,
                   "branches": {"A": true, "B": false}
               }
           }
        },
        "block_sequence": ["branching_block", {"A": ["block_2"], "B": ["block_1"]}]
    }
    

    Given this setup, participants who achieved a minimum accuracy of 80% on trials from the template "TRIAL_TEMPLATE_NAME" that contain the response "RESPONSE_NAME" within the "branching_block" will see "block_2", while the rest of the participants will see "block_1". The assignment of participants into these branches are conditioned upon their online performance in a preceding block. Thus the name conditional branching.

    Note

    In the example above, the branching block that declares Branches "A" and "B" must be placed before the actual condition branch dictionary. Otherwise, when the study progresses to the conditional branch dictionary, it is impossible to know how to assign participants into branches because they haven’t taken the branching test in a corresponding branching block yet.

    Intervening blocks can be inserted between a branching block and its corresponding branch dictionaries. For example, the "block_sequence" in the above example can be:

    {
        "block_sequence": ["branching_block", "block_3",
                           {"A": ["block_2"], "B": ["block_1"]}]
    }
    

    In this case, participants are assigned to either Branch A or Branch B at the end of "branching_block". However, this assignment has no effect on the presentation of "block_3" - all participants will see it. The effect of conditional branching takes place only when a branch dictionary is encountered in the "block_sequence".

  • Counter-balanced participant groups branches differently based on the same branching instruction: It is possible to combine participant grouping and conditional branching so that all participant groups are given identical branching evaluation with the same branching block, but different groups will see different outcomes of the branching. This is achieved by putting the branching block outside participant group dictionaries, and the branch dictionaries within participant group dictionaries:

    {
        "type": "blocking",
        "blocks": {
           "block_1": {
               // block_1 definition here
           },
           "block_2": {
               // block_2 definition here
           },
           "branching_block": {
               // we only show the branching instructions here for illustration purposes
               // one still needs to define all the other components of a block
               "branching": {
                   "method": "accuracy",
                   "triggers": [{"trial_template": "TRIAL_TEMPLATE_NAME",
                                "response": "RESPONSE_NAME"}],
                   "min_score": 0.8,
                   "branches": {"A": true, "B": false}
               }
           }
        },
        "block_sequence": ["branching_block",
                           // notice how branch dictionaries are nested within the participant group dictionary
                           {"groupA": {"A": ["block_2"], "B": ["block_1"]},
                            "groupB": {"A": ["block_1"], "B": ["block_2"]}
                           }
                          ]
    }
    

    In this case, FindingFive will automatically assign participants to either "groupA" or "groupB" at the beginning of a study. This is how participant grouping works. Both groups will then experience the same branching block. Participants of both groups will get assigned to either Branch A or Branch B depending on their responses in the branching block. Crucially, participants of "groupA" will see "block_2" if they get assigned to Branch A, whereas participants of "groupB" who are assigned to Branch A will see "block_1". In other words, participants of different groups can branch differently.

  • Counter-balanced participant groups with different branching instructions: It is also possible to combine participant grouping and conditional branching so that different participant groups receive different branching evaluation. This is achieved by putting different branching blocks inside different participant group dictionaries:

    {
        "type": "blocking",
        "blocks": {
           "block_1": {
               // block_1 definition here
           },
           "block_2": {
               // block_2 definition here
           },
           "branching_block": {
               // we only show the branching instructions here for illustration purposes
               // one still needs to define all the other components of a block
               "branching": {
                   "method": "accuracy",
                   "triggers": [{"trial_template": "TRIAL_TEMPLATE_NAME",
                                "response": "RESPONSE_NAME"}],
                   "min_score": 0.8,
                   "branches": {"A": true, "B": false}
               }
           }
           "branching_block2": {
               // we only show the branching instructions here for illustration purposes
               // one still needs to define all the other components of a block
               "branching": {
                   "method": "accuracy",
                   "triggers": [{"trial_template": "TRIAL_TEMPLATE_NAME",
                                "response": "RESPONSE_NAME"}],
                   "min_score": 0.50001,
                   "branches": {"A": true, "B": false}
               }
           }
    
        },
        "block_sequence": [
                           {"groupA": ["branching_block",
                                       {"A": ["block_2"], "B": ["block_1"]}],
                            "groupB": ["branching_block2",
                                       {"A": ["block_1"], "B": ["block_2"]}]
                           }
                          ]
    }
    

    In this case, FindingFive will automatically assign participants to either "groupA" or "groupB" at the beginning of a study. This is how participant grouping works. Participants in "groupA" will then have a more strict accuracy requirement than those in "groupB" (above chance vs 80%) due to the different branching blocks used in the two groups. Finally, participants of both groups will be assigned to either Branch A or Branch B depending on their accuracy in the branching block.

    In fact, this example shows not only group-specific branching, but it also uses group-specific branches (as explained in the previous section). Such a complex experimental design is perhaps not common, but the good news is that FindingFive does support it if your study definitely requires such a setup.

Definition of a block

A block is defined by a property-value pair inside the property blocks when a blocking procedure is used. Its key serves as the name of the block, which can be referenced in other parts of the procedure. Its value defines the trial templates and the presentation pattern of that block.

The maximum number of trials in a single block is currently set at 150 to ensure an optimal participant experience.

Properties

trial_templates

A list containing the names of trial templates used in the block

pattern

A dictionary with two properties in the form of

{"order": SOME_ORDER, "repeat": NUMBER_OF_REPEATS}
“order”:

(string) the order in which the trial templates are presented, from the following options:

  • "fixed", which presents each complete trial template in the order specified in "trial_templates".
  • "randomized_templates", which presents each complete trial template specified in "trial_templates", but in a randomized order.
  • "randomized_trials", which presents one trial at a time from a randomly selected trial template, but leaves the stimulus pattern within each trial template intact. This order can be thought of as sampling the first available trial from a randomly selected template until all trials have been exhausted from all templates.
  • "alternate", which presents one trial from each template at a time, sequentially alternating between all specified trial templates. This order also leaves the stimulus pattern within each trial template intact. It can be thought of as “yoking” trials together from multiple templates and presenting them as groups of “yoked” trials.
  • "alternate_random", which first does the “trial yoking” in the same way as "alternate", and then performs randomization on the yoked groups of trials. This can be quite helpful in experimental designs where individual stimuli are presented across multiple trials in a set (e.g., mask, prime, target, fixation).
  • "random" (deprecated), see "randomized_trials"
“repeat”:

(integer) the total number of times that all trial templates will be cycled through.

Note

Interactions between the “pattern” of a block and the “stimulus_pattern” of a trial template are hierarchically organized. For instance, consider a block with two trial templates (A and B), each of which will generate 5 actual trials. The block has a “pattern” of {"order": "alternate"}, while both trial templates are set to use a “stimulus_pattern” of {"order": "random"}.

The net effect of this configuration is that, in the resulting block, trials will alternate between those of Template A and those of Template B, as in ABABABABAB. At the same time, if you ignore all the B trials, the stimuli in A trials are randomized. Similarly, if you ignore A trials, stimuli in B trials are randomized as well.

All interactions follow this rule (e.g., setting both patterns to random will mix everything together).

catch_trial

(deprecated) see catch_trials

catch_trials

(optional) A list (i.e., enclosed in []) of dictionaries, each of which specifies a trial template to be inserted in this block as catch trials. If more than one catch trial applies to the same trial template (see “after” below), they will be applied in order. Each dictionary must have the following form:

{"template": NAME_OF_TEMPLATE, "frequency": K, "after": [], "cycle_through": false}
“template”:(string) specifies the name of the template from which catch trials are created. FindingFive will create the catch trials in the same way that regular trials are created from the definitions of trial templates.
“frequency”:(integer) the frequency at which the catch trials appear in a study, defined as one catch trial every K number of non-catch trials.
“after”:(optional, a list of trial template names) the trials of which templates should the catch trial be inserted after. By default, catch trials are inserted after all trials of a block, regardless of the trial template. Specifying the "after" property allows one to insert catch trials after only trials created from a subset of trial templates in this block.
“cycle_through”:
 (optional, default is false) when there are more slots to insert catch trials than the number of catch trials in this block, whether FindingFive should cycle through the catch trials created from the catch trial templates. The most common usage for setting this to true is when one needs to insert the same catch trial over and over again.

For example, {"template": "TS1", "frequency": 5} will result in a block of trials where trials created from TS1, with its own stimulus pattern, will appear after every 5 trials (that is, 5 non-catch trials would appear between catch trials). These trials are intended to provide a periodic “check” on participants throughout a block (e.g., by checking a participant’s attention with an N-back task, or asking whether the participant understands the instructions at various points throughout the task).

cover_trials

(optional) Additionally specify a list of trial templates that will be used as the beginning trials of the entire block

end_trials

(optional) Additionally specify a list of trial templates that will be used as the ending trials of the entire block

fetching_point

(optional, default true) Turns on or off incremental fetching at the end of this block. If set to false, then the next block in the "block_sequence" of the procedure will be loaded together with this block (i.e., incremental fetching is off). The default behavior of a FindingFive study is fetching each block of trials incrementally as participants progress through the study.

For example, the following snippet from the above example defines a block:

"BL2": {
   "trial_templates": ["TS2", "TS3", "ATS1", "CommentTS"],
   "pattern": {"order": "fixed", "repeat": 1},
   "catch_trial": {"template": "CatchTS", "frequency": 3}
}

Definition of a branching block

A branching block extends the regular block and thus has all the properties of a block. It has an additional property named branching which specifies the branching instructions that should be carried out at the end of a branching block.

Note

Currently, FindingFive only performs branching evaluation when the participant reaches the end of a branching block. It is not yet possible to dynamically monitor the performance of a participant and apply conditional branching during the course of a branching block. We are working on it.

Properties

All properties of a regular block are applicable to a branching block. In addition:

branching

A dictionary specifying the branching instructions, consisting of the following:

  • method - the branching method, currently supporting either "accuracy" or "match".

  • triggers - a list of dictionaries specifying which responses should be used for evaluation. Each dictionary should be of the format {"trial_template": TRIAL_TEMPLATE_NAME, "response": RESPONSE_NAME} (see examples below). The trial template referenced must exist in the branching block, and the response referenced must exist within the referenced trial template. When method is "accuracy", the trigger response(s) must have a defined "target" so that correctness can be determined.

  • min_score - the minimum accuracy when method is "accuracy".

  • iterations - (optional) when method is "accuracy", iterations is a dictionary of the following form, which enables iterative training of participants on the same block of trials:

    {"min": MINIMUM_ITERATIONS, "max": MAXIMUM_ITERATIONS, "eval_all": false}
    

    Intuitively, "min" takes an integer number that specifies the minimum iterations of training (defaults to 1), "max" takes an integer that specifies the maximum iterations of training (also defaults to 1), and "eval_all" takes a boolean value (true or false) that specifies whether training accuracy should be evaluated on all iterations (true) or only the last iteration (false, which is the default).

  • branches - a dictionary where the keys define branch names and the values define target response outcomes.

    • When method is "accuracy", only two branches can be defined and must be in the form of {"success_branch_name": true, "fail_branch_name": false}. The first key defines the name of the branch to take when participants’ accuracy is at least the min_score and the second when accuracy is lower than min_score.
    • When method is "match", two or more branches can be defined in the form of {"branch_1": [...], "branch_2": [...], ..., "default_branch": null}. The keys are the names of the branches to take. The values are lists of target response outcomes. The length of each list must match the number of triggers. If participant responses match the target response outcomes, then the corresponding branch will be taken for the participant. If participant respones match none of the target response outcomes, the default branch (whose value is null) will be taken.

Warning

When method is "match", if no default branch is defined and there’s no match, your study will skip over the branching action entirely. Please go over your study design carefully to ensure all possible participant responses are accounted for.

Examples of Branching Blocks

The following snippet defines a branching block that implements the "accuracy" conditional branching method:

"BB_Accuracy": {
    // define trial templates and other properties here
    "branching": {
         "method": "accuracy",
         "triggers": [{"trial_template": "some_trial",
                      "response": "some_choice_response"}],
         "min_score": 0.8,
         "branches": {"A": true, "B": false}
    }
}

The following snippet defines a branching block that implements the "match" conditional branching method:

"BB_Accuracy": {
    // define trial templates and other properties here
    "branching": {
         "method": "match",
         "triggers": [{"trial_template": "some_trial",
                      "response": "another_choice_response"},
                     {"trial_template": "some_trial",
                      "response": "a_rating_response"}],
         "branches": {"A": ['left button', 4],
                      "B": ['right button', 5],
                      "C": null}
    }
}

In this examples, participants who click on the “left button” choice and give a rating of 4 will be directed to Branch “A”; those who click on “right button” and give a rating of 5 will be directed to Branch “B”; all other participants will be assigned to Branch “C”.