TurkSuite MTurk Tutorial

Please cite as: Alper, Morris (2014). "TurkSuite MTurk Tutorial". Available online at http://mturk.mit.edu/tutorial/


0. Introduction
1. The MTurk User Interface
2. Simple Surveys
3. More Advanced Surveys
4. Analyzing Data

3. More Advanced Surveys

One limitation of the type of survey we coded above is that all of the questions are displayed at once. This means that we have no control over the order in which the subject answers the questions — an important variable in social science surveys. A better design would have each question be displayed one at a time, with the user pressing a "next" button to advance to the next question. We could then easily vary the order in which the questions appear.

In order to do this, we will use the following procedure: Each page of our experiment will be surrounded by a <div>...</div> tag. You can think of this as being an invisible box. Each div tag will have a unique id attribute to identify it. We only want the first page to be visible when you open the survey, so all of the other divs will have the attribute style="display: none;", which makes them invisible. Finally, we will add a button to each page which makes that page become invisible and the next page become visible. The button which submits the survey will be located on the final page.

To generate such surveys, you may use the TurkSuite Template Generator. This tool generates multi-page surveys with custom introductory test and a variety of other optional features. It includes a few library imports, the most important of which is Navigation.js, which provides functions for going to the previous or next page of the survey. You can use it by adding the attribute onclick="next();" or onclick="back();" to your buttons (e.g. <input type="button" onclick="next();" />).

By default, the Template Generator creates a HIT with only one intermediate page that says "[your content goes here]". You can change this by modifying the HTML of the HIT. If you want to add more pages, make sure that they are of the following form:

<div class="page" id="#" style="display: none;">
<div class="cell">

Where in place of "#" you use the page's number in the sequence of all pages (e.g. the 1st, 2nd, 3rd page, etc.). Also make sure that all following pages have the correct number for their id attribute.

Of course, you can extend this structure to as many pages of questions as you would like. However, this will always result in the questions being presented in the same order to all workers. We would like to randomize the order, so that each worker can answer the questions in a different order, so that possible biases which the question order could introduce will be averaged out over many trials. The Navigation.js library provides a convenient way to do just this. Insert the following code into your HTML code, right below the first three lines where we include the JavaScript libraries:

randomize(m, n);

Where m and n are the numbers of the first and last pages which you would like randomized. The function will randomize these pages and all intervening pages so that they appear in a random order each time the HIT is opened.

Sometimes we would like to split workers into different groups which see different questions. This can be done using what is known as grouping. Grouping means that workers are split into equally-sized groups, and each group is assigned a unique name. We can then check what the group name is of the worker, and show him or her different content based on what the name is.

The way grouping works in MTurk is as follows: In your HTML file, you may enter the special sequence ${group}. Although you see this as just the text ${group} when you preview the project, the workers will instead see some text based on what group they are in. Let's say, for example, that you put the following HTML code into your project:

Hello! You are in group <b>${group}</b>.

Then it could be that some workers will see this:

Hello! You are in group 1.

While others will see this:

Hello! You are in group 2.

This is because you yourself, as the requester, get to chose what ${group} can be replaced with, as will be explained below.

Technically, the three possible things that the workers can see above are different HITs. A HIT is defined as a particular task that a worker does, hence the name Human Intelligence Task. Since each worker sees something different, they are all performing a different task. The code that you wrote to generate all of these HITs is the project. Each one of these HITs can be completed by multiple workers, depending on what you entered for "Number of assignments per HIT" in the "(1) Enter Properties" tab of the "Edit Project" menu on MTurk. Each time a worker completes a HIT is called an assignment. A batch is a collection of assignments for all of the HITs that your project generates, which you pay for as a package when you click on "New Batch" in the "Create" page.

You may specify what ${group} gets replaced with as follows: Try taking the following simple code and making an MTurk project out of it on the sandbox:

Hello! You are in group <b>${group}</b>.
What is your name? <input type="text" name="nm" />
<input type="submit" />

Once you create the project and click on the orange "New Batch" button, you will see the following pop-up window:

Open a text editor, enter the following text, go to Save as, and save the file as testing.csv:


On MTurk, press the orange "Choose File" button, select your file testing.csv, and press "Upload." MTurk will take you to a preview screen. You can preview all three of the different HITs that this project generates by pressing the "Previous HIT" and "Next HIT" buttons below the preview box, as you can see below:

You can use this feature to show different stimuli to subjects depending on what group they are in.