TurkSuite MTurk Tutorial

Please cite as: Alper, Morris (2014). "TurkSuite MTurk Tutorial". Available online at http://mturk.mit.edu/tutorial/

Contents:

0. Introduction
1. The MTurk User Interface
2. Simple Surveys
3. More Advanced Surveys
4. Analyzing Data

2. Simple Surveys

Now that you know how to upload HITs to MTurk, you can start making your own custom HITs.

When a worker accepts a HIT, the HIT is displayed embedded within the MTurk website. To understand how HITs are coded, you must therefore understand something about how websites are coded in general.

Every website has two elements: how it is displayed, and its source code. The source code is the hidden machinery which makes the page appear the way that it does. On Firefox, you can view the source code of a website by typing the keystroke Ctrl-U (or in Mac, Cmd-U). A window should pop up looking like the following:

This is the source code for this website. The language it is written in is called HTML — HyperText Markup Language. You can find a full HTML tutorial in the external links, but for our purposes we will only need to learn some very basic HTML.

To start off, we'll code a very basic HTML file. Open a text editor (such as Notepad in Windows or TextEdit in Mac OS X) and enter the following text:
<b>Hello</b> world!

Go to File > Save As... and save this file as "test.html". This type of file, ending in .html, is called an HTML file. Now if you navigate to the HTML file and double-click on it, it should automatically open in a web browser. You should see something like this:

The file will open in whatever your default browser is. If you would like to open in a different browser, you can right-click on the file icon and use the Open With tool. This tutorial will assume that you are using Firefox to view HTML files.

Notice that instead of the text <b>Hello</b> which you entered into the text editor, the browser is displaying Hello, in boldface text. Things like <b>...</b> are known as tags. The <b> part is called a start tag, the </b> part is called an end tag, and the letter "b" is called the tag name of the tag <b>...</b>. HTML has many tags which change the appearance of whatever is sandwitched between them. The <b>...</b> tag has tag name "b" because it bolds the text that it surrounds.

Now, let's do a slightly more complicated example. We need to edit the source code of this HTML file. Of course, you can view the source code by typing Ctrl-U (or Cmd-U) as described above, but this doesn't let you modify the code. To do this, you must right-click on the file icon, go to Open With, and select your preferred text editor. Modify the code to read:

<h1>Heading 1</h1>
<h2>Heading 2</h2>
<h3>Heading 3</h3>

<p>This is a paragraph of text.</p>
<p>This is <i>another</i> paragraph.
But this is not.</p>

Upon saving this and opening the file in Firefox, you should see the following:

In addition to now knowing how to modify the source code of an HTML file, you can see the use of a number of new HTML tags. "h1", "h2", and "h3" are used to create respectively smaller headings. "p" is used for paragraphs of text, and makes them appear separately from each other. Notice that a line break alone (pressing enter) does not create a new paragraph, since the text "But this is not." is displayed on the same line as the preceding text. "i" is used for italic text.

Sometimes there are HTML tags which do not enclose anything, such as the line break tag <br />. Notice that after the tag name there is a space followed by "/>", and no end tag. The tag <br /> creates a line break wherever you put it. For example, try modifying the source code to read:

<h1>Heading 1</h1><br />
<h2>Heading 2</h2>
<h3>Heading 3</h3>

<p>This is a paragraph of text.</p>
<p>This is <i>another</i> paragraph.
But this is not.</p>

The resulting web page will look like this:

As you can see, an extra line break was inserted between the first two headings.

It's worth mentioning that you don't need to put each tag on its own line. In theory, you could write our HTML code entirely on one line, as follows:

<h1>Heading 1</h1><br /><h2>Heading 2</h2><h3>Heading 3</h3><p>This is a paragraph of text.</p><p>This is <i>another</i> paragraph. But this is not.</p>

Obviously this is ugly and should not be done in practice. It's a good idea to try to keep things readable. However, it is evident that newlines don't matter much in HTML, so feel free to put them wherever you think would be best.

Having mastered the bare-bones basics of HTML, we can start learning how to put surveys into websites. The idea behind a survey is to have the subject input data, and in fact the main HTML tag used for this is the input tag. This tag is used to create input tools such as:

Here is HTML code which contains two of these inputs. Try saving it in an HTML file and viewing it in Firefox:

Type here: <input type="text" />
<br />
What is your gender?
<input type="checkbox" /> Male
<input type="checkbox" /> Female

The result should look like this:

Notice that the tag name for all of these input tools is the same — "input". The key difference is in what follows the tag name. Something like type="text" is called an attribute of the tag. Attributes allow you to customize HTML tags, changing their details while keeping the general idea the same. All input tags have the general idea of accepting user input, but how they do this depends on their type attribute. We say that text is the value of the attribute type in the expression type="text". Notice that the value is surrounded by quotation marks, while the attribute name is not.

We see that the difference between the text field input and the checkbox inputs is their type attribute, which has value text for the text field and value checkbox for the checkboxes.

There are two more attributes which are worth adding to our inputs, though they are not very useful to us yet. These are the name and value attributes. Try modifying the source code of the above HTML file to read as follows:

Type here: <input type="text" name="textinput" />
<br />
What is your gender?
<input type="checkbox" name="gender" value="m" /> Male
<input type="checkbox" name="gender" value="f" /> Female

When you open this in Firefox you should not see any different in how it is displayed. The advantage of giving names and values to inputs will be relevant once we get our results from MTurk. MTurk will effectively tell us, for example, that "The first worker entered the text Hello into the input named textinput.", or that "The first worker clicked on the checkbox with name gender and value m" Therefore we need to assign names to our inputs so that MTurk can give us useful information.

Now we can add radio buttons. Modify the code to read as follows:

Type here: <input type="text" name="textinput" />
<br />
What is your age?
<input type="checkbox" name="age" value="y" /> Young
<input type="checkbox" name="age" value="o" /> Old
<br />
What is your gender?
<input type="radio" name="gender" value="m" /> Male
<input type="radio" name="gender" value="f" /> Female

The page will now look like this:

You may have noticed that both checkboxes were given the same name—age—and similarly for both of the radio buttons. This is because both checkboxes are really answering the same question, and the data that the two encode are related to one another. This is more obvious with the radio buttons. Notice that it is impossible to check both at the same time, since checking one automatically unchecks the other. This only works if the names of both radio buttons are the same, so that the browser knows that they are connected to each other.

Finally, let's add a submit button. The button doesn't need a name since it doesn't store any data. However, it does use an attribute called value which specifies what text should be shown on the button. Modify the code as follows:

Type here: <input type="text" name="textinput" />
<br />
What is your age?
<input type="checkbox" name="age" value="y" /> Young
<input type="checkbox" name="age" value="o" /> Old
<br />
What is your gender?
<input type="radio" name="gender" value="m" /> Male
<input type="radio" name="gender" value="f" /> Female
<br />
<input type="submit" value="Done!" />

You should see the following:

Clicking on the button will not do anything in your browser now, but in fact this will be a fully functional HIT once we upload it to MTurk. Make a new project on the requester sandbox with any of the default templates, and then press the "Edit HTML Source" button and copy-paste in our code. Then go to the tab "(3) Preview and Finish". You should see the same page in the "HIT Preview" window:

Clicking on the submit button (reading "Done!") will now give the following error:

This is perfectly fine. We cannot actually submit the HIT because we were not assigned it as a worker. However, if you assign workers to this HIT, then they will be able to submit and you will get real data.

Of course, the data from the above survey is rather trivial, and we'd also like to have an introduction, disclaimer, and a section for comments at the end. We can do most of this using the skills we have already learned. Here is our first attempt at making a real survey, which in this case asks subject about their use of pronouns:

<h1>Pronoun Test</h1>
<h3>This test is being conducted by the University of Foobar. If you feel uncomfortable at any time, you do not need to continue taking the test.</h3>

<p>Please select whichever sentence sounds more natural to you from the pairs of sentences below</p>

<input type="radio" name="question1" value="1" /> I and John went to the store.
<br />
<input type="radio" name="question1" value="2" /> John and I went to the store.
<br />

<br />
<input type="radio" name="question2" value="1" /> He hates I and John.
<br />
<input type="radio" name="question2" value="2" /> He hates John and I.
<br />

<br />
<input type="radio" name="question3" value="1" /> He can see John and me.
<br />
<input type="radio" name="question3" value="2" /> He can see me and John.
<br />

<br />
<input type="radio" name="question4" value="1" /> Me and John like ice cream.
<br />
<input type="radio" name="question4" value="2" /> John and me like ice cream.
<br />

<br />
Comments: <textarea name="comments">Enter text here.</textarea>
<br />

<br />
<input type="submit" value="Done!" />

The only new tag this used was "textarea". As you can see, the start and end tags for "textarea" enclose the default text which starts out in the textarea (in this case, "Enter text here."). Again, we give the textarea a name attribute so that MTurk can tell us what the workers enter into it.

Try this out in Firefox and on MTurk. You should get an survey like the one below:

Pronoun Test

This test is being conducted by the University of Foobar. If you feel uncomfortable at any time, you do not need to continue taking the test.

Please select whichever sentence sounds more natural to you from the pairs of sentences below

I and John went to the store.
John and I went to the store.

He hates I and John.
He hates John and I.

He can see John and me.
He can see me and John.

Me and John like ice cream.
John and me like ice cream.

Comments:

Congratulations — you now have the skills to run meaningful experiments on MTurk. Of course, the surveys you are currently capable of making are rather limited. In subsequent sections we will broaden the range of surveys which you can create, but we will always be relying on the basic machinery developed in this section.