How to block proxies and VPNs in Qualtrics
EDIT:
Unfortunately, this method will no longer work because the webservice it used (getipintel.net) has blocked Qualtrics. The method should be able to be adapted to a different webservice, but I have not yet found and tested such a service. I hope to find time to do this soon, but I am not planning to run any MTurk samples in the near future so please forgive me if this take a while.
Original Post:
In recent months some questions were raised about the quality of MTurk data. The drop in quality was originally presumed to be caused by bots, but research by TurkPrime demonstrated that the problem was not due to bots, but rather due to non-US residents from less affluent countries (primarily India) using server farms to hide their location. Given the importance of knowing that your research is being conducted on the population you think it is, and, more basically, that the English language skills of your sample are sufficiently good to complete the survey accurately, blocking such participants is important for trust in MTurk research.
It is possible to block such users with MTurk directly. However, it requires a TurkPrime account with 'Pro Features'. For those of us without such an account (and a bureaucratic nightmare to work out if such an account can be acquired) this solution is no good. This problem inspired me to try and determine if this could be solved in Qualtrics directly and it turns out it can.
The solution I will present uses a third party web service that checks IP addresses against a list of known proxy and VPN IP addresses. We thus need to tell Quatrics to send each participant's IP address to this web service and for it to return a value indicating whether or not the IP address in question is a proxy or VPN. It is then a simple matter of setting up some branching logic to take those using a proxy or VPN to an 'End of Survey' element and allowing others to continue the survey as normal. The following steps will set this up in your survey:
1. In your survey click 'Survey Flow' in the top left of the survey screen.
2. Click 'Add a New Element Here' where you want the query to occur in your survey flow (or move it there after adding it elsewhere).
3. Select 'Web Service'
4. In the URL box enter: http://check.getipintel.net/check.php
5. Click 'Add a query parameter to send to web service'
6. Enter 'ip' in the 'Parameter to Web Service' box. Click on 'Set a value now' and enter '${loc://IPAddress}'
7. Repeat step 6 with the following values (each on a new line):
'contact = youremail@youremail.com' [enter you actual email here]
'flag = m'
'format = json'
Once you are done, it should look like this ('Method' should remain as 'GET'):
It is important you enter your actual email in point 8. The administrator of the web service used to check for proxies and VPNs may wish to contact you if your queries are causing them problems for some reason. If you have entered a bogus email they will not be able to contact you and your bogus email will be blocked. The service will then cease to work with that email. Similarly, you will have to check your email regularly in case you are contacted. Make sure you respond within two days to ensure your legitimate email is not blocked.
9. Once these values are entered, click 'Test' (directly to the right of the URL box, see image above) and select 'result' in the popup box that will appear. When this is done, click 'Add Embedded Data' (see image).
Once you have completed up to step 9, it should look like this:
Your survey will now check all participants' IP addresses to determine if they are using a proxy or VPN and store this information as Embedded Data. All that is now needed is to remove participants who are found to be using a proxy or VPN. To do this we use some simple branch logic.
10. Click 'Add a New Element Here' immediately after the Web Service element (or move it there as for step 2).
11. Select 'Branch'.
12. Click 'Add a Condition'.
13. Select the drop down menu where 'Question' appears and select 'Embedded Data'
14. In the space that will appear next to 'Embedded Data' enter 'result'.
15. In the final box on the right, enter '1'. (The second drop down menu should remain 'Is Equal to').
Once step 15 is complete the branch logic element should look like this:
16. Click 'OK'.
Now we need to add the final element, which is an 'End of Survey' element that will exclude those who were found to be using a proxy or VPN.
17. Click 'Add a New Element Here' directly below the branch element.
18. Select 'End of Survey'.
The End of Survey element will show participants the standard end of survey, which is unlikely to be appropriate in this case since participants will not have actually done the survey. Therefore, you may wish to add a customised message.
19. To add a customised message, click 'Customize' in the End of Survey element.
20. Select 'Override Survey Options'.
21. Select 'Custom' and then select a message from the drop-down menu. You will need to create a new custom message the first time you do this (see image).
If you do not want to write your own custom message feel free to use mine:
"You are using a proxy or VPN service. Although there are many legitimate reasons to be using such a service, it can also be used for people to hide their location. We need to know which country our participants are from and so have decided to prohibit those using a proxy or VPN from participating.
Sorry for any inconvenience this may cause."
There are a number of other options you may or may not want to select in the End of Survey element. See Qualtrics' End of Survey page for details.
If everything has been done correctly, this is what the section of your survey flow should look like:
Finally, please note that this solution uses getipintel.net to check IP addresses. The service is free, but has certain terms and conditions of use, which I would advise anyone planning on using this solution to check before commencing. In particular, if you are likely to require more than 15 checks per minute or 500 checks per day, you will require a custom plan (which may not be free). The website also contains useful additional options for those who want something more complicated than the basic solution presented here.
That's it folks. I hope this solution works for you and gives you piece of mind with regards to the quality of your data using MTurk. If you have any questions I can be reached at t.bainbridge@student.unimelb.edu.au or you can fill in the form on the main page of the website. (Sorry I don't have Twitter).