Trust in Second Life

I was pleased to learn today that there has been some discussion about my research on Terra Nova. John Duffy of the University of Pittsburgh Economics Department has written an article criticizing the reliability of data collected from the experiments in virtual worlds I conducted under the guidance of Louis Putterman of Brown University and Enrico Spolaore of Tufts University. Thomas Chesney of Nottingham University Business School picked it up on the blog, Terra Nova, so it seems appropriate for me to provide my perspective given that I have not yet published the findings of my recent experiment in the virtual lab.

Certainly, there are some unresolved methodological concerns with gathering data in online settings. However, it is important to weigh these costs against the benefits of online research: our methods allow us to test the external validity of general principles by experimenting on a different subject pool than the usual undergraduates; additionally, the combination of automated data gathering scripts and a population who are willing to participate at a fraction of the normal cost allows samples to be dramatically larger. The end result was that I collected over 1,200 data points over two months on a graduate student budget, realizing a 95% cost savings compared with more traditional laboratory methods. This allowed me to test five treatments on this subject pool and tease out more subtle factors that influence behavior that might not be detected in a smaller sample.

More, after the jump.

Duffy's objections to virtual experimental methods appear to be:
1) Data accuracy. Subjects can be dishonest about their demographic information and are more likely to do so compared with real life
2) Selection Bias. There is little control over who shows up to participate, their knowledge of economics, and the low stakes nature of the experiment may result in some subjects dropping out prematurely
3) Identity Mapping. "There is little control over whether the same individual is logged in on multiple machines, under different identities, perhaps playing a two-person game with himself."

It is important to note that even if the demographic data may not be perfectly accurate, the substance of the experiment was about subjects' behavior in the trust game. On this issue subjects were making decisions with real (in virtual terms) stakes about which they would be truthful. For example, while Duffy did not feel compelled to be accurate about his age and gender, he indeed answered the core experiment question on trust with what he truly believed to be the "best" course of action. Other participants selections provided results that were consistent with the trust and reciprocity effects observed in the 1995 Berg, Dickhaut, and McCabe experiment, in contrast with the subgame-perfect equilibrium expected by neoclassical economic assumptions.

What remains of questionable data integrity, however, are the 28 demographic and background questions that followed the experiment. Indeed some (17%) of participants did not choose to complete the experiment and followup questions. These "partial" data points were dropped from the resulting analysis. Whether there were patterns in peoples' decisions to drop out prematurely does indeed affect the outcome is a matter that could be tested by further experiments and data analysis.

While the followup questions were indeed not for "extra payment," it is not accurate to say that there was no incentive to complete the survey aspect of the experiment because subjects were required to complete the questions in order to receive any earnings from the prior question. Nevertheless, Duffy's concerns about the accuracy of subjects' responses is noted and is a real practical consideration in the design of experiments in any setting. In online experiments (both in virtual worlds and in web-based experiments), the absence of an authority could result in users providing inaccurate information on their demographics. One possible way to assess the accuracy of the demographic data would be to verify aspects of the data with previous data provided to Linden Labs. However, this does beg for further research into mechanisms to elicit truth-telling in anonymous online settings.

Duffy's final concern is that "there is little control over whether the same individual is logged in on multiple machines, under different identities, perhaps playing a two-person game with himself." In anticipation of this, our script prevented individual avatars from participating in the experiment more than once. We also used a delay mechanism between matched players so subjects would not know the identity of their counterpart. These two features made a two-player game with oneself practically impossible.

Underlying this concern, however, is a legitimate issue about players' use of alternate characters, known as "alts" to participate in the experiment multiple times using different avatars, a practice I refer to as experiment farming. This can sometimes be manually cleaned by noticing obviously duplicated avatars with names such as "Po Potez," "Po1 Potez," etc. My experience is that such experiment farming is most prevalent when participants are offered a large reward for participating in a relatively short experiment.

In closing, Duffy has identified some very real concerns to be addressed in designing effective virtual experiments. In truth, I think he is just scratching the surface about the issues that virtual experimentation needs to overcome. However, to invalidate these methods while in such a nascent state would be an overreaction. I believe the solution is to expand academic inquiry into experimentation in virtual worlds and develop better tools for collecting online data. In the mean time such confounding issues should certainly be addressed by researchers, and the field is wide open for the design of experiments to demonstrate the dimensions along which subjects behave differently in virtual worlds than the real world.