Hi,

I conducted a study today and ran into massive issues with people not getting grouped, even though there were enough players ready and waiting to be grouped.

After quite some time checking the code, debugging, and looking stuff up, I finally found out that the code was just fine, but instead in oTree >= 5.8 a new dropout detection mechanism with group_by_arrival_time was implemented. (see https://otree.readthedocs.io/en/latest/misc/version_history.html and https://groups.google.com/g/otree/c/XsFMNoZR7PY) Unfortunately, this is nowhere mentioned in the handbook.

While I believe this can be a very useful feature, it can mess up a study just the same. Also, most likely you won't even notice any issues when testing, as this only occurs when people are not grouped quickly enough (before the first auto-refresh), which they usually are when testing (especially with browser bots). 

Because of this, I wanted to ask if this can be added to the handbook and if there is any way to turn this off or getting an option to do so?

Thanks in advance and please forgive me my frustration.


All the best,
Sebastian

Hi, thank you for the feedback; you're right that this should be mentioned in the documentation; I will put it there.

I am also thinking of putting some default text on the wait page that says "Don't switch away from this tab".

For the time being, you can set some text like that yourself to make participants aware:

body_text = "Don't switch away from this tab"

If users are made aware that they need to stay on the tab, do you think that is OK? Or do you still prefer the previous behavior? (If so, I would be curious for your reason.)

Hey Chris, 

thanks for the quick reply!

I basically did exactly what you suggest and quite prominently added some text on the grouping wait page, telling participants to please not switch away from this tab in any way. However, even though I saw slight improvements, I still had significant problems due to this feature. 


Here is the exact situation I was facing (sorry if this gets a bit long):

We have a multiplayer game with groups of four and 10 rounds. Once a group is formed it persists. All four players then need to make a decision before the group can proceed to the next round. However, as far as we can tell, the issue only occurs on the grouping wait page, but not on the wait pages between the rounds.

Additionally, we have four different treatments, which are randomly assigned to participants. Groups can only be formed with players in the same treatment.

As participants need to be paid by an hourly rate, I have to limit the amount of time people spend on the grouping wait page. Also, since it is possible that there are not enough players left to even form a group, those players who are waiting, need to be forwarded after some time to collect their participation fee. After all, they are there, but just cannot find a group, which is not their fault. I set this time to 10 minutes. 

So, when players land on the grouping wait page, they have 10 minutes to find a group or get forwarded to collect their participating fee. Due to the speed participants get sampled into the study (spreading them across 4 treatments slows this down further), usually players spend about three minutes for four people to be available for a group. 

Ultimately, even though we explicitly and very prominently tell them to not switch away, players do switch away. Maybe they do not care or do not understand what switching away means exactly. Either way, it seems to me that they do something else and every now and then check the wait page.  

The end result was that players were ready for the experiment, but never got into a group, even though there would have been enough players (I know this from monitoring). In fact, many participants even messaged me that they were waiting, but never got into a group and if they can re apply for the study. So, they certainly were waiting to play the game. 

In my case this feature is much to strict and really messes up the study. I believe roughly 1/4 of the participants that were sampled hit the 10 minute threshold and were forwarded to the end, even though there would have been numerous occasions with enough players to group with while they were waiting.

Also, I cannot distinguish from participants who just did not exactly comply with the rules of not switching away, or just did not find a group due to lack of players. Therefore, I have to pay them all. 

By the way, this worked perfectly fine in otree 3, however, there I had performance issues and needed to upgrade to otree 5. 


So, yes, I would very much prefer the previous behavior or an option that allows for turning this new feature off.

Thank you again and I very much appreciate your support!


Best,
Sebastian

Hi, thank you very much for explaining. A lot of interesting points.

I would like to hear what other people think about this feature also. Depending on whether people prefer the new or old behavior, I might remove it and revert to the previous behavior. (Supporting both modes and making an option to switch will be too complex.)

I implemented this feature because there were major problems with dropouts that result in groups getting stuck, which often means we have to discard them. I think in many cases the participant doesn't consciously decide to drop out, but just gets distracted, bored, or is half-hearted about participating in the study.

Ideally, the group_by_arrival_time algorithm should include everyone who is properly paying attention, and exclude everyone who is distracted/unreachable. It's hard to do this perfectly, so no matter what we will fall too far on one side or the other. The optimal tradeoff depends on a few things, such as the volume of participants. I can understand that if you have a small number of participants and are struggling to get enough people to form a group, this feature can cause additional difficulty. On the other hand if you have a large number of participants and have groups constantly forming, then it's more important to filter out people who are going to cause issues with their group.

Given how dropouts are a major issue in online experiments, it seems reasonable to require people to stay active on the page. Participating in these experiments is work (usually paid), and it should require the user's undivided attention, especially because them getting distracted could affect other players. I know that it is boring to sit at a wait page and some people want to multitask productively, but I just think multitasking opens up too many issues. Even if you are paying attention, it's still possible to get distracted if you notice some important email, etc. And even if you check the oTree tab periodically, you can still end up a minute or two late and end up slowing down the rest of the group.

As you observed, participants may not notice/understand the instruction not to switch away, but I think that problem could be mitigated through changes in oTree-core to more clearly communicate it, like indicating the status of whether the user was considered active or not. For example, we could show in the browser tab something like "You are inactive", or when the user switches back to the wait page we can say "welcome back, you are now eligible for grouping again". Also since you said participants may not understand what it means to "switch away", we could word it more explicitly or prominently.

I'm sure some people will still ignore these instructions and switch away, but it may actually be a good thing to filter out people who don't follow instructions.

We can also indicate in the admin interface which participants are really waiting, vs. which ones have switched away.

Alternatively, one technique with the previous algorithm was the "audio alert" that makes a beeping sound so the user knows to switch back to that tab: https://www.otreehub.com/projects/otree-snippets/  I think it can work some of the time, but I question whether it is enough.

Anyway I'm curious for your thoughts as well as anyone else reading this.

By the way, we should also think of ways to keep users engaged on the wait page. For example, you can put a game in the wait page. Using this technique: https://otree.readthedocs.io/en/latest/misc/advanced.html#custom-wait-page-template

Use this template puts an online 'snake' game in the page:

{{ extends 'otree/WaitPage.html' }}
{{ block title }}{{ title_text }}{{ endblock }}
{{ block content }}
    <p>Please play this game while you wait for other participants...</p>
    <iframe src="http://slither.io/" width="100%" style="max-height: 500px" height="500px"></iframe>
{{ endblock }}

Hi Chris and Sebastian,

Multiplayer game lobbies have the same problem. Players queue up, but leave the computer / console to get a drink and are nnot responsive when they enter the match automatically. 

The solution in these contexts is usually to announce that a match has been found and give all players of a potential match a couple of seconds to click a button to confirm that they are ready to go. If all players click the button in time, they form a group and advance to the match. 
If a player fails to click the button in time, they are marked as a "dropout". Typically they have to actively join the queue again. Others, who clicked the button but could not be matched because somebody else did not join in time are simply returned to the queue. 
Here is a video of such an implementation: https://youtu.be/3taTQYor7ZQ 

Maybe a similar implementation could help. It would have to be combined with a visual cue (blinking tab title?) and an audio cue, such that people can switch back to the relevant tab.

In any case, I'd appreciate an option to turn off the automatic dropout or at least to detect it (some flag on the player that one could check in the group_by_arrival_time method?). I want to be able to ignore their dropout and match them anyway. The reason is that I can then at least have the group progress through the experiment (using page timeouts).

Thanks for considering it!

Best
Christian

Thank you Christian, clicking to confirm the group seems like a very promising idea. I need to think through some of the implications, will spend some time on this.

Hi all,

I just released a new version of oTree with a change to group_by_arrival_time, making it show a presence icon in the page tab. If the page tab is open, it shows 🟢. If the user is on a different window, the tab shows 🟡.

*There is also a fix to a possible race condition that could have been causing people to get stuck on wait pages, even if everyone was present.*

The presence icons are a minimal enhancement to make it clearer to participants when they are considered active and eligible for grouping. It also is useful for oTree users who are testing their own experiment to ensure that it works properly.

It would be clearer still if we put some text like "You are active and eligible for grouping" etc, but I am leaving that for a bit later because that will require deciding the exact text to show, translating it to multiple languages, etc.

=====

I also released a BETA version of oTree with the above change, plus a change to show wait page status in the admin "Monitor" tab. So you can see who is waiting actively, who is in another window/tab, and who has disconnected. See the attached screenshot.

You can install it with:

pip3 install otree -U --pre

Hi Chris

I can add my opinion on this issue. For me, and other researchers I know, the old behaviour of keeping inactive players in the grouping is much preferred over the new behaviour. And also more intuitive. 

We run experiments where groups persist over multiple rounds. We use the group_by_arrival_time method to detect when all players have arrived in each group and move them forward, as is recommended in the docs. We see a lot of problems with people switching away from the browser while waiting - people will do that and it is understandable. That then causes the entire group to stall. In our experiments we now find it necessary to be on chat with participants and tell them to refresh the page when this happens. 

It also causes a few headaches with testing as we run multiple participants in different browser tabs, and need to refresh those pages to get otree to pick up the players. 

Is there no way to workaround the issue? I see that you said that offering both options would be too complex, but I really feel it would be beneficial. For example if we had access to a second list e.g. all_waiting_players and active_waiting_players and then we can use whichever suits best for the experiment? Or even better if we can access "last_active_time" of each player so we can choose to not include them if they are inactive. 

Grouping and handling dropouts is the hardest part of running experiments. Otree generally handles this very well but only including active players complicates things in many situations. 

Do you know of any potential workaround? 

Warm Regards and Thank you for all your work. 
Charlie Pilgrim

Thanks for the feedback!

> We see a lot of problems with people switching away from the browser while waiting - people will do that and it is understandable. That then causes the entire group to stall. In our experiments we now find it necessary to be on chat with participants and tell them to refresh the page when this happens.

How about telling participants not to switch away, or at least to keep the tab visible? (they can be using another app/tab as long as the oTree page is still visible, e.g. if they have 2 windows side by side) This seems reasonable to me, because if they have switched to another program and can't see the oTree page at all, how will they know it is time to start playing again? Especially if it's something time sensitive that others will be waiting for.

Also, they don't need to refresh the page. Just switching back is enough. There is an indicator light in the tab title (green/yellow) that shows whether they are considered active. Tell them that they will be ignored unless they keep it green.