Research:Account creation UX
This is a project of the Editor Engagement Experiments team at the Wikimedia Foundation.
Goals
[edit]
Our goal for account creation UX experiments is to make small, measurable, iterative changes to the account creation process to find what, if anything:
- increases account creation completion rates and
- increase the number of new accounts that edit
Methodology
[edit]- Pre-launch
- Permanently disable current A/B test that has been live for months or more with no analysis, bug 37787 (links from mw m w) filed. ACIP is partially documented here. Done mid-August 2012
- ACUX-1
- Test the current signup page against a fully redesigned version. October 4, 2012
- ACUX-2
- Test of current signup page against redesigned version, with no list of benefits and added error logging. October 18, 2012
- ACUX-3
- Test of client-side form validation vs control. November 6, 2012
- CAPTCHA
- Test of removal of the CAPTCHA on overall registration rates. See /CAPTCHA for documentation. January 10, 2013
Metrics
[edit]- Account creation click-through rates
- Number of users clicking on the submit button per landing page impression
- Account creation conversions
- Number of accounts successfully created for a given treatment per landing page impression
- Active editors
- Proportion of users who complete at least one edit
- Productive editors
- Proportion of users who complete an NS0 edit in 24 hours that is not reverted
- User blocks
- Proportion of new accounts that go on to be blocked for username-related reasons (as measured by all blocks, filtered by block log comment)
- Email authentication rate
- Proportion of user accounts per treatment with a verified email address
Experiments
[edit]Methods
[edit]Our first experiment will randomly deliver an account creation landing page to potential new users: 50% will receive the current default as control, with the other 50% receiving the new signup page (full description of differences is here). Users who do not have JavaScript enabled or who are visiting from an unsupported browser will not be bucketed in either group, and will be served the default registration page.
The experiment involves the measurement of conversions, blocked accounts, and user contributions as a function of the treatment received. As of October 2012, there are roughly 4,000 new account registrations per day, therefore in a one-week duration we should expect approximately 30,000 data points (15,000 per treatment). The vast majority of these users will be genuine new registrations (some "attached" users, identifiable sockpuppets and bots will be filtered prior to the analysis).
Since client-side form validation is a new addition to MediaWiki account registration, the version deployed in the first iteration will not include it, but rather update the user interface significantly while still relying on the old server-side validation method.
The activation time of this experiment is as follows (all times UTC):
| Iteration | Start | End | Bucket IDs | Log version |
|---|---|---|---|---|
| ACUX-1 (new design, server-side validation vs control) | 2012-10-04 23:30:00 | 2012-10-18 23:30:00 | acux_1, control_1 | v.3 |
| ACUX-2 (new design without list of benefits, server-side validation vs control, high-level error logging) | 2012-10-18 23:30:00 | 2012-11-02 06:28:04 | acux_2, control_2 | v.4 |
| ACUX-2* (same as ACUX-2, patch for IE introduced, log version bumped) | 2012-11-02 06:28:04 | 2012-11-06 11:30:00 | acux_2, control_2 | v.5 |
| ACUX-3 (new design, with client-side validation vs control) | 2012-11-06 11:30:00 | 2012-11-23 11:30:00 | acux_3, control_3 | v.6 |
| ACUX-3* (same as ACUX-3, removed bucketing) | 2012-11-29 23:16 deployed on 1.21wmf4 | 2012-12-03 19:05 (lost when wmf5 deployed to enwiki) | acux_3 | v.7 |
| ACUX-3* (same as ACUX-3, removed bucketing) | 2012-12-04 01:31 re-deployed on wmf5 | 2013-28-03 | acux_3 | v.7 |
| CAPTCHA | TBD | TBD | TBD | NA |
-
The first iteration of the account creation test (ACUX-1). The second iteration (ACUX-2) used an identical design but did not include benefits listed. -
The third iteration (ACUX-3), including clientside validation.
Notes
- ACUX-1 A temporary interruption of the experiment occurred between 2012-10-08 11:36:00 and 2012-1-09 23:34:43 which effectively disabled the ACUX test for new users clicking on the Create Account link. This resulted in 36 hours of missing client-side events and server-side events with no bucket.
- ACUX-2 A good faith change in Setup.php that went live on all wikis with 1.21wmf2 [1] produced as an unintended effect the display of the "real name" field on the account creation page of all Wikipedias. This resulted in an abnormal growth of new users (58% of all registrations) with a real name set. We should consider filtering out all these users (N=5,366) registered between 2012-10-22 18:17:02 and 2012-10-24 05:58:56 if we have reasons to believe this change will affect their behavior compared to regular users.
- ACUX-2. A test of a Signup Call to Action for Article Feedback users was deployed on 2012-10-25 22:10:33. All these users are directed to the ACUX_2 interface and the events they generate include a aftv5_cta4 flag in the additional data field of the clicktracking log as well as in the userbuckets field tracked by server events.
- ACUX-2 On 2012-11-01, a patch was applied to ACUX that addresses an issue with IE7 and IE8 users not correctly being assigned to ACUX buckets. The bucket identifiers remain the same, but the version number in the log has been bumped to 5.
- ACUX-3 The benefits did not appear as consistently in this edition of the test, with older versions of IE not supporting the method used to dynamically resize them. This caveat should not directly impact the ability of client-side validation to improve the ux, since that feature worked as expected in IE6 and better.
- ACUX-3 Client-side validation is not perfect, and may produce errors on submit, even though it previously validated as correct. This is due to extensions such as AntiSpoof which don't yet get exposed to the client very well. See bugs 41849 and 40648.
- ACUX-3 Starting on November 16 we observed a large seasonal spike in registrations, at 5,642 total for the day. This was likely due to an extremely successful fundraising test, which resulted in more than 100,000 individual donations. These donors received an invitation to join Wikipedia and edit in a thank-you letter, and this was the largest site-wide change on the day of the spike.
Data collection requirements
[edit]
We will be logging client and server generated events in order to measure the click-through and conversion rates described above. Details on the log format can be found on this page.
Results
[edit]Notes
[edit]In the following cases, registered, logged in editors may unintentionally trigger an event on the account creation page:
- Users bucketed as part of the Account Creation Improvement Project (ACIP), who are taken through multiple signup screens
- Users who are browsing Wikipedia in multiple tabs, log in through one, and then attempt to log in through another (may be clicking "Create an account" to log in, possibly because the login/create account links used to be unified and were only recently split)
- Users entering an invalid password in the signup form