Summary
We wanted to cheaply test some big platform changes that would involve features and technology we were yet to build, so we decided to fake them!
💡 Hypothesis - If a client and a worker seem to be a good match (common interests, preferences align, live close to each other etc) and we can facilitate a low-friction meeting between the two, it will result in an ongoing booking relationship.
Could we automate a team build?
We already knew that clients who work with three workers in the first 3 months of joining the platform have + 50% better chance of retention after 12 months. We wanted to test an assumption that we could “automate” a team build for a client which would lead to ongoing relationships with support workers.
We didn’t have any engineering resource available, nor did we have a lot of the infrastructure we thought we might need in order for it to be an experiment we could run on in the production environment. Note: It feels strange to say this, but this experiment was conducted pre-AI. I can only imagine how different it would be to run today.
The vision: building teams
The vision wasn’t based on a hunch, instead we had compelling data to suggest that healthy teams led to prolonged platform retention, and we had designed a process which we had relative confidence in.
Designing a Wizard of Oz experiment
We opted to explore using the The Wizard Of Oz method to validate our assumptions. A PM and myself would act like the platform for thirty days in an attempt to validate a large potentially transformative theory.
I'd love to say we had thoroughly planned the entire Wizard of Oz trial before it started. In truth, it was a roller coaster, encountering surprise after surprise and we were forced to design a lot of it as we went.
Recruiting users
We recruited users via an internal platform notice, as well as an email sent to a cohort of a 130 clients who seemed to have become inactive on the platform inviting them to “trial a new feature”.
We had hoped to get up to 30 clients into the trial, though ended up with only 8. We very quickly learned that we would not have been able to run the experiment with 30 people in the trial. Pretending to be the platform immediately became an incredibly time consuming activity.
Simulating automated platform interactions
Pretending to be “the machine” in a Wizard of Oz experiment ended up being more grueling than anything either of us could imagine. It consumed our days and evenings and dangerously blurred the lines between work and home, yet it felt exhilarating at the same time.
Gathering client data
With no engineering resources available to us, we instead used survey building tools to create production-ready looking interfaces to collect client data.
Posting jobs on behalf of trial participants
We created a dummy user profile called "Hireup", and used this account to post jobs on the client's behalf, ensuring "the platform" would do all the heavy lifting for trial participants.
"Algorithmically" sifting job applications
We designed our own matching algorithm which took a bunch of data points into consideration that Hireup was not yet able to technically incorporate.
We evaluated the potential match by assessing the client’s and the workers’ profile quality. We looked for existing messaging relationships and within those, analysed message sentiment. Did they have an existing booking relationship? How many bookings? When was the last one?
We laid this additional analysis on top of Hireup’s existing matching algorithm, which largely focused on geographic proximity and compatibility of support needs.
Pretending to be a bot when messaging workers
Part of maintaining the illusion of being an automated service meant we needed to be prompt and consistent with our communication over messages, and various scenarios were scripted in advance.
Beep boop, the computer says you match
We also tried to proactively match workers to our trial participants, using our makeshift algorithm to locate appropriate looking workers and approach them as our pretend "bot" account.
Everything seemed okay… until the wheels fell off.
Clients were too slow to respond. We believe 5 out of 7 clients would have been successful if they had responded quickly and effectively to their worker shortlist (71% success rate).
What did we learn?
to do
Outcomes
to do