Setting the STAGE for MSN Apps
The VP needed to know which of the 7 MSN apps would be competitive on iOS and Android. Here’s how I stopped two from shipping.
TL;DR
To evaluate 28 distinct cross-platform app experiences on a tight timeline, I designed a hybrid methodology (“STAGE”) combining solo trials with group evaluation across 85 participants. The data empowered leadership to halt two uncompetitive apps before launch, focusing engineering efforts on the contenders that went on to earn sustained 4-to-5-star ratings.
Project
Cross-platform viability assessment for Microsoft’s MSN-branded content app suite
People
My role: UX Researcher, responsible for planning, execution, and reporting
Collaborators: The app Designers, who helped co-moderate these large sessions
Primary stakeholders: VP of MSN; Leads for each app
Context & Problem
Microsoft was porting seven core apps (News, Weather, Sports, Travel, Money, Health & Fitness, Food & Drink) to iOS and Android on both phone and tablet form factors. On Windows, these apps had the massive advantage of being pre-installed. In external app stores, they would have to fight for survival.
The VP gave me just a few weeks to answer a daunting, high-stakes question: across 28 distinct app experiences (7 apps × 2 OSs × 2 device types), which were worth the continued engineering investment, and which should be cut before launch?
Approach
Standard usability testing was too slow for this scale, so I designed a custom hybrid methodology I named “STAGE” (Solo Trial and Group Evaluation), which I had evolved from my experience with multiplayer video game playtesting methods.
In just under two weeks of execution time, I ran 14 cohorts totaling 85 participants, targeting each app category and preferred platform. Participants first evaluated the apps individually on demo devices to prevent groupthink, providing feedback on a one-page feedback form and rating how likely they were to replace their current daily-driver apps with ours. We then transitioned into a focus group to dig into the “why.” Having the Designers help moderate this part was very valuable, as it gave them a chance to ask the participants questions about their designs firsthand. By having each OS cohort evaluate both phone and tablet versions, I drastically cut down recruiting time while maintaining high-quality, segmented feedback.
Please explore the app and try to do tasks you might normally do in your current weather app.
Here are some other tasks you may try:
- Check the extended forecast.
- Search for the weather in your hometown or favorite vacation spot.
- Add a location to your favorites.
- Change Fahrenheit to Celsius.
Key Insights
Two clear at-risk apps emerged, with confidence strengthened by cross-referencing this rapid qualitative feedback with our historical research data.
Despite being intuitive and well-designed, the Weather app failed the “replacement test.” It’s usable and pretty, but users were perfectly happy with their native iOS/Android weather apps and saw no reason to seek out and download a new one. Meanwhile, the Travel app simply lacked the robust booking features required to pull users away from established giants like Kayak or TripAdvisor.
Impact
The Strongest Survived: In two focused slides, I delivered a clear viability summary to the VP, empowering him to halt the launch of the uncompetitive apps and reprioritize engineering resources onto the strongest contenders. This ruthless prioritization paid off: the Microsoft apps that ultimately shipped to iOS and Android maintained an average 4-to-5-star rating throughout their critical first year on the market.
NDAs are serious business. While I can’t share proprietary deliverables, real participant photos, or raw data, I’ve designed this portfolio to highlight the strategic thinking and impact of the work.