Channel Agent

Challenge

As the product was released to Ring 0 (the first group of users), the purpose of this research was to evaluate how it performed in a live environment and identify priorities for before and after the next release.

Key research questions included:

• Would users understand the value of AI-generated Status Reports?
• How might users incorporate the experience into their existing workflows?
• What would build or reduce trust in an AI assistant operating on team knowledge?
• What additional capabilities would create the most value in future iterations?

Role

I worked alongside the lead researcher for Channel Agent. I led the study but we split the sessions and collaborated on synthesis.

Process

I combined cross-functional workshops and qualitative beta research to understand usability challenges, adoption barriers, and opportunities to improve the AI experience.

Alignment Workshop

I facilitated an end-to-end workshop with Product, Design, and Engineering partners to evaluate launch readiness and uncover risks that may not have been identified through AI-as-a-judge evaluation alone.

The workshop helped the team:

• Align on launch readiness risks
• Identify usability issues before release
• Generate research questions for beta validation
• Prioritize quick wins to improve the experience

AI Experience Evaluation Framework

Using a metric framework for pilot research on AI features was standard practice at Microsoft as a structured way to assess user sentiment and across success criteria and assess launch readiness. To this end, Teams across Microsoft measured similar concepts in different ways (e.g. accuracy, time saved, etc.).

I consolidated a couple of metric frameworks, wanting to reuse where possible for consistency and comparability. These included metrics such as accuracy, relevance, satisfaction, trustworthiness, etc.

I then created product-specific measures based on project objectives and user goals (gleaned from previous generative research), including status report utilization, level of customization required, sentiment about proactivity level, etc.

I included these as questions on a survey which we administered as part of our beta research.

Beta Research

For the qualitative feedback, I conducted onboarding sessions with early users to capture the first use experience in real time, as well as follow-up longitudinal interviews to get feedback after some usage.

We explored:

• Discoverability and onboarding
• Mental models for interacting with AI
• Trust and permissions
• Collaboration dynamics
• Collecting survey responses for our afore mentioned metric framework
• Future workflow opportunities

As with any non-deterministic experience, we needed to evaluate this experience in a live environment. This usually brings recruitment challenges, but as this also involved group usage, it was even more challenging. Given these challenges, we conducted and analyzed sessions on a rolling basis.

Insights

1. Users needed clearer guidance for interacting with AI

New users required guidance at almost every step due to usability issues including:

• Unclear entry point and onboarding
• It was unclear how to add a knowledge source, there was a frequently forgotten step during the process and no feedback to indicate whether you had/hadn't added it
• The name was based on the channel but for most people it was in the “General” channel which was vague
• People didn’t know what to put in for the prompt
• The Loop editing experience was disconnected from the prompt experience

2. Trust and transparency influenced adoption

Users had concerns about permissions, knowledge sources, and what information the AI could access within collaborative spaces. They also did not trust the Status Report enough yet to rely on it.

• How it would handle varying access across Channels and knowledge sources (i.e. if it was drawing upon knowledge that only some users had access to)
• If it drew upon everything that was said in the channel and/or meetings, including unrelated or off-the-record chatter
• Who could add to the knowledge base and whether admins could view/manage the knowledge base (which at that time, they could not)

3. Translating AI Workflow to a Group Scenario Presented Challenges

Melding Gen AI workflows in a collaborative context provided some challenges which showed us a need to recalibrate including:

• Individual Gen AI workflow is iterative but the lack of a sandbox/preview meant that it would ping the Reviewers Chat to review every time a change was made
• All changes that were pushed, big or small, were announced to the channel
• Even entering into the pilot and adding the agent was very public and required channel admins to explain to their channel members, which provided some friction with adoption