Deployment

Errors

Error Types

  • Identify error sources
  • Consider background errors
  • Consider context errors
  • Consider mislabeled or misclassified results
  • Consider poor inference or incorrect model

Error Prevention

  • Account for negative impact
  • Auto-detect and display errors
  • Disambiguate systems hierarchy errors
  • Diagnose errors that users don’t perceive
  • Check output quality for relevance errors
  • Fail gracefully
  • Discover prediction and training data errors
  • Cue the correct interactions
  • Categorize user-perceived errors
  • Provide paths forward from failure
  • Understand the significance of a mistake’s consequences
  • As you work on reducing mistakes in one area, always consider the effect your work has on other areas and overall accuracy
  • When possible, address mistakes without complicating the UI
  • Learn from corrections when it makes sense
  • When possible, use guided corrections instead of freeform corrections
  • Let people correct their corrections
  • Provide immediate value when people make a correction
  • Give people familiar easy ways to make corrections
  • Immediately provide assistance if progress stalls
  • Be especially careful to avoid mistakes in proactive features
  • Support efficient dismissal

Error Handling

  • Assume subversive use
  • Imagine potential pitfalls
  • Gauge the risk for potential errors
  • Identify user, system, and context errors
  • Weigh situational stakes and error risk
  • Avoid compounding errors from other ML models
  • Define “errors” and “failure”
  • Predict or plan for input errors
  • Never rely on corrections to make up for low-quality results
  • Always balance the benefits of a feature with the effort required to make a correction

Feedback

Collecting Feedback

  • Collect explicit feedback.
  • Monitor over time.
  • Allow for opting out.
  • Plan for co-learning.
  • Connect feedback with personalization.
  • Create opportunities for feedback.
  • Provide editability.
  • Be prepared for changes in implicit feedback when you make changes to your app’s UI.
  • Don’t ask for both positive and negative feedback.
  • Make it easy for people to correct frequent or predictable mistakes.
  • Always make providing explicit feedback a voluntary task.
  • Request explicit feedback only when necessary.
  • Consider using explicit feedback to help improve when and where you show results.
  • Remember recent interactions.
  • Encourage granular feedback.
  • Support efficient correction.
  • Learn from user behavior.

Addressing Feedback

  • Review implicit feedback.
  • Adapt to the evolving user journey.
  • Remind, reinforce, and adjust.
  • Communicate value and time to impact.
  • Align feedback with model improvement.
  • Manage influence on user decisions.
  • Connect feedback to user experience changes.
  • When possible, use multiple feedback signals to improve suggestions - and mitigatie mistakes.
  • Prioritize recent feedback.
  • Learn from selections when it makes sense.
  • Update and adapt cautiously.
  • Use feedback to update predictions on a cadence that matches the - user's mental model of the feature.
  • Act immediately when you receive explicit feedback and persist the resulting changes.
  • Don’t let implicit feedback decrease people’s opportunities to - explore.
  • Continuously update your feature to reflect people’s evolving - interests and preferences.