Training Data

  • Review how often your data sources are refreshed
  • Collect live data from users
  • Provide easy access to labels
  • Translate user needs into data needs
  • Only introduce new features when needed
  • Identify your data sources
  • Identify any outliers, and investigate whether they are actual outliers or due to errors in the data
  • Source your data responsibly
  • Design for raters and labeling
  • Split your data
  • Let raters change their minds
  • Evaluate rater tools
  • Consider missing or incomplete data
  • Consider unexpected input
  • Investigate rater context and incentives
  • Articulate your data sources
  • Beware of confirmation bias

Training Procedure

  • Design for experimentation
  • Inspect the features possible values, units, and data types
  • Evaluate the reward function outcomes
  • Weigh false positive & negative
  • Consider precision and recall tradeoffs
  • Balance underfitting and overfitting
  • Tune your model
  • Map existing workflows
  • Design and evaluate the reward function
  • Design for model tuning