Seonglae ChoAccuracy is Not Enough and does not necessarily improve user satisfaction. Is it important to think about evaluation in a holistic manner
Model evaluation
Behavioral metrics:
- Clicks, Scroll, Zoom, Swipe, Likes, Shares
- Dwell time, session length, Inactivity, Transitions
- Gestures, eye tracking
Feedback
- Item Level Feedback
- Page Level Feedback
- Session Level Feedback
- Intra-Session Feedback (loyalty / Cohort Retention )
Feedback Types
- Explicit Feedback - annotator & survey
- Implicit Feedback - understand user engagement from behavioral metrics
Final: Satisfaction Prediction (SAT Prediction)
Recommender Satisfaction Considerations
NeurIPS 2020 : (Track2) Beyond Accuracy: Grounding Evaluation Metrics for Human-Machine Learning Systems
Please be aware that videos might show the previous tutorial in the same track.
https://neurips.cc/virtual/2020/protected/tutorial_a9588aa82388c0579d8f74b4d02b895f.html