An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendation
Do standard recommendation benchmarks genuinely require sophisticated models, or can their structure reward simpler shortcuts? This study investigates that question with a graph heuristic that starts from a user’s last one or two interactions, retrieves nearby candidate items and ranks them by feature similarity. Across fourteen datasets, the authors report that the heuristic remains competitive on ten, with model rankings changing as dataset properties change. They examine local transition patterns, similarity between successive items and the value of longer interaction histories as possible explanations. The accompanying critique distinguishes those observed associations from causal evidence: the study compares datasets rather than removing a shortcut within a fixed benchmark. It also notes that the apparently simple method uses representations from a pretrained text encoder and that comparisons depend on the authors’ implementations and chosen settings. The paper is useful for understanding why a strong leaderboard result does not, by itself, establish the advanced capability a benchmark is intended to test.