Xianyu Chen

Ph.D. dissertation · University of Minnesota · 2025

Building Human-Like Machine Intelligence: Advancing Attention by Modeling, Alignment, and Explainability

2026 Distinguished Dissertation Award · University of Minnesota

One of two recipients university-wide · University nominee for the CGS/ProQuest Distinguished Dissertation Award

Xianyu Chen · Advisor: Prof. Catherine Qi Zhao

Summary

AI systems can describe images and answer questions about them, yet they still struggle to reason step by step through real-world problems and to justify their answers. This dissertation uses human visual attention (where people look, and in what order) as a bridge between perception and reasoning, in three parts: Modeling, Alignment and Explainability.

Four figures from the dissertation's chapters: VQA scanpaths, individual observers' scanpaths, an illustrated VisualHow problem, and fixation explanations.
Figures from Chapters 2, 3, 4 and 6.

Contents

Chapters 2–6 appeared as the papers below.

  1. I. Modeling

    Predict how people move their eyes during a task, and how each person differs.

    1. Ch. 2Predicting Human Scanpaths in Visual Question AnsweringCVPR 2021
    2. Ch. 3Beyond Average: Individualized Visual Scanpath PredictionCVPR 2024
  2. II. Alignment

    Align machine attention with human attention, improving step-by-step problem solving.

    1. Ch. 4VisualHow: Multimodal Problem SolvingCVPR 2022
    2. Ch. 5Every Problem, Every Step, All in FocusTPAMI 2024
  3. III. Explainability

    Predict where people look and explain in plain language what each fixation looks at.

    1. Ch. 6GazeXplain: Learning to Predict Natural Language Explanations of Visual ScanpathsECCV 2024 · Oral

Cite

@phdthesis{chen2025building,
  title  = {Building Human-Like Machine Intelligence: Advancing Attention by Modeling, Alignment, and Explainability},
  author = {Chen, Xianyu},
  school = {University of Minnesota},
  year   = {2025},
  month  = nov,
  url    = {https://hdl.handle.net/11299/279218}
}