Huijuan Xu

Postdoc scholar
EECS, UC Berkeley
Email: huijuan@eecs.berkeley.edu
hxu@bu.edu

About Me

I am a postdoctoral scholar in the EECS department at UC Berkeley advised by Prof. Trevor Darrell. I received my PhD degree from the computer science department at Boston University in 2018 advised by Prof. Kate Saenko. I interned at Disney Research, Pittsburgh with Prof. Leonid Sigal. Before coming to US, I received a Master’s degree from Graduate University of Chinese Academy of Sciences in 2012, advised by Professor Hua Yu, and a Bachelor’s degree in computer science from Hefei University of Technology in 2009.

My research focuses on deep learning, computer vision and natural language processing, particularly in the area of action understanding in video. The goal of my research is building robust action understanding algorithms with human level structural knowledge and efficient multi-modal supervision. To achieve this goal, my research touches the following two aspects: compositional action recognition with knowledge reasoning, and multi-modal action detection with less supervision.

News

Code for EM-MIL model is released.

Our paper Auxiliary Task Reweighting for Minimum-data Learning got accepted to NeurIPS 2020.

I am invited to participate in UC Berkeley Rising Star in EECS 2020.

Our work Spatio-Temporal Action Detection with Multi-Object Interaction got accepted to ECCV 2020 The Seventh International Workshop on Egocentric Perception, Interaction and Computing (EPIC) Workshop. (Video spotlight)

Two papers Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning and Learning Canonical Representations for Scene Graph to Image Generation got accepted to ECCV 2020.

Our work Revisiting Few-shot Activity Detection with Class Similarity Control got accepted to CVPR 2020 Visual Learning with Limited Labels Workshop. (Video spotlight)

Our paper Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks got accepted to CVPR 2020.

Selected Publications

LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval

Reuben Tan, Huijuan Xu, Kate Saenko, Bryan A. Plummer
IEEE Winter Conference on Applications of Computer Vision (WACV), 2021
Paper Code

Auxiliary Task Reweighting for Minimum-data Learning

Baifeng Shi, Judy Hoffman, Kate Saenko, Trevor Darrell, Huijuan Xu
Neural Information Processing Systems (NeurIPS), 2020
Paper Code Project page

Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning

Zhekun Luo, Devin Guillory, Baifeng Shi, Wei Ke, Fang Wan, Trevor Darrell, Huijuan Xu
European Conference on Computer Vision (ECCV), 2020
Paper Code Demo1 Demo2

Learning Canonical Representations for Scene Graph to Image Generation

Roei Herzig*, Amir Bar*, Huijuan Xu, Gal Chechik, Trevor Darrell, Amir Globerson
European Conference on Computer Vision (ECCV), 2020
Paper Code Project page

Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks

Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu†, Xiaolong Wang†, Trevor Darrell†
Computer Vision and Pattern Recognition (CVPR), 2020
Paper Dataset Project page

TwoStreamVAN: Improving Motion Modeling in Video Generation

Ximeng Sun, Huijuan Xu, Kate Saenko
IEEE Winter Conference on Applications of Computer Vision (WACV), 2020
Paper Code Demo

Spatio-Temporal Action Graph Networks

Roei Herzig*, Elad Levi*, Huijuan Xu*, Hang Gao, Eli Brosh, Xiaolong Wang, Amir Globerson, Trevor Darrell
International Conference on Computer Vision Workshop (ICCVW), 2019
Paper Code

Learning Instance Activation Maps for Weakly Supervised Instance Segmentation

Yi Zhu, Yanzhao Zhou, Huijuan Xu, Qixiang Ye, David Doermann, Jianbin Jiao
Computer Vision and Pattern Recognition (CVPR), 2019
Paper Project page

Two-Stream Region Convolutional 3D Network for Temporal Activity Detection

Huijuan Xu, Abir Das, Kate Saenko
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019
Paper Code

Multilevel Language and Vision Integration for Text-to-Clip Retrieval

Huijuan Xu, Kun He, Bryan A. Plummer, Leonid Sigal, Stan Sclaroff, Kate Saenko
Thirty-Third AAAI Conference on Artificial Intelligence (AAAI), 2019
Paper Code Demo1 Demo2

Joint Event Detection and Description in Continuous Video Streams

Huijuan Xu, Boyang Li, Vasili Ramanishka, Leonid Sigal, Kate Saenko
IEEE Winter Conference on Applications of Computer Vision (WACV), 2019
Paper Code Demo1 Demo2

R-C3D: Region Convolutional 3D Network for Temporal Activity Detection

Huijuan Xu, Abir Das, Kate Saenko
International Conference on Computer Vision (ICCV), 2017
Paper Code Demo Project page

Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering

Huijuan Xu, Kate Saenko
European Conference on Computer Vision (ECCV), 2016
Paper Code Video spotlight

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko
North American Chapter of the Association for Computational Linguistics (NAACL), 2015
Paper Project page

Workshop Papers

Huijuan Xu, Lizhi Yang, Stan Sclaroff, Kate Saenko, and Trevor Darrell. Spatio-Temporal Action Detection with Multi-Object Interaction. The Seventh International Workshop on Egocentric Perception, Interaction and Computing (EPIC) at ECCV, 2020. (Extended abstract)
Huijuan Xu, Ximeng Sun, Eric Tzeng, Abir Das, Kate Saenko, and Trevor Darrell. Revisiting Few-Shot Activity Detection with Class Similarity Control. Visual Learning with Limited Labels (VL3) Workshop at CVPR, 2020. (Extended abstract)
Huijuan Xu, Boyang Li, Vasili Ramanishka, Leonid Sigal and Kate Saenko. Joint Event Detection and Description in Continuous Video Streams. Human Activity Detection in Multi-Camera, Continuous, Long-Duration Video (HADCV) Workshop at WACV, 2019. (Extended abstract)
Huijuan Xu, Abir Das, and Kate Saenko. R-C3D: Region Convolutional 3D Network for Temporal Activity Detection. ActivityNet Large Scale Activity Recognition Challenge at CVPR, 2017.
Huijuan Xu, Abir Das, and Kate Saenko. R-C3D: Region Convolutional 3D Network for Temporal Activity Detection. Charades Activity Challenge at CVPR, 2017.
Huijuan Xu and Kate Saenko. Dual Attention Network for Visual Question Answering. 2nd Workshop on Storytelling with Images and Videos (VisStory) at ECCV, 2016. (Extended abstract)
Huijuan Xu and Kate Saenko. Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering. Visual Question Answering (VQA) Challenge at CVPR 2016. (Extended abstract)
Huijuan Xu, Subhashini Venugopalan, Vasili Ramanishka, Marcus Rohrbach and Kate Saenko. A Multi-scale Multiple Instance Video Description Network. First workshop on Closing the Loop Between Vision and Language (CLVL) at ICCV, 2015. (Extended abstract)

Teaching

Teaching Assistant (Spring 2015)
91.422/545: Machine Learning

Lab Instructor (Fall 2013, Spring 2014)
91.103: Computing I Lab

Research Internship

Disney Research, Pittsburgh, Summer 2017

Professional Activities

Reviewer for journals:

Transactions on Pattern Analysis and Machine Intelligence (TPAMI), International Journal of Computer Vision (IJCV), ACM Transactions on Multimedia Activities Computing Communications and Applications (TOMM), IEEE Transactions on Image Processing (TIP), Pattern Recognition (PR), IEEE/ASME Transactions on Mechatronics

Reviewer/program committee member for conferences:

ICCV2017, CVPR2018, ECCV2018, ACCV2018, CVPR2019, NAACL-HLT2019, IJCAI2019, ICCV2019, AAAI2020, CVPR2020, ECCV2020, IJCAI2020, ICML2020, CVPR2021, EMNLP2021, NeurIPS2021

Reviewer/program committee member for workshops:

WiCV2018, WiCV2019, HADCV'20, DL-HAU2021