Local Descriptors Optimized for Average Precision

Kun He, Yan Lu, Stan Sclaroff


Extraction of local feature descriptors is a vital stage in numerous computer vision pipelines. We improve the learning of local feature descriptors by optimizing the performance of descriptor matching, which is a common stage that follows descriptor extraction in local feature based pipelines, and can be formulated as nearest neighbor retrieval. Specifically, we directly optimize a ranking-based retrieval performance metric, Average Precision, using deep neural networks. This general-purpose solution can also be viewed as a "learning to rank" approach with a listwise loss, which is advantageous compared to recent local ranking approaches.


Kun He, Yan Lu, and Stan Sclaroff.
"Local Descriptors Optimized for Average Precision,"
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[PDF] [arXiv] [Poster]

Please cite this paper as:

    title={Local Descriptors Optimized for Average Precision},
    author={He, Kun and Lu, Yan and Sclaroff, Stan},
    booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},


Pretrained models

We provide pretrained models on the benchmarks used in the paper.

Our implementation is based on MatConvNet, and the models are in DagNN format. We use the L2-Net architecture proposed in [2]. Models with suffix "_ST" are augmented with the Spatial Transformer module [3].

Real-valued (128-d with L2-normalization):
HPatches: HPatches_ST_LM_128d.mat
UBC-Liberty: Liberty_ST_128d.mat
RomePatches: RomePatches_128d.mat

Binary (256-bit):
UBC-Liberty: Liberty_ST_256b.mat
RomePatches: RomePatches_256b.mat

HPatches results

We provide original .csv files corresponding to the results reported on the HPatches benchmark [4] in the paper.
verification.csv retrieval.csv matching.csv

Note: all results are obtained on the test set of the "a" split.



Coming soon.

Code for CVPR'18 "Tie-Aware Hashing" paper

Our AP optimization builds on the techniques developed in [1], which solves the "supervised hashing" problem for image retrieval. Code for that paper is here: https://github.com/kunhe/TALR


[1] Kun He, Fatih Cakir, Sarah Adel Bargal, and Stan Sclaroff. "Hashing as Tie-Aware Learning to Rank," IEEE CVPR 2018

[2] Yurun Tian, Bin Fan, and Fuchao Wu. "L2-Net: Deep Learning of Discriminative Patch Descriptor in Euclidean Space," IEEE CVPR 2017

[3] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. "Spatial Transformer Networks," NIPS 2015

[4] Vassileios Balntas*, Karel Lenc*, Andrea Vedaldi and Krystian Mikolajczyk. "HPatches: A benchmark and evaluation of handcrafted and learned local descriptors," IEEE CVPR 2017


For questions/comments, please contact:
Kun He, hekun@bu.edu