Abstract

For many recognition problems, state-of-the-art solutions build on foundational models that are trained on huge datasets. Performance of such methods on standad tests in areas like visual recognition, retrieval and multimodal vision-language tasks is impressive, steadily improving over time. However, benchmaking methods that have seen, in the extreme cases, nearly all of the internet is difficult, for at least two reasons. First, ensuring zero overlap between evaluation and training data becomes challenging. Second, as the performance gets comparable to the level of errors in the "ground truth", improvements of the methods may in fact by just overfitting to paticular benchmarks. In the talk, I will present our efforts to provide quality data for reliable benchmaking. In the case of instance image retrieval, a novel approach to data collection addressing the issues will be presented. For visual recognition, I will focus on the very widely used ImageNet, discuss its flaws and describe our approach to fixing them. The new benchmarks support interesting conclusions about the behaviour, strengths and weaknesses of deep nets, VLMs and LLMs.

Biography

Jiri Matas is the head of the Visual Recognition Group at the Center for Machine Perception, Department of Cybernetics, Czech Technical University in Prague. He holds a PhD degree from the University of Surrey, UK (1995). He has published more than 300 papers that have been cited more than 80000 in Google Scholar (h-index = 103). He received the best paper prize at the British Machine Vision Conference in 2002, 2005 and 2022, at the Asian Conference on Computer Vision in 2007 and at Int. Conf. on Document analysis and Recognition in 2015. J. Matas served as a programme or general chair at ECCV 2004, 2016, 2022 and CVPR 2007 and 2022. He was an Editor-in-Chief of the International Journal of Computer Vision was an Associate Editor-in-Chief of IEEE T. Pattern Analysis and Machine Intelligence. He has been on the computer science panel of the ERC. His research interests include visual tracking, object recognition, image matching and retrieval, sequential pattern recognition, and RANSAC-type optimization metods. He has co-founded two companies, Eyedea Recognition (computer vision) and Locksley (combinatorial optimization).