Serge Belongie

What do you research?

I do research in Computer Vision, Machine Learning, and Augmented Reality. One of my focus areas is fine grained visual categorization, e.g. for plants & animals. Another research focus in my lab involves detecting and stopping the spread of misinformation in text, images, audio, and video, e.g. in social media. In both of these endeavors, we use human-in-the-loop approaches, which leverage the complementary strengths of people and machines.

Why is it so important?

The “fine grained” designation refers to categories that require specialized expertise. It’s not about determining whether an image contains a bird or a bicycle, which — using modern AI techniques — is generally considered easy. Instead, we ask, what is the species of the bird? Or what is wrong with this bicycle? To answer these questions, one must tap into human communities with specialized expertise. Discovering solutions to such problems requires humans and machines to work together in a harmonious way. Our work on misinformation detection also taps into the complementary strengths of humans and machines. It goes beyond image recognition into the analysis of narratives, i.e. the message behind the message in a piece of shared content. With the rise of Generative AI techniques such as LLM-based chatbots, users of social media will need ever more powerful tools to detect when something they are reading is, for example, machine-generated propaganda.

A project you are proud of?

As part of the Visipedia project, my group co-developed the AI functionality of Merlin Bird ID and iNaturalist, based on visual and auditory information. These free, publicly available apps have been used by millions of people to identify plants, animals, and fungi everywhere from hiking trails to their own back yards, and in the process, generate a trove of data for the global community of biodiversity researchers.

Updated October 2024

In previous policy fellowships, Serge has worked with Ditte Bjerregaard, Deputy Tech Ambassador in the Danish Ministry of Foreign Affairs (Alumner - Policy Fellowship).