Systematically evaluating the psychological validity of computer-vision models for vision-based conceptual representations

• Applicant: Dr. Fritz Günther
• Subject Area: General, Cognitive and Mathematical Psychology
• Term: from 2020 to 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 452322374

Humans excel in extracting information from the structure of their environment, and can build a conceptual system from this information that helps them navigate the world. The last two decades have seen the rise of successful computational models that emulate this human capacity of pattern learning, mainly in the domain of language. This is because large amounts of language data are readily available in the form of digitized corpora, and because relatively simple algorithms (referred to as distributional semantic models) have been remarkably successful in extracting conceptual information in the form of word meanings – and their similarities – from word distributions in text. However, the human conceptual system is not only informed by language experience, but is also heavily shaped by our perceptual experience with the world around us. For example, we would say that a pizza is more similar to a coin than to a bank note, even though their word meanings are as unrelated. Recent advancements in the field of computer vision now allow us to derive vision-based representations from large collections of images, which in turn allows us to compute quantitative measures of this vision-based similarity, both on the level of individual images and on a conceptual prototype level (by aggregating information over the individual images in a category). Previous studies in which we employed these models have yielded very promising results. However, a systematic evaluation of this vision-based system is still missing, which stands in stark contrast to the language-based research. The present project is aimed at providing this systematic evaluation, in order to establish these vision-based models as a reliable and fruitful resource for psychological (and other) studies. To this end, we plan to investigate (a) the intuitive plausibility of the model-derived visual similarity scores, (b) the similarity between exemplars (individual images) and prototypes, (c) the role of visual similarity in on-line processing under time pressure, and (d) whether the model can measure visual similarity as a purely perceptual phenomenon, without high-level conceptual interference. All investigations will be conducted in the form of large-scale experiments, to establish a solid empirical database and to create high-quality, reliable datasets that can be used as gold standards in future research. In addition, the project aims at creating an easy-to-use web interface, so that no prior technical knowledge is required to access the vision-based models and the measures derived from them (such interfaces have proven to be valuable assets for the language-based models).

• DFG Programme: Research Grants
• International Connection: Italy
• Cooperation Partners: Professor Dr. Marco Marelli; Dr. Marco Alessandro Petilli