WP3: Learning of Representations from Language and Visual Data

WP3 focuses on the development of innovative algorithms designed to capture the intricate relational structure of language and the geometric and appearance facets of visual data. The emphasis is on anticipatory representations inspired by how the brain works. The objective is to create more robust and informative representations for both visual and textual data.

In the past, there has been a lot of research on text representation learning methods, and with CALCULUS we add to the discussions in this line of research [6,10], contribute evaluation benchmarks [13], and make algorithmic contributions, especially with regards to anticipatory models inspired by the brain. Specifically, several of our publications are concerned with integrating predictive coding theory into the pretraining of neural networks [1, 4, 9, 14], which aim to make representations predictive of future stimuli. While the former works are concerned with textual stimuli only, we argue that a powerful language representation needs to be competent at predicting visual stimuli, too, and learn to navigate the environment of a confined world [3]. For example, our autonomous agent LAD (Layout-aware Dreamer) [15], imagines the destination of its goal for deciding the next action.

Beyond this work, visual-linguistic representation learning is central to the goals of CALCULUS: A burgeoning area of research focuses on the intersection of text and image data, aiming to ground text representations within the context of visual stimuli. In this paradigm, textual semantics are not just inferred from text but also anchored in associated images, creating a more holistic understanding. The research here aligns precisely with the overarching aim of WP3, which is to establish robust and contextually aware representations from both language and visual data, ultimately resulting in more intelligent and predictive systems. Indeed, we show that learning visually grounded representations improves performance even on tasks that only involve text [7]. Furthermore, we investigate whether multimodal representations have learned visual structures analogous to linguisic structures [12], discover the underlying causal structure of the data [11], and learn to map verbal descriptions of objects’ spatial relationships to the image [5], with practical applications such as self-driving [2,8].

A third research path in the context of anticipatory, brain-inspired context is closely aligned with neuroscientific methods and insights to explore the connection between human brain activity and machine-learned language representations. These works involve neural encoding techniques to measure how well the activations in artificial neural networks correspond with human neural processes during language processing. Various types of sentence embedding models and fine-tuning approaches are being investigated for their effectiveness in replicating patterns found in human brain activity [16, 17, 18].