Publications

Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities

December 2023
Jingyuan Sun, Mingxiao Li, Zijiao Chen, Yunhao Zhang, Shaonan Wang, Marie-Francine Moens

Decoding visual stimuli from neural responses recorded by functional Magnetic Resonance Imaging (fMRI) is challenging. To mitigate these challenges, we introduce a two-phase fMRI representation learning framework.

Causal Factor Disentanglement for Few-Shot Domain Adaptation in Video Prediction

November 2023
Cornille, Nathan and Sun, Jinguan and Laenen, Katrien and Moens, Marie-Francine

We evaluate whether we can use Causal Factor Disentanglement to isolate parameters that model different causal mechanisms, and subsequently adapt more quickly in response to a Sparse Mechanism Shift.

Tuning In to Neural Encoding: Linking Human Brain and Artificial Supervised Representations of Language

October 2023
Jingyuan Sun, Xiaohan Zhang and Marie-Francine Moens

Linking human brain and supervised ANN representations of the Chinese language.

Investigating Neural Fit Approaches for Sentence Embedding Model Paradigms

October 2023
Helena Balabin, Antonietta Gabriella Liuzzi, Jingyuan Sun, Patrick Dupont, Rik Vandenberghe, Marie-Francine Moens

We analyze the link (i.e., neural fit) between functional MRI data and pre-trained language models using different brain networks, neural fit approaches and sentence modeling paradigms.

What Can We Learn from the Structures Found in Visual and Language Data and their Correlations?

End of 2023
Vitor Milewski, Maria Trusca, Marie-Francine

We explore structures and their rules in differnet modalities and compare them to make a proposal for future research directions.

Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations?

August 2023
Jingyuan Sun and Marie-Francine Moens

Investiging various supervised method and the correlation to how brains represent language.

Simulating Task-Free Continual Learning Streams From Existing Datasets

June 2023
Aristotelis Chrysakis and Marie-Francine Moens

A Memory Model for Question Answering from Streaming Data Supported by Rehearsal and Anticipation of Coreference Information

July 2023
Vladimir Araujo, Alvaro Soto, Marie-Francine Moens

Drawing inspiration from human mechanisms, we propose a memory model that performs rehearsal and anticipation while processing inputs to memorize important information for solving question answering tasks from streaming data.

Implicit Temporal Reasoning for Evidence-Based Fact-Checking

May 2023
Liesbeth Allein, Marlon Saelens, Ruben Cartuyvels, and Marie-Francine Moens

Shows that time positively influences the claim verification process of evidence-based fact-checking.

Online Bias Correction for Task-Free Continual Learning

May 2023
Aristotelis Chrysakis and Marie-Francine Moens

We explain both theoretically and empirically how experience replay biases the outputs of the model towards recent stream observations.

Layout-aware Dreamer for Embodied Visual Referring Expression Grounding

February 2023
Li, Mingxiao and Wang, Zehao and Tuytelaars, Tinne and Moens, Marie-Francine

We have designed an autonomous agent called Layout-aware Dreamer (LAD) including two novel modules, the Layout Learner and the Goal Dreamer, to mimic a humans cognitive decision process

Learning Sentence-Level Representations with Predictive Coding

January 2023
Araujo, Vladimir and Moens, Marie-Francine and Soto, Alvaro

This work explores how to improve sentence-level representations of pre-trained models by borrowing ideas from predictive coding theory

Evaluation Benchmarks for Spanish Sentence Representations

June 2022
Vladimir Araujo, Andrés Carvallo, Souvik Kundu, José Cañete, Marcelo Mendoza, Robert E. Mercer, Felipe Bravo-Marquez, Marie-Francine Moens, Alvaro Soto

A new benchmark for spanish sentence representations

Finding Structural Knowledge in Multimodal-BERT

May 2022
Milewski, Victor and de Lhoneux, Miryam and Moens, Marie-Francine

we introduce scene trees, by mapping the linguistic dependency tree ontop of regions, to investigate if BERT learns structures over the image regions.

Entropy-based Stability-Plasticity for Lifelong Learning

November 2022
Vladimir Araujo, Julio Hurtado, Alvaro Soto, and Marie-Francine Moens

A novel method called Entropy-based Stability-Plasticity is introduced to address the stability-plasticity dilemma in neural networks.

Critical Analysis of Deconfounded Pretraining to Improve Visio-Linguistic Models

March 2022
Cornille, Nathan and Laenen, Katrien and Moens, Marie-Francine

We critically analyze a recent technique that uses the toolbox of causality to improve on OOD performance, elucidating to what extent it actually finds confounders, under what assumptions it performs deconfounding, and whether the reported OOD performance is actually linked to the causal tools.

How Relevant is Selective Memory Population in Lifelong Language Learning?

November 2022
Vladimir Araujo, Helena Balabin, Julio Hurtado, Alvaro Soto, and Marie-Francine Moens

by investigate relevance of selective memory population in the lifelong learning for language, methods that randomly store a uniform number of samples lead to high performances

A Brief Overview of Universal Sentence Representation Methods: A Linguistic View.

January 2023
Li, Ruiqi and Moens, Marie-Francine

Accepted for upcomming issue!

Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations

November 2021
Araujo, Vladimir and Villa, Andres and Mendoza, Marcelo and Moens, Marie-Francine and Soto, Alvaro

We propose to use ideas from predictive coding theory to augment BERT-style language models with a mechanism that allows them to learn suitable discourse-level representations.

Giving Commands to a Self-Driving Car: How to Deal with Uncertain Situations?

May 2021
Deruyterre, Thierry and Milewski, Victor and Moens, Marie-Francine

When a command is given to a self-driving cars, this can cause ambiguous solutions. A method to solve this through visual and textual means is proposed.

Visual Grounding Strategies for Text-Only Natural Language Processing

April 2021
Sileo, Damien

Conception, categorization and strategies to leverage multimodal pretraining for text-only tasks

How Do Simple Transformations of Text and Image Features Impact Cosine-based Semantic Match

April 2021
Collell, Guillem and Moens, Marie-Francine

We investigate the impact of transformations on semantic distances between embeddings produced by common language models and image CNNs.

Discrete and continuous representations and processing in deep learning: Looking forward

January 2021
Ruben Cartuyvels, Graham Spinks, Marie-Francine Moens

A position paper that reflects on the role of discrete and continuous representations and processing in the deep learning era.

Are Scene Graphs Good Enough to Improve Image Captioning?

december 2020
Milewski, Victor and Moens, Marie-Francine and Calixto, Iacer

To better describe relations in captions, several studies propose to use scene graphs. We develop and analyse methods for using these graphs and find that they are currently too noisy.

Autoregressive Reasoning over Chains of Facts with Transformers

December 2020
Ruben Cartuyvels, Graham Spinks, Marie-Francine Moens

An iterative inference algorithm for multi-hop explanation regeneration, that retrieves relevant factual evidence in the form of text snippets, given a natural language question and its answer.

Convolutional Generation of Textured 3D Meshes.

December 2020
Pavllo, Dario, Graham Spinks, Thomas Hofmann, Marie-Francine Moens, and Aurélien Lucchi

Generating 3D images with 2D supervision by conditioning the model on class labels, attributes, and text.

Decoding Language Spatial Relations to 2D Spatial Arrangements

November 2020
Radevski, Gorjan and Collell, Guillem and Moens, Marie-Francine and Tuytelaars, Tinne

We propose Spatial-Reasoning Bert (SR-Bert) for the problem of multimodal spatial understanding by decoding a set of language-expressed spatial relations to a set of 2D spatial arrangements in a multi-object and multi-relationship setting.

Improving Language Understanding in Machines through Anticipation.

November 2020
Cornille, Nathan and Collel, Guillem and Moens, Marie-Francine

Poster that reflects on some of the issues with an internal contrastive objective that aims to improve representation learning.

Learning Grammar in Confined Worlds

October 2020
Spinks, Graham and Cartuyvels, Ruben and Moens, Marie-Francine

In this position paper we argue that modern machine learning approaches fail to adequately address how grammar and common sense should be learned.We advocate for experiments with the use of abstract, confined world environments where agents interact with the emphasis on learning world models.

Structured (De)composable Representations Trained with Neural Networks

October 2020
Spinks, Graham and Moens, Marie-Francine

End-to-end deep learning technique to learn structured and composable representations.

Towards Extracting Absolute Event Timelines from English Clinical Reports

december 2020
Leeuwenberg, Tuur and Moens, Marie-Francine

An approach towards extraction of more complete temporal information for all events, and obtain probabilistic absolute event timelines by modeling temporal uncertainty with information bounds.

Structured (De) composable Representations Trained with Neural Networks

September 2020
Spinks, Graham and Moens, Marie-Francine

Novel technique for representing templates and instances of concept classes. The technique learns structured and composable representations from input images and discrete labels.

Online Continual Learning from Imbalanced Data

July 2020
Chrysakis, Aristotelis and Moens, Marie-Francine

Improves online continual learning performance in imbalanced settings by extending reservoir sampling.

A Survey on Temporal Reasoning for Temporal Information Extraction from Text (Extended Abstract)

July 2020
Leeuwenberg, Artuur and Moens, Marie-Francine

This article presents a comprehensive survey of the research from the past decades on temporal reasoning for automatic temporal information extraction from text, providing a case study on the integration of symbolic reasoning with machine learning-based information extraction systems.

Giving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding

February 2020
Deruyttere, Thierry and Collell, Guillem and Moens, Marie-Francine

A new spatial memory module and a spatial reasoner for the Visual Grounding task. We focus on integrating the regions of a Region Proposal Network into a new multi-step reasoning model.

Improving Natural Language Understanding through Anticipation-Enriched Representations.

December 2019
Cornille, Nathan and Moens, Marie-Francine

Poster with first idea for internal-self-prediction objective for BERT, presented at Human Brain Project workshop in Glasgow.

A survey on temporal reasoning for temporal information extraction from text

September 2019
Leeuwenberg, Artuur and Moens, Marie-Francine

This article presents a comprehensive survey of the research from the past decades on temporal reasoning for automatic temporal information extraction from text, providing a case study on the integration of symbolic reasoning with machine learning-based information extraction systems.

Justifying diagnosis decisions by deep neural networks

August 2019
Spinks, Graham and Moens, Marie-Francine

A holistic approach to justification for neural networks outperforms saliency maps. Visualizing the nearest alternative diagnosis is a powerful, novel approach. Creating a continuous textual representation is useful to bridge modalities.