Publications

NeuralFlix: A Simple While Effective Framewrok of Reconstructing Vivid Videos from Non-invasive Brain Recordings

2025
Jingyuan Sun, Mingxiao Li, Marie-Francine Moens

We propose NeuralFlix, a novel framework that reconstructs dynamic visual experiences from fMRI data using spatial-temporal augmentation and a diffusion model. It addresses challenges like noise and temporal lags, advancing brain activity decoding into video formats.

DM-Align: Leveraging the Power of Natural Language Instructions to Make Changes to Images

2025
Maria Mihaela Trusca, Tinne Tuytelaars and Marie-Francine Moens

Interpretation Modeling: Social Grounding of Sentences by Reasoning over their Implicit Moral Judgments

2025
Liesbeth Allein, Maria Mihaela Trusca and Marie-Francine Moens

Interpretation Modeling: Social Grounding of Sentences by Reasoning over their Implicit Moral Judgments

2024
Allein Liesbeth, Trusca Maria Mihaela, Moens Marie-Francine

This work introduces the interpretation modeling (IM) task to capture multiple plausible interpretations of sentence semantics, focusing on social relations and moral judgments. Results highlight the complexity of IM and its potential to enhance content moderation and refine toxicity filters in online discourse.

ERC Advanced Grant Project CALCULUS: Extending the Boundary of Machine Translation

2024
Jingyuan Sun, Mingxiao Li, Ruben Cartuyvels and Marie-Francine Moens

Leveraging the Syntactic Structure of the Text Prompt to Enhance Object-Attribute Binding in Image Generation

2024
Maria Mihaela Trusca, Wolf Nuyts, Jonathan Thomm, Robert Hönig, Thomas Hofmann, Tinne, Tuytelaars, and Marie-Francine Moens

Learning to Route for Dynamic Adapter composition in Lifelong Language Learning

2024
Vladimir Araujo, Marie-Francine Moensand Tinne Tuytelaars

Improving Language Modeling by Increasing Test-time Planning Compute

2024
Florian Mai, Nathan Cornille and Marie-Francine Moens

Introducing Routing Functions to Vision-Language Parameter-Efficient Fine-Tuning with Low-Rank Bottlenecks

2024
Tingyu Qu, Tinne Tuytelaars and Marie-Francine Moens

Animate Your Motion: Turning Still Images into Dynamic Videos

2024
Mingxiao Li, Bo Wan, Sien Moens and Tinne Tuytelaars

Deep Learning for Brain Encoding and Decoding: Principles, Practices and Beyond

2024
Jingyuan Sun, Weihao Xia, Jixing Li, Shaonan Wang, Jiajun Zhang, A. Cengiz Oztireli and Marie-Francine Moens

Sequence-to-Sequence Spanish Pre-trained Language Models

2024
Vladimir Araujo, Maria Mihaela Trusca, Rodrigo Tufiño and Marie-Francine Moens

Computational Linguistics for Brain Encoding and Decoding: Principles, Practices and Beyond

2024
Jingyuan Sun, Shaonan Wang, Zijiao Chen, Jixing Li and Marie-Francine Moens

Spatial and Temporal Language Understanding: Representation, Reasoning, and Grounding

2024
Parisa Kordjamshidi, Qiang Ning, James Pustejovsky and Marie-Francine Moens

Learning to Plan for Language Modeling from Unlabeled Data

2024
Nathan Cornille, Marie-Francine Moens, Florian Mai

A planning module is introduced for language models, enabling structured text generation through latent plans learned via self-supervision. This external module improves text coherence and can be easily shared for broader use.

Alleviating Exposure Bias in Diffusion Models through Sampling with Shifted Time Steps

2024
Mingxiao Li, Tingyu Qu, Ruicong Yao, Wei Sun, Marie-Francine Moens

We propose Time-Shift Sampler to reduce exposure bias in Diffusion Probabilistic Models (DPM) without retraining. Our method significantly improving performance and FID scores.

Explicitly Representing Syntax Improves Sentence-to-Layout Prediction of Unexpected Situations

2024
Wolf Nuyts, Ruben Cartuyvels, Marie-Francine Moens

We show that current models fail to generalize to unexpected situations when generating 2D visual scenes from text. Syntax improves compositional generalization in this context, if enforced by a learning objective that we propose in order to align the syntax structure with visual output representations.

Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities

December 2023
Jingyuan Sun, Mingxiao Li, Zijiao Chen, Yunhao Zhang, Shaonan Wang, Marie-Francine Moens

Decoding visual stimuli from neural responses recorded by functional Magnetic Resonance Imaging (fMRI) is challenging. To mitigate these challenges, we introduce a two-phase fMRI representation learning framework.

Decoding Realistic Image from Brain Activity with Contrastive Self-supervision and Latent Diffusion

2023
Jingyuan Sun, Mingxiao Li and Marie-Francine Moens

Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations?

2023
Jingyuan Sun and Marie-Francine Moens

Spatiotemporal self-supervised pre-training on satellite imagery improves food insecurity prediction

2023
Ruben Cartuyvels, Tom Fierens, Emiel Coppieters, Marie-Francine Moens, Damien Sileo

We leverage spatiotemporal relations to define effective self-supervised pre-training objectives that allow us to learn informative representations for satellite imagery. These representations lead to improved accuracy on the downstream task of food insecurity prediction.

Investigating the Effect of Causal Mechanism Alignment on Transfer Learning for Few-Shot Video Prediction

2023
Cornille, Nathan; Moens, Marie-Francine; Laenen, Katrien

This work investigates using Causal Representation Learning to improve few-shot transfer learning under sparse distribution shifts. Experiments show that alignment aids in retaining unchanged mechanisms but has mixed effects on few-shot adaptation.

Learning Sentence-Level Representations with Predictive Coding

2023
Vladimir Araujo, Marie-Francine Moens and Alvaro Soto

Probing Neural Language Models for Understanding of Words of Estimative Probability

2023
Damien Sileo and Marie-Francine Moens

Causal Factor Disentanglement for Few-Shot Domain Adaptation in Video Prediction

November 2023
Cornille, Nathan and Sun, Jinguan and Laenen, Katrien and Moens, Marie-Francine

We evaluate whether we can use Causal Factor Disentanglement to isolate parameters that model different causal mechanisms, and subsequently adapt more quickly in response to a Sparse Mechanism Shift.

Tuning In to Neural Encoding: Linking Human Brain and Artificial Supervised Representations of Language

October 2023
Jingyuan Sun, Xiaohan Zhang and Marie-Francine Moens

Linking human brain and supervised ANN representations of the Chinese language.

Investigating Neural Fit Approaches for Sentence Embedding Model Paradigms

October 2023
Helena Balabin, Antonietta Gabriella Liuzzi, Jingyuan Sun, Patrick Dupont, Rik Vandenberghe, Marie-Francine Moens

We analyze the link (i.e., neural fit) between functional MRI data and pre-trained language models using different brain networks, neural fit approaches and sentence modeling paradigms.

What Can We Learn from the Structures Found in Visual and Language Data and their Correlations?

End of 2023
Vitor Milewski, Maria Trusca, Marie-Francine

We explore structures and their rules in differnet modalities and compare them to make a proposal for future research directions.

Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations?

August 2023
Jingyuan Sun and Marie-Francine Moens

Investiging various supervised method and the correlation to how brains represent language.

Simulating Task-Free Continual Learning Streams From Existing Datasets

June 2023
Aristotelis Chrysakis and Marie-Francine Moens

A Memory Model for Question Answering from Streaming Data Supported by Rehearsal and Anticipation of Coreference Information

July 2023
Vladimir Araujo, Alvaro Soto, Marie-Francine Moens

Drawing inspiration from human mechanisms, we propose a memory model that performs rehearsal and anticipation while processing inputs to memorize important information for solving question answering tasks from streaming data.

Implicit Temporal Reasoning for Evidence-Based Fact-Checking

May 2023
Liesbeth Allein, Marlon Saelens, Ruben Cartuyvels, and Marie-Francine Moens

Shows that time positively influences the claim verification process of evidence-based fact-checking.

Online Bias Correction for Task-Free Continual Learning

May 2023
Aristotelis Chrysakis and Marie-Francine Moens

We explain both theoretically and empirically how experience replay biases the outputs of the model towards recent stream observations.

Layout-aware Dreamer for Embodied Visual Referring Expression Grounding

February 2023
Li, Mingxiao and Wang, Zehao and Tuytelaars, Tinne and Moens, Marie-Francine

We have designed an autonomous agent called Layout-aware Dreamer (LAD) including two novel modules, the Layout Learner and the Goal Dreamer, to mimic a humans cognitive decision process

Learning Sentence-Level Representations with Predictive Coding

January 2023
Araujo, Vladimir and Moens, Marie-Francine and Soto, Alvaro

This work explores how to improve sentence-level representations of pre-trained models by borrowing ideas from predictive coding theory

When Do Discourse Markers Affect Computational Sentence Understanding?

2022
Li Ruiqi, Allein Liesbeth, Sileo Damien, Moens Marie-Francine

This chapter examines how NLP systems process discourse connectives, assessing nine popular models' comprehension and the impact of context and tasks. Results show disparities in processing connectives, highlighting the importance of explicit connective comprehension for improving computational language understanding.

Evaluation Benchmarks for Spanish Sentence Representations

June 2022
Vladimir Araujo, Andrés Carvallo, Souvik Kundu, José Cañete, Marcelo Mendoza, Robert E. Mercer, Felipe Bravo-Marquez, Marie-Francine Moens, Alvaro Soto

A new benchmark for spanish sentence representations

Finding Structural Knowledge in Multimodal-BERT

May 2022
Milewski, Victor and de Lhoneux, Miryam and Moens, Marie-Francine

we introduce scene trees, by mapping the linguistic dependency tree ontop of regions, to investigate if BERT learns structures over the image regions.

Entropy-based Stability-Plasticity for Lifelong Learning

November 2022
Vladimir Araujo, Julio Hurtado, Alvaro Soto, and Marie-Francine Moens

A novel method called Entropy-based Stability-Plasticity is introduced to address the stability-plasticity dilemma in neural networks.

Critical Analysis of Deconfounded Pretraining to Improve Visio-Linguistic Models

March 2022
Cornille, Nathan and Laenen, Katrien and Moens, Marie-Francine

We critically analyze a recent technique that uses the toolbox of causality to improve on OOD performance, elucidating to what extent it actually finds confounders, under what assumptions it performs deconfounding, and whether the reported OOD performance is actually linked to the causal tools.

How Relevant is Selective Memory Population in Lifelong Language Learning?

November 2022
Vladimir Araujo, Helena Balabin, Julio Hurtado, Alvaro Soto, and Marie-Francine Moens

by investigate relevance of selective memory population in the lifelong learning for language, methods that randomly store a uniform number of samples lead to high performances

A Brief Overview of Universal Sentence Representation Methods: A Linguistic View.

January 2023
Li, Ruiqi and Moens, Marie-Francine

Accepted for upcomming issue!

Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations

November 2021
Araujo, Vladimir and Villa, Andres and Mendoza, Marcelo and Moens, Marie-Francine and Soto, Alvaro

We propose to use ideas from predictive coding theory to augment BERT-style language models with a mechanism that allows them to learn suitable discourse-level representations.

Giving Commands to a Self-Driving Car: How to Deal with Uncertain Situations?

May 2021
Deruyterre, Thierry and Milewski, Victor and Moens, Marie-Francine

When a command is given to a self-driving cars, this can cause ambiguous solutions. A method to solve this through visual and textual means is proposed.

Visual Grounding Strategies for Text-Only Natural Language Processing

April 2021
Sileo, Damien

Conception, categorization and strategies to leverage multimodal pretraining for text-only tasks

How Do Simple Transformations of Text and Image Features Impact Cosine-based Semantic Match

April 2021
Collell, Guillem and Moens, Marie-Francine

We investigate the impact of transformations on semantic distances between embeddings produced by common language models and image CNNs.

Discrete and continuous representations and processing in deep learning: Looking forward

January 2021
Ruben Cartuyvels, Graham Spinks, Marie-Francine Moens

A position paper that reflects on the role of discrete and continuous representations and processing in the deep learning era.

Improving representation learning with Pervasive Internal Regression (PIR)

November 2020
Cornille, Nathan, Moens, Marie-Francine

PIR is an unsupervised learning method inspired by predictive coding in the brain, where models predict both input and future internal activations to learn representations, avoiding trivial solutions with a contrastive loss.

Are Scene Graphs Good Enough to Improve Image Captioning?

december 2020
Milewski, Victor and Moens, Marie-Francine and Calixto, Iacer

To better describe relations in captions, several studies propose to use scene graphs. We develop and analyse methods for using these graphs and find that they are currently too noisy.

Autoregressive Reasoning over Chains of Facts with Transformers

December 2020
Ruben Cartuyvels, Graham Spinks, Marie-Francine Moens

An iterative inference algorithm for multi-hop explanation regeneration, that retrieves relevant factual evidence in the form of text snippets, given a natural language question and its answer.

Convolutional Generation of Textured 3D Meshes.

December 2020
Pavllo, Dario, Graham Spinks, Thomas Hofmann, Marie-Francine Moens, and Aurélien Lucchi

Generating 3D images with 2D supervision by conditioning the model on class labels, attributes, and text.

Decoding Language Spatial Relations to 2D Spatial Arrangements

November 2020
Radevski, Gorjan and Collell, Guillem and Moens, Marie-Francine and Tuytelaars, Tinne

We propose Spatial-Reasoning Bert (SR-Bert) for the problem of multimodal spatial understanding by decoding a set of language-expressed spatial relations to a set of 2D spatial arrangements in a multi-object and multi-relationship setting.

Improving Language Understanding in Machines through Anticipation.

November 2020
Cornille, Nathan and Collel, Guillem and Moens, Marie-Francine

Poster that reflects on some of the issues with an internal contrastive objective that aims to improve representation learning.

Learning Grammar in Confined Worlds

October 2020
Spinks, Graham and Cartuyvels, Ruben and Moens, Marie-Francine

In this position paper we argue that modern machine learning approaches fail to adequately address how grammar and common sense should be learned.We advocate for experiments with the use of abstract, confined world environments where agents interact with the emphasis on learning world models.

Structured (De)composable Representations Trained with Neural Networks

October 2020
Spinks, Graham and Moens, Marie-Francine

End-to-end deep learning technique to learn structured and composable representations.

Towards Extracting Absolute Event Timelines from English Clinical Reports

december 2020
Leeuwenberg, Tuur and Moens, Marie-Francine

An approach towards extraction of more complete temporal information for all events, and obtain probabilistic absolute event timelines by modeling temporal uncertainty with information bounds.

Structured (De) composable Representations Trained with Neural Networks

September 2020
Spinks, Graham and Moens, Marie-Francine

Novel technique for representing templates and instances of concept classes. The technique learns structured and composable representations from input images and discrete labels.

Online Continual Learning from Imbalanced Data

July 2020
Chrysakis, Aristotelis and Moens, Marie-Francine

Improves online continual learning performance in imbalanced settings by extending reservoir sampling.

A Survey on Temporal Reasoning for Temporal Information Extraction from Text (Extended Abstract)

July 2020
Leeuwenberg, Artuur and Moens, Marie-Francine

This article presents a comprehensive survey of the research from the past decades on temporal reasoning for automatic temporal information extraction from text, providing a case study on the integration of symbolic reasoning with machine learning-based information extraction systems.

Giving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding

February 2020
Deruyttere, Thierry and Collell, Guillem and Moens, Marie-Francine

A new spatial memory module and a spatial reasoner for the Visual Grounding task. We focus on integrating the regions of a Region Proposal Network into a new multi-step reasoning model.

Improving Natural Language Understanding through Anticipation-Enriched Representations.

December 2019
Cornille, Nathan and Moens, Marie-Francine

Poster with first idea for internal-self-prediction objective for BERT, presented at Human Brain Project workshop in Glasgow.

A survey on temporal reasoning for temporal information extraction from text

September 2019
Leeuwenberg, Artuur and Moens, Marie-Francine

This article presents a comprehensive survey of the research from the past decades on temporal reasoning for automatic temporal information extraction from text, providing a case study on the integration of symbolic reasoning with machine learning-based information extraction systems.

Justifying diagnosis decisions by deep neural networks

August 2019
Spinks, Graham and Moens, Marie-Francine

A holistic approach to justification for neural networks outperforms saliency maps. Visualizing the nearest alternative diagnosis is a powerful, novel approach. Creating a continuous textual representation is useful to bridge modalities.