Translating Latent State World Model Plans into Natural Language

TY - CONF

T1 - Translating Latent State World Model Plans into Natural Language

AU - Barker, Matthew

AU - Leonetti, Matteo

PY - 2025

Y1 - 2025

N2 - Recent successes in model-based reinforcement learning have stemmed from models that learn a latent representation of the world. However, this latent representation is unintelligible. When planning in this latent space, an autonomous agent thus has no way to communicate what it plans to do and what it expects to happen to the world, which poses challenges for developing interpretable agents. For high-stake scenarios, ambiguous goals, or changeable environments, the ability to know what an agent plans to do before it acts is vital: knowing what dosage of medicine a hospital assistive robot is planning on giving a patient; which possessions a household robot tasked to ``clean the room'' is planning to throw away or store; and whether the same household robot plans on bringing you a glass of red wine via your newly re-carpeted room. In this work we take a step towards tackling this challenge and present a method for learning and translating a sequence of world model latent states into a natural language description of what these states mean in terms of an RL agent's interaction with the world. Our method makes an explicit connection between the learned latent state representation and its corresponding natural language description. We demonstrate that our method enables an RL agent to plan in latent space and verbalise, in natural language, what it is that they plan to do and what they expect to observe before acting. We validate our method in the complex Crafter environment, showing that what the agent says it will do and expects to observe corresponds closely to what it actually does and observes. Finally, we demonstrate and discuss how our method can handle stochastic environments, along with future avenues of work.

AB - Recent successes in model-based reinforcement learning have stemmed from models that learn a latent representation of the world. However, this latent representation is unintelligible. When planning in this latent space, an autonomous agent thus has no way to communicate what it plans to do and what it expects to happen to the world, which poses challenges for developing interpretable agents. For high-stake scenarios, ambiguous goals, or changeable environments, the ability to know what an agent plans to do before it acts is vital: knowing what dosage of medicine a hospital assistive robot is planning on giving a patient; which possessions a household robot tasked to ``clean the room'' is planning to throw away or store; and whether the same household robot plans on bringing you a glass of red wine via your newly re-carpeted room. In this work we take a step towards tackling this challenge and present a method for learning and translating a sequence of world model latent states into a natural language description of what these states mean in terms of an RL agent's interaction with the world. Our method makes an explicit connection between the learned latent state representation and its corresponding natural language description. We demonstrate that our method enables an RL agent to plan in latent space and verbalise, in natural language, what it is that they plan to do and what they expect to observe before acting. We validate our method in the complex Crafter environment, showing that what the agent says it will do and expects to observe corresponds closely to what it actually does and observes. Finally, we demonstrate and discuss how our method can handle stochastic environments, along with future avenues of work.

KW - reinforcement learning

KW - world model

KW - planning

KW - interpretable reinforcement learning

M3 - Abstract

SP - 78

EP - 82

T2 - Multi-Disciplinary Conference on Reinforcement Learning and Decision Making 2025

Y2 - 11 June 2025 through 14 June 2025

ER -