GRASP: Improving Long-Horizon Planning with Learned World Models

Planning over long time horizons using learned world models has been a persistent challenge in AI and robotics. While large, learned world models have become increasingly capable at predicting future states, using them effectively for control and planning remains difficult. A new approach called GRASP (Gradient-based Planning for World Models at Longer Horizons) aims to address these challenges and make long-horizon planning more practical.

The Promise and Pitfalls of World Models

World models are learned dynamics models that can predict future states given the current state and a sequence of actions. Formally, they define a predictive distribution:

$P_\theta(s_{t+1} \mid s_{t-h:t},\; a_t)$

Where $s_t$ represents the state (e.g. images or latent vectors) and $a_t$ represents actions. For simplicity, we can consider a deterministic Markovian model:

$s_{t+1} = F_\theta(s_t, a_t)$

The key advantage of world models is that they provide a differentiable simulator, you can roll them forward under hypothetical action sequences and backpropagate through the predictions. This opens up possibilities for gradient-based planning and control.

However, using these models for long-horizon planning introduces several challenges:

1. Long rollouts create deep, ill-conditioned computation graphs, leading to exploding or vanishing gradients.

2. The optimization landscape becomes non-greedy and full of local minima as horizons grow.

3. For deep learning-based models, optimizing directly over states can be extremely brittle due to sensitivity of state-input gradients.

GRASP: A New Approach to Long-Horizon Planning

GRASP addresses these challenges through three key innovations:

1. Lifting the trajectory into virtual states, allowing optimization to happen in parallel across time steps.

2. Adding stochasticity directly to the state iterates to aid exploration.

3. Reshaping gradients to provide clean signals for actions while avoiding brittle 'state-input' gradients through high-dimensional vision models.

The core idea is to treat the dynamics constraint $s_{t+1} = F_{\theta}(s_t, a_t)$ as a soft constraint, optimizing over both actions and states:

$\min_{\mathbf{s},\mathbf{a}} \mathcal{L}(\mathbf{s}, \mathbf{a}) = \sum_{t=0}^{T-1} \big\|F_\theta(s_t,a_t), s_{t+1}\big\|_2^2$

This formulation, sometimes called collocation, shares the same global minimizers as the original rollout objective but offers several advantages:

• Each world model evaluation depends only on local variables, enabling parallel computation across time steps.
• It avoids backpropagating through a deep T-step composition, mitigating the exploding/vanishing gradient problem.
• Direct optimization of states allows temporary navigation through unphysical domains, aiding exploration.

Addressing the Brittleness of Deep Learning Models

While the lifted formulation offers clear benefits, it introduces a new challenge when working with deep learning-based world models. Directly optimizing states through a learned $F_{\theta}$ can be extremely brittle, similar to issues of adversarial robustness in image classification.

Even when training world models in lower-dimensional state spaces, the training process can create sharp landscapes for unseen states or directions orthogonal to the data manifold. This sensitivity of state-input gradients can make optimization difficult and unstable.

GRASP tackles this issue through careful gradient reshaping, providing clean signals for actions while avoiding the pitfalls of optimizing directly over high-dimensional state spaces.

The Significance of GRASP

By addressing key challenges in long-horizon planning with learned world models, GRASP represents an important step towards more practical and robust planning systems. Its approach of lifting trajectories, adding stochasticity, and reshaping gradients offers a promising direction for leveraging the power of large, learned world models for control and decision-making tasks.

As world models continue to improve in their predictive capabilities, techniques like GRASP will be crucial for translating those improvements into real-world planning and control applications. However, it's important to note that while GRASP shows promise, further research will be needed to fully understand its limitations and applicability across different domains and model architectures.