Anomalies by Synthesis

Anomaly Detection using Generative Diffusion
Models for Off-Road Navigation

Sunshine Jiang*

MIT CSAIL

Siddharth Ancha*

MIT CSAIL

Travis Manderson

MIT CSAIL

Laura Brandt

MIT CSAIL

Yilun Du

MIT CSAIL

Philip R. Osteen

US Army Research Lab

Nicholas Roy

MIT CSAIL

ICRA 2025

Paper Technical Report 3-min Video Code

Colab Notebook

Pipeline for anomaly detection using analysis by synthesis

Land navigation videos with anomalies removed

Input image

Synthesized image

Anomaly scores

The diffusion model blends the building into the sky in the background, since the building and sky happen to be of the same color. The modified region looks like a clearing in the vegetation. Futhermore, the building's door is morphed into a short tree stump! This allows the analysis pipeline to detect the anomalies (right).

Input image

Synthesized image

Anomaly scores

The diffusion model blends the brown vehicle into mud in the background. However, it does not modify the semantics of other parts of the scene. Later in the video, the diffusion model removes a metal container lying on the ground. Both the brown vehicle and container are detected as anomalies.

Anomaly detection on underwater images

Input image

Synthesized image

Anomaly scores

The scene contains a lionfish that camouflages against rocks. Pertinently, the diffusion model transforms the dark, gray fish into rocks in the background. The diffusion model also grows more rocks to bridge the gap between the fish and the background! This allows our analysis pipeline to cleanly detect the fish as an anomaly.

Input image

Synthesized image

Anomaly scores

The scene contains a diver, a fish, and a robot — all are anomalies in the dataset. The diffusion model removes the diver fish, and robot from the scene! It removes the foreground objects but retains the background almost unchanged. This allows our analysis pipeline to cleanly detect the three anomalies on the right.

Input image

Synthesized image

Anomaly scores

A stingray appears midway in this video. While the stingray is of similar color to the seabed and camouflages against it, our diffusion model is able to remove the stingray without affecting the rest of the scene. This allows our analysis pipeline to cleanly detect the stingray as an anomaly, only in the frames that it appears in.

Acknowledgements

This material is based upon work supported by the Army Research Office under Cooperative Agreement No. W911NF-21-2-0150. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.