Deep Evidential Uncertainty Estimation for Semantic Segmentation under OOD Obstacles

Siddharth Ancha
MIT CSAIL
Philip R. Osteen
US Army Research Lab
Nicholas Roy
MIT CSAIL

ICRA 2024
Paper Technical Report Video
Talk (3 min)

Pipeline for pixel-wise Bayesian evidential uncertainty

Our pipeline for estimating Bayesian evidential uncertainty for each pixel. A learned feature extractor (encoder), that forms the backbone of standard semantic segmentation pipelines, inputs image \(\mathbf{I}\) and outputs a latent feature representation \(\mathbf{x}^i\) for each pixel \(i\). A semantic classifier head classifies the latent representation into one of \(C\) class labels \(y^i\) by predicting \(p(y^i \,\vert\, \mathbf{x}^i)\); this helps determine aleatoric uncertainty over labels. A normalizing flow learns an invertible transformation from a fitted Gaussian mixture model to the latent distribution. The invertible transformation allows computing the probability density \(p(\mathbf{x}^i)\) of a given latent vector \(\mathbf{x}^i\) to determine epistemic uncertainty. The classification output and density estimates are combined in an evidential uncertainty framework to produce a Dirichlet-based hierarchical uncertainty distribution. Finally, a decoder inputs the pixel's shared latent representation \(\mathbf{x}^i\) to reconstruct a patch in the original image centered at the pixel. Patch reconstruction encourages the feature extractor to learn expressive features \(\mathbf{x}^i\) that enable detecting OOD segments. The decoder is not required at test time and is discarded; therefore the decoder does not add to the computation cost of our method.
OOD uncertainty on RUGD

Visualization of epistemic uncertainty estimated using normalizing flows on a scene from the RUGD dataset.

  • • Epistemic uncertainty is high for the white building and the puddle of water; buildings and water are OOD classes were never seen during training time.
  • • Our method is able to identify long-tail anomalies such as fallen trees as out-of-distribution, even though standing trees are correctly predicted as in-distribution.
  • • Our method produces high uncertainty for vehicles that are out-of-distribution even though they are far away.
OOD uncertainty on Cityscapes
These examples visualize epistemic uncertainty estimated using normalizing flows on the Cityscapes dataset, a semantic segmentation dataset in urban driving environments. Epistemic uncertainty is high for out-of-distribution objects such as bicycles and motorcycles that were never seen during training time. Epistemic uncertainty is also high for long-tail, low-probability objects such as fences and containers on the side of the street.

Epistemic and Aleatoric uncertainty
Left: The input/query image at test time containing a puddle of water; water or puddles were never seen by the model during training. Middle: Estimated epistemic uncertainty is high at unfamiliar, out-of-distribution regions that are not well represented in the training data. Right: Aleatoric uncertainty is the inherent and irreducible uncertainty due to ambiguous labels. It estimated to be high at the boundaries between semantic classes, where the true label is ambiguous or the the training labels were noisy. Although aleatoric uncertainty is high only at the boundary of the puddle (due to misclassification), epistemic uncertainty is high throughout the puddle. Therefore, we distinguish between, model and predict both uncertainties in the Bayesian evidential framework.
Acknowledgements

This material is based upon work supported by the DEVCOM Army Research Laboratory under Grant No. W911NF-21-2-0150. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the DEVCOM Army Research Laboratory and DARPA.