CAVE: Interpretable 3D Neural Object Volumes for Robust Conceptual Reasoning

With the rise of deep neural networks, especially in safety-critical applications, robustness and interpretability are crucial to ensure their trustworthiness. Recent advances in 3D-aware classifiers that map image features to volumetric representation of objects, rather than relying solely on 2D appearance, have greatly improved robustness on out-of-distribution (OOD) data. Such classifiers have not yet been studied from the perspective of interpretability. Meanwhile, current concept-based XAI methods often neglect OOD robustness. We aim to address both aspects with CAVE - Concept Aware Volumes for Explanations - a new direction that unifies interpretability and robustness in image classification. We design CAVE as a robust and inherently interpretable classifier that learns sparse concepts from 3D object representation. We further propose 3D Consistency, a metric to measure spatial consistency of concepts. Unlike existing metrics that rely on human-annotated parts on images, 3D-C leverages ground-truth object meshes as a common surface to project and compare explanations across concept-based methods. CAVE achieves competitive classification performance while discovering consistent and meaningful concepts across images in various OOD settings.

We introduce CAVE, a framework for robust conceptual reasoning and classification through 3D-aware concept-based ellipsoid neural object volumes (NOVs). In this visual illustration, colors indicate the top-5 concepts within each class. For classification, CAVE combines (a) extracted image features F_x and (b) interpretable sparse concept-aware NOVs ℋ through a bag-of-word concept matching (c), where each feature f_i ∈ F_x is best aligned with ℋ by cosine similarity. Correct classification happens when many image features activate Car concepts, while concepts in other classes fail to align with any feature (crossed-out arrows).

We introduce 3D consistency (3D-C), a geometry-based metric that measures if a concept consistently maps to the same semantic region of an object across viewpoints and distribution shifts. By projecting concept attributions onto a common 3D object surface, 3D-C evaluates concept stability without relying on human-annotated parts, providing a principled measure of concept consistency. Higher 3D-C means more consistent mapping to the same region.

To obtain input-level explanations for our neural object volume (NOV) concepts, we adapt layer-wise relevance propagation (LRP) to volumetric architectures. Our NOV-aware redistribution rule preserves LRP conservation property through concept matching, enabling faithful pixel-level concept attributions. Please refer to our paper for full derivation and more qualitative examples.

Our NOV-aware LRP correctly attributes concepts and yields localised explanations, even under different OOD settings: snow and 40-60% (heavy) occlusion. Colors indicate the top-5 class-wise concepts per row and are not comparable across rows.