logoTotems: Physical Objects for Verifying Visual Integrity

1University of Washington, 2MIT, 3Meta AI
teaser

Totems can be placed into a scene to protect photos against adversarial manipulations.

Skip to:    [Abstract]   [Video]   [The Totem Framework]   [Method]   [Results]   [Ablation]   [Cite]  

Abstract

We introduce a new approach to image forensics: placing physical refractive objects, which we call totems, into a scene so as to protect any photograph taken of that scene. Totems bend and redirect light rays, thus providing multiple, albeit distorted, views of the scene within a single image. A defender can use these distorted totem pixels to detect if an image has been manipulated.

Our approach unscrambles the light rays passing through the totems by estimating their positions in the scene and using their known geometric and material properties. To verify a totem-protected image, we detect inconsistencies between the scene reconstructed from totem viewpoints and the scene’s appearance from the camera viewpoint. Such an approach makes the adversarial manipulation task more difficult, as the adversary must modify both the totem and image pixels in a geometrically consistent manner without knowing the physical properties of the totem. Unlike prior learning-based approaches, our method does not require training on datasets of specific manipulations, and instead uses physical properties of the scene and camera to solve the forensics problem.

Video

The Totem Framework

teaser

Totems as a general method to protect a piece of signal (of Alice). Even if the middle person (Eve) can tamper the content, they can’t easily edit both the signal and the totem signature consistently. A defender (Bob) can thus detect such manipulations.

Two-stage verification method

Stage 1: Reconstruction

teaser

Reconstruction overview. Given a single camera view that contains totem view(s), we reconstruct the 3D scene from only totem pixels. More specifically, we map totem pixels to refracted rays and optimize a radiance field.

teaser

The inner workings of mapping totem pixels to rays. We first map each pixel to camera ray and then analytically compute the refractions using known totem physical properties (i.e. geometry and refractive index) and estimated totem pose P. We jointly optimize the totem pose with the radiance field and add a regularization loss to keep totems from drifting away.


Stage 2: Detection

teaser

Detection pipeline. We render the same camera view from the reconstructed radiance field and compute patch-wise difference scores with a "maybe-manipulated" image. Then we resize the score grid to image size, filter out pixels with low reconstruction confidence and overlay the heatmap.

Results

#1 - Color Patch

results#1

Manipulation type #1 - adding random color patches
(Click here to zoom in on the manipulated images)

#2 - Image splice

results#2

Manipulation type #2 - image splicing, pasting in content from another photo
(Click here to zoom in on the manipulated images)

#3 - Content aware fill

results#3

Manipulation type #3 - using Photoshop's content aware fill to remove content
(Click here to zoom in on the manipulated images)

#4 - Reference shift

results#4

Manipulation type #4 - reference shift, shifting the person in both camera and totem views to the same reference point
(Click here to zoom in on the manipulated images)

Ablation study

#1 - Changing totem layouts

results#2

Here we hold the scene and the manipulation type unchanged to demonstrate the effects of changing totem layout. Our method accurately identifies the manipulations for all above layouts.

We observe that the protection mask shrinks when totems appear smaller in the photo (e.g. row 1). Note that this is only a limitation of the current implementation, not the general totem framework.

#2 - Reconstruction quality (without vs. with totem pose optimization)

results#2

Here we use unmanipulated photos to demonstrate the effects of joint optimization. We observe that the reconstruction quality improves when we jointly optimize the totem pose along with the radiance field.

BibTeX

@article{ma2022totems,
  author    = {Ma, Jingwei and Chai, Lucy and Huh, Minyoung and Wang, Tongzhou and Lim, Ser-Nam and Isola, Phillip and Torralba, Antonio},
  title     = {Totems: Physical Objects for Verifying Visual Integrity},
  journal   = {ECCV},
  year      = {2022},
}