We introduce a new approach to image forensics: placing physical refractive objects, which we call totems, into a scene so as to protect any photograph taken of that scene. Totems bend and redirect light rays, thus providing multiple, albeit distorted, views of the scene within a single image. A defender can use these distorted totem pixels to detect if an image has been manipulated.
Our approach unscrambles the light rays passing through the totems by estimating their positions in the scene and using their known geometric and material properties. To verify a totem-protected image, we detect inconsistencies between the scene reconstructed from totem viewpoints and the scene’s appearance from the camera viewpoint. Such an approach makes the adversarial manipulation task more difficult, as the adversary must modify both the totem and image pixels in a geometrically consistent manner without knowing the physical properties of the totem. Unlike prior learning-based approaches, our method does not require training on datasets of specific manipulations, and instead uses physical properties of the scene and camera to solve the forensics problem.
Totems as a general method to protect a piece of signal (of Alice). Even if the middle person (Eve) can tamper the content, they can’t easily edit both the signal and the totem signature consistently. A defender (Bob) can thus detect such manipulations.
Reconstruction overview. Given a single camera view that contains totem view(s), we reconstruct the 3D scene from only totem pixels. More specifically, we map totem pixels to refracted rays and optimize a radiance field.
The inner workings of mapping totem pixels to rays. We first map each pixel to camera ray and then analytically compute the refractions using known totem physical properties (i.e. geometry and refractive index) and estimated totem pose P. We jointly optimize the totem pose with the radiance field and add a regularization loss to keep totems from drifting away.
Detection pipeline. We render the same camera view from the reconstructed radiance field and compute patch-wise difference scores with a "maybe-manipulated" image. Then we resize the score grid to image size, filter out pixels with low reconstruction confidence and overlay the heatmap.
Manipulation type #1 - adding random color patches
(Click here to zoom in on the manipulated images)
Manipulation type #2 - image splicing, pasting in content from another photo
(Click here to zoom in on the manipulated images)
Manipulation type #3 - using Photoshop's content aware fill to remove content
(Click here to zoom in on the manipulated images)
Manipulation type #4 - reference shift, shifting the person in both camera and totem views to the same reference point
(Click here to zoom in on the manipulated images)
Here we hold the scene and the manipulation type unchanged to demonstrate the effects of changing totem layout. Our method accurately identifies the manipulations for all above layouts.
We observe that the protection mask shrinks when totems appear smaller in the photo (e.g. row 1). Note that this is only a limitation of the current implementation, not the general totem framework.
Here we use unmanipulated photos to demonstrate the effects of joint optimization. We observe that the reconstruction quality improves when we jointly optimize the totem pose along with the radiance field.
Totems: Physical Objects for Verifying Visual
Integrity
@article{ma2022totems,
author = {Ma, Jingwei and Chai, Lucy and Huh, Minyoung and Wang, Tongzhou and Lim, Ser-Nam and Isola, Phillip and Torralba, Antonio},
title = {Totems: Physical Objects for Verifying Visual Integrity},
journal = {ECCV},
year = {2022},
}