Adversarial Imaging Pipelines
AAdversarial Imaging Pipelines
Buu PhanAlgolux Fahim MannanAlgolux Felix HeideAlgolux, Princeton University
Abstract
Adversarial attacks play an essential role in understand-ing deep neural network predictions and improving their ro-bustness. Existing attack methods aim to deceive convolu-tional neural network (CNN)-based classifiers by manipu-lating RGB images that are fed directly to the classifiers.However, these approaches typically neglect the influenceof the camera optics and image processing pipeline (ISP)that produce the network inputs. ISPs transform RAW mea-surements to RGB images and traditionally are assumedto preserve adversarial patterns. However, these low-levelpipelines can, in fact, destroy, introduce or amplify adver-sarial patterns that can deceive a downstream detector. Asa result, optimized patterns can become adversarial for theclassifier after being transformed by a certain camera ISPand optic but not for others. In this work, we examineand develop such an attack that deceives a specific cam-era ISP while leaving others intact, using the same down-stream classifier. We frame camera-specific attacks as amulti-task optimization problem, relying on a differentiableapproximation for the ISP itself. We validate the proposedmethod using recent state-of-the-art automotive hardwareISPs, achieving 92% fooling rate when attacking a specificISP. We demonstrate physical optics attacks with 90% fool-ing rate for a specific camera lenses.
1. Introduction
Deep neural networks have become a cornerstonemethod in computer vision [7, 19, 20, 24, 57] with diverseapplications across fields, including safety-critical percep-tion for self-driving vehicles, medical diagnosis, video se-curity, medical imaging and assistive robotics. Although awide range of high-stakes applications base their decisionmaking on the output of deep networks, existing deep mod-els have been shown to be susceptible to adversarial attackson the image that the network ingests. Specifically, existingadversarial attacks perturb the input image with carefullydesigned patterns to deceive the model while being imper-ceptible to a human viewer [33, 40, 43, 47, 36, 51]. As such,understanding and exploring adversarial perturbations offer
Original Image:
75% R.V.
Adversarial Image:
99% Anole
Original Image:
85% R.V.
Adversarial Image:
85% R.V.
Original Image:
93% R.V.
Adversarial Image:
93% R.V.
CLASSIFIER CLASSIFIER CLASSIFIER
OPTICS AISP A OPTICS BISP B OPTICS CISP C
OBJECT: R.V.
Figure 1: We illustrate and show the camera-specific attack. Theimage is tampered such that it becomes only adversarial for a spe-cific camera pipeline, even when the three pipelines deploy thesame classifier. insights into the failure cases of today’s models and it allowsresearchers to develop defense methods and models that areresilient against proposed attacks [3, 32, 33, 41, 54].Existing adversarial attacks find post-capture adver-saries, tampering with the image after capture before it isinput to the deep network. Recently, a number of attackmethods have been demonstrated in the form of physicalobjects that are placed in real-world scenes to generate ad-versarial patterns by capturing images of the physical ob-jects [2, 13, 28]. The most successful methods for com-puting adversarial perturbations rely on network gradientsto form adversarial examples [47, 17, 28, 35, 40, 4] foreach input image, that struggle to transfer to other net-works or images [47, 31, 38]. Alternative approaches relyon only the network predictions [23, 37, 46] and use sur-1 a r X i v : . [ c s . C V ] F e b ogate networks [39] or gradient approximations [1]. Allof these methods, both physical and synthetic attacks, havein common that they assume that the camera image pro-cessing pipeline (ISP) to preserve the attack pattern . Al-though modern image processing pipelines implement com-plex algorithms, such as tonemapping, sharpening or de-noising [26, 27], which transform RAW measurements toRGB images on embedded camera processors, the influ-ence of this pipeline is ignored by existing attack methods.Some of the processing blocks in camera image processingpipelines have even been suggested as defenses against ex-isting attacks [18, 30].In this work, we close this gap between scene-basedphysical attacks and attacks on post-processed images.Specifically, we propose a novel method that allows us toattack cameras with a specific ISP, while leaving the detec-tions of other cameras intact for the identical classifier buta different ISP. As such, the attack mechanism proposed inthis work is a camera-specific attack that not only targetsthe deep network but conventional hardware ISPs that tradi-tionally have been not been considered susceptible to adver-sarial attacks. As a further camera-specific attack, we alsoattack the optical system of a camera system. The proposedmethod can incorporate proprietary black-box ISP and com-plex compound optics, without accurate models, by rely-ing differentiable approximations as gradient oracles. Wevalidate our method using recent automotive hardware ISPprocessors and automotive optics, where the novel attackachieves a fooling rate of 92% on RAW images in experi-mental captures.Specifically, we make the following contributions• We introduce the first method for finding adversarialattacks that deceives a specific camera ISP and opticswhile leaving cameras with other ISPs or optics intactalthough they employ the same classifier network.• We demonstrate attacks for embedded hardware ISPsthat are not differentiable and only available as black-box algorithms. To this end, we learn differentiableapproximations of the image processing and sensingpipeline that serves as gradient oracles for our attack.• We analyze and validate the attack on RAW input mea-surements for state-of-the-art hardware ISPs.• We validate physical attacks of the proposed methodon recent automotive camera ISPs and automotive op-tics. The proposed method achieves more than 90%success rate.
2. Related Work
Our work considers the problem of adversarial attacks oncamera pipelines. We review the relevant literature below.
Camera Image Processing Pipelines.
Research on high-level vision tasks has often overlooked the existence of thelow-level image signal processing (ISP) pipeline in the cam-era. In practice, the role of these ISPs is critical in a vi-sion system because their ability to recover high qualityimages from noisy and distorted RAW measurements di-rectly affects the downstream processing modules [21, 50].For display applications, domain-specific image processingmethods [15, 16, 5, 9, 56, 14, 21] have been successfulto tackle low-light, shot-noise and optical aberrations. Un-fortunately, these methods are computationally expensive,and, as such, their application is limited to off-line tasks. Incontrast, real-time applications, such as robotics and aug-mented reality demand real-time processing at more than30 Hz for double-digit megapixel streams. As a result, inte-grated system-on-chip ISPs are today employed for roboticvision systems, such as autonomous robots, self-driving ve-hicles, and drones. For example, the ARM Mali-C71 ASICISP is capable of processing 12 megapixel streams at up to100 Hz with less than one Watt power consumption. How-ever, although hardware ISPs are efficient, these processingpipelines are typically highly optimized proprietary com-pute units that are not differentiable and their behavior isunknown to the user [50]. In this work, we present thefirst adversarial attack that targets these hardware process-ing blocks, which, in contrast to deep neural networks, tra-ditionally have been assumed to be not susceptible to ad-versarial perturbations and instead have been suggested aspotential defense units [18, 30].
Adversarial Attacks.
A large body of work has ex-plored adversarial attacks on deep networks in computervision. A common formulation describes an attack asan (cid:96) p norm-ball constrained perturbation that deceives aspecific classifier [33]. Depending on the knowledge ofthe model ( i.e . weights and architecture) that the adver-sary has, attacks can be grouped into two settings: white-box and black-box attacks. In the white-box setting, themodel specification are known and the adversaries lever-age it to synthesize the perturbation. By treating the at-tack as a solution of an optimization problem, techniquesranges from mixed-integer programming [49, 53] to 1 st -order gradient method [17, 33, 35, 47] have been proposed.Additionally, by manipulating the optimization objectivesand constraints, attacks can reveal interesting propertiesof the target network, such as sparsity and interpretabil-ity [58, 4, 34, 55, 48]. In the black-box setting, adversariescan only query the input-output pairs, and, hence, the tar-get model is more difficult to be deceived. Nevertheless,existing approaches have shown that adversaries can suc-cessfully approximate the gradients and apply the white-box method. This is achieved by approximating the targetnetwork function [39, 1] (transfer methods) or by numer-ical estimation (score methods) [22, 52, 6, 29, 8]. In this2ork, we propose a transfer approach that approximatesnon-differentiable camera pipelines, including the cameraoptics and ISP, with differentiable proxy functions.Going beyond synthetically generated adversarial exam-ples, researchers have shown to be able to recreate themin the wild by placing adversarial patterns on physicalobjects. Kurakin et al. [28] demonstrate such a physi-cal attack by printing the digital adversarial image on pa-per and capturing it with a camera, assuming that the ac-quisition and capture pipeline itself is not susceptible tothe adversarial pattern. Athalye et al. [2] propose an at-tack which optimizes the perturbation under different im-age augmentations, a direction further explored by a line ofwork [13, 25, 45, 12, 10] to achieve higher attack ratios. Allof these existing methods have in common that they assumethat the scene light transport and acquisition preserve theadversarial patterns, including the optics, sensors and ISPin the camera as non-susceptible image transforms. As a di-rect result, existing physical attacks have failed to achievethe high fooling rates of synthetic attacks [28]. Our workfills this gap and, as a result, shows that it is possible toachieve high fooling rates when including the acquisitionand processing operations in adversarial attacks. Buildingon this insight, we realize attacks of individual camera typesby exploiting slight differences in their acquisition and im-age processing pipeline.
3. Background
In this section, we review the differentiable proxy frame-work from Tseng et al. [50] and the projected gradient de-scent (cid:96) p norm-bounded adversarial attack [33], and we in-troduce relevant notation for the following sections. A given non-differentiable hardware ISP is approxi-mated by a differentiable proxy function, which implementsa mapping from RAW input data to post-ISP images viaa convolutional neural network (CNN). We note that thisframework can also be extended to include the compoundoptics in the pipeline (see the Supplementary Document).
Proxy ISP Model.
We denote h : R d −→ R d × as a black-box ISP function that maps a RAW image x ∈ R d to anRGB image, where d is the RAW image dimension ( e.g . × ). The proxy ISP function ˜ h θ : R d −→ R d × de-pends on θ as learnable parameters ( i.e . CNN weights) alsomaps a RAW image to a post-ISP image. As a departurefrom Tseng et al. [50], we found that bilinear demosaicingas a first layer in this proxy module improves training stabil-ity and accuracy. This demosaicing layer is differentiable.The demosaiced RGB image is fed into a U-Net [44], whichis trained to approximate the output of the hardware ISP. Proxy Training.
Given a set of RAW captures: X = { x , x , ..., x N } where each x i ∈ R d , we train the proxy function ˜ h θ by minimizing the (cid:96) reconstruction loss. Let us denote a probabilistic classifier that maps an input x ∈ R d to a categorical distribution vector as f : R d −→ R K , where d is the input dimension and K is the number ofclasses. We define a decision function c ( x ) , which assignsa label to x as: c ( x ) = arg max k =1 , ,...,K f k ( x ) . (cid:96) p norm-bounded attack. For an input x , an additive per-turbation δ ∈ B d ( p ; (cid:15) ) is adversarial when c ( x + δ ) = t ,where t is a target label and B d ( p ; (cid:15) ) = { r ∈ R d : (cid:107) r (cid:107) p <(cid:15) } is an (cid:96) p norm-ball with radius (cid:15) . We will use (cid:96) ∞ throughout this paper. To create such a perturbation, wesolve the following constrained optimization problemminimize (cid:107) δ (cid:107) ∞ ≤ (cid:15) L ( f ( x + δ ) , t ) , (1)where L is the cross-entropy loss. Projected Gradient Descent (PGD).
In the case of (cid:96) ∞ , wecan solve (1) by first randomly initializing δ ∈ B d ( ∞ ; (cid:15) ) and iteratively perform the following PGD update δ ←− δ − α · sgn ( ∇ δ L ( f ( x + δ ) , t )) . (2)where α is the step size, which can depend on the currentordinal iteration number. This attack can be denoted as a“targeted” attack. The “untargeted” variant, where we aimto fool the model independently of the target class, can beformulated in a similar fashion by maximizing the loss in(1) and set the target label to the original prediction.
4. Camera Pipeline Adversarial Attack
In the following, we consider an camera pipeline con-sisting of a black-box, non-differentiable ISP followed bya downstream RGB image classifier. A direct RAW attackon such a pipeline involves manipulating the captured RAWimage. For a physical camera attack, our pipeline also in-cludes the optical system that captures an adversarial scene.In this section we only explain the direct RAW attack with-out any loss of generality.Next, we describe two types of attacks on these pipelinesand the method to generate them. The first type of attack,referred to as untargeted camera attack , aims to craft anadversarial RAW perturbation to the pipeline, without con-sidering its transferability to the other pipelines. The secondtype, referred to as targeted camera attack , generates a per-turbation that deceives a specific pipeline while leaving theother intact, even when the same classifier is deployed. Fig-ure 2 provides an overview of the proposed targeted cameraattack and corresponding proxy functions. As an image, ( x + δ ) needs to stay within the valid range ( e.g . [0,255]for RGB images), which can be achieved by the clipping operation. Weimplicitly assume this condition throughout this paper without stating it. inear Demosaicing Local Proxy ISP B
Linear Demosaicing
Local Proxy ISP A
BLACK-BOX ISP A
Optics RAW Image Camera B
Black-box ISP A
Optics A Captured RAW Image Camera A
UNET
Local Proxy Camera BLocal Proxy Camera A
UNET
Classifier
Camera A/ ISP A: PlaneCamera B/ ISP B: Black Swan
Blocked gradient flow Original Scene s Adversarial Perturbation 𝛿 Original Scene s Gradient flow: Feedforward flow: Physical Camera Attack:
ISP Output
Captured RAW Image x Adversarial Perturbation 𝛿 Physical ISP Attack:
Figure 2: Overview of the proposed targeted camera attack. We perturb either the display scene (physical camera attack) or the capturedRAW image (physical ISP attack), whose label is “black swan”, such that they are misclassified into “plane” by pipeline A but not bypipeline B. To find such an attack, we solve an optimization problem, using the estimated gradients from the proxies approximation of theblack-box, non-differentiable imaging modules. The objective function is a weighted sum of two cross-entropy losses, where the first termencourages the attack to fool pipeline A and the second term prevents it from changing the original prediction probability of pipeline B.
Algorithm 1
Local Proxy Training
Input: h ; ˜ h ; g ; Number of augmented images M ; number of at-tack iterations n ; a list of targeted images S ; a predefinedbound (cid:15) ; update step size α . Output:
A local proxy function ˆ h ˆ S = S ˜ f = ( g ◦ ˜ h ) for all x i ∈ S do : for m ←− ...M do : ˆ (cid:15) ∼ uniform ( α, (cid:15) + α ) δ ←− PGD ( x i , ˜ f, n, ˆ (cid:15) ; α ) (cid:46) perform n-steps PGD up-date, target random class ˆ S = ˆ S ∪ { x i + δ, h ( x i + δ ) } end for end for ˆ h ←− TRAIN (˜ h, ˆ S ) (cid:46) Train the local proxy ˆ h from ˆ S and ˜ h return ˆ h We define a black-box ISP function as h : R d −→ R d × ,a trained proxy function that approximates h as ˜ h θ : R d −→ R d × and an RGB image classifier as g : R d × −→ R K .Given a RAW image x ∈ R d , we define the camerapipelines using the original ISP and proxy ISP separatelyas: f ( x ) = ( g ◦ h )( x ) and ˜ f ( x ) = ( g ◦ ˜ h )( x ) . Similar toSec.3.2, c ( x ) and ˜ c ( x ) are the corresponding decision func-tions. Before describing the two camera attacks, we next in-troduce the a local proxy function, which is a modificationof Tseng et al. [50]’s model that is essential to the successof the proposed attack. In our experiments, we found that, despite ˜ h ( x ) beingperceptually similar to h ( x ) , performing the PGD-updatebased on the estimated gradient from ˜ h does not result in high success rate in many cases, especially for the targetedcamera attack. To this end, we propose using a local proxymodel as an alternative gradient-oracle, which is trained byfine-tuning the existing proxy model ˜ h with a set of targetimages and Jacobian augmentation [39]. We find that sucha local proxy model effectively improves the success ratefor both untargeted and targeted camera attacks.Specifically, given an image set S that we wish to attack,we create M different Jacobian-augmented pairs: { ( x i + δ i ) , h ( x i + δ i ) } for each x i ∈ S , where δ i ∈ B d ( p ; ˆ (cid:15) ) isthe adversarial perturbation on the proxy pipeline ˜ f and thebounded radius ˆ (cid:15) is uniformly sampled within [ α, (cid:15) + α ] ,where α is the PGD update step size. The local proxy model ˆ h is obtained by finetuning ˜ h with the newly augmentedtraining set ˆ S . This method is formalized in Algorithm 1. For this attack type, we aim to generate an adversarialperturbation δ ∈ B d ( ∞ ; (cid:15) ) to a RAW image x such that: c ( x + δ ) = t independent of the camera pipeline. We re-place the black-box ISP h with its local proxy function ˆ h and generate adversarial perturbations δ from the PGD up-date on ˆ f = g ◦ ˆ h , that is: δ ←− δ − α · sgn ( ∇ δ L ( ˆ f ( x + δ ) , t )) (3)We found that, despite both ˜ h ( x ) , ˆ h ( x ) being perceptuallysimilar to h ( x ) , estimated gradient using ˆ h consistentlyyields a higher success rate than ˜ h (refer to the supplemen-tary document for quantitative comparisons). We illustratethis in Figure 3, showing that being trained with differentperturbations enables ˆ h to provide accurate gradients for theattack to transfer well to h .4 arget Class: ShetlandProxy: Shetland 83% Target Class: ShetlandPhysical: Snowbird 48% Target Class: ShetlandPhysical: Shetland 89%Target Class: ShetlandProxy: Shetland 91% Input RAWISP Output (a) Tseng et al.
Proxy (b) Proposed local Proxy
Original RAW (Snowbird) RAW Perturbations Original RAW (Snowbird) RAW Perturbations
Figure 3: Local Proxies: Example that demonstrates the proxyfrom Tseng et al. [50] fails to approximate the real ISP adequatelyfor adversarial attack. The proposed local proxy attack success-fully causes the physical pipeline to misclassify the image into thetarget class “Shetland”, while Tseng et al. ’s proxy fails.
For this attack type, we aim to craft a perturbation thatdeceives a specific camera pipeline h , while leaving theclassifications of other camera pipelines intact, even whenall the pipelines deploy the same classifier g . Let h i , for i ∈ { , , ..., T } , be one of the ISPs that we do not wantto attack, its associated camera pipeline and decision func-tion are f i ( x ) = ( g ◦ h i )( x ) and c i ( x ) . We assume that g is transferable across different ISPs, i.e . accuracy higherthan for each ISP in the Imagenet dataset. Ideally, anadversarial perturbation δ ∈ B d ( ∞ ; (cid:15) ) to an image x withlabel y , should satisfy: c ( x + δ ) = t and f i ( x + δ ) = f i ( x ) ,given that c ( x ) = y and every c i ( x ) = y . Such a perturba-tion can be found as a solution of the following optimizationproblemminimize || δ || p ≤ (cid:15) L ( f ( x + δ ) , t ) s.t f i ( x + δ ) = f i ( x ) , ∀ i ∈ , ..., T . (4)The problem from (4) is a challenging nonlinear-equalityconstrained problem that may have only have a feasible so-lution with large cross-entropy loss. Soft-Constrained Objective
When h and h i are known anddifferentiable, we can relax (4) using soft-constraints andapplying the PGD update on δ to jointly minimize the objec-tive function and the distance between f i ( x + δ ) and f i ( x ) minimize || δ || p ≤ (cid:15) L obj ( x, δ, t ) , (5)where L obj ( x, δ, t ) = L ( f ( x + δ ) , t ) + (cid:80) Ti =1 λ i L ( f i ( x + δ ) , f i ( x )) . The second term measures the cross-entropy lossbetween f i ( x + δ ) and f i ( x ) and each λ i is set to 1 in ourexperiment. We note that minimizing the cross-entropy lossin this case is equivalent to minimizing the KL divergencebetween the two categorical distributions as L ( f i ( x + δ ) , f i ( x ))= D KL ( f i ( x + δ ) || f i ( x )) − H ( f i ( x ))= D KL ( f i ( x + δ ) || f i ( x ))+ const . (6) Algorithm 2
Targeted Camera Adversarial Perturbation
Input:
Targeted ISP h ; Untargeted ISPs { h , h , ..., h T } ; Pre-trained local proxies: ˆ h, { ˆ h , ˆ h , ..., ˆ h T } ; RGB classifier g ;number of attack iterations n ; targeted image x ; targeted class t ; perturbation bound (cid:15) . Output: adversarial image x (cid:48) ∈ R d / / Construct the proxy pipelines: ˆ f = ( g ◦ ˆ h ); ˆ f i = ( g ◦ ˆ h i ) / / Construct the objective function: ˆ L obj ( x, δ, t ) = L ( ˆ f ( x + δ ) , t )+ (cid:80) Ti =1 λ i L ( ˆ f i ( x + δ ) , f i ( x )) / / Attack the targeted image: δ ∼ uniform ( − (cid:15), (cid:15) ) for k ←− ...n do : δ ←− clip ( x + δ ) − x (cid:46) Clip δ to the valid range δ ←− δ − α · sgn ( ∇ δ ˆ L obj ( x, δ, t )) end for x (cid:48) = clip ( x + δ ) (cid:46) Clip x + δ to the valid range return x (cid:48) Objective Function with Local Proxy ISP.
Since h and h i can be non-differentiable, we optimize δ on the new objec-tive function, which replace h and h i with their correspond-ing local proxy ˆ h and ˆ h i , that is ˆ L obj ( x, δ, t )= L ( ˆ f ( x + δ ) , t )+ T (cid:88) i =1 λ i L ( ˆ f i ( x + δ ) , f i ( x )) . (7)The ISP-specific perturbation is found by performing thePGD update on ˆ L obj , that is δ ←− δ − α · sgn ( ∇ δ ˆ L obj ( x, δ, t )) . (8)We note that replacing h and h i with ˜ h and ˜ h i does notgive a high success rate, even for large (cid:15) . This is becausethe gradient estimation quality from ˜ h, ˜ h i is not accurateenough for satisfying several constraints. Finally, while theproposed objective (7) only minimizes the KL divergencebetween ˆ f i ( x + δ ) and f i ( x ) , training the local proxy modelhas indirectly minimized the distance between ˆ f i ( x + δ ) and f i ( x + δ ) around the perturbation radius (cid:15) . We formulate thismethod in Algorithm 2.
5. Assessment
We validate our methods using hardware ISPs and opti-cal assemblies for direct RAW and physical camera attacks.
Dataset.
For all the experiments, we use a subset of 1,000ImageNet validation images [11].
Image Processing Pipelines.
We evaluate our method forthe black-box/non-differentiable hardware ARM Mali C71and Movidius Myriad 2 ISPs. In addition to the two hard-ware ISPs, we also jointly evaluate with two differentiableISPs. The first one only performs bilinear demosaicing, andwill be referred to as the Demosaicing ISP. The second one5 isplay Image ViewSonic VP2785-4k MonitorTripod FLIR Blackfly S Camera + Lens
Figure 4: Setup for Evaluation of Camera-Specific Attacks. Weemploy a monitor placed in front of the target camera system,which is attacked by the proposed method. The proposed setupallows us to evaluate attacks on specific cameras, including theirISPs and camera optics using physical captures. performs bilinear demosaicing operation followed by bilat-eral filtering [42], and referred to as Bilateral Filter ISP.All ISPs are described in detail in the Supplementary Doc-ument.
Optics.
We use a Fujinon CF12.5HA-1 lens with 54° fieldof view as the default lens for our experiments. As thiscompound optics is a proprietary design, we evaluate theproposed attacks on a Cooke triplet optimized for imagequality using Zemax Hammer optimization and fabricatedusing PMMA. Details are described in the SupplementaryDocument.
Classifier.
We use a large Resnet-101 [11] classifier, whichachieves . Top-1 accuracy. Since each ISP has a dif-ferent set of parameters (such as white-balance coefficients,color-correction matrix, etc.), we prevent the domain-shiftproblem by finetuning the pretrained Resnet-101 model ona set of ISP output images.
Evaluation Metrics.
We evaluate success rate, transferrate and targeted success rate as metrics in our evaluation.Success rate measures whether an attack for a given cam-era pipeline is able to change that pipeline’s prediction tothe target class. Transfer rate measures whether an adver-sarial RAW is misclassified by other pipelines. Targetedsuccess rate measures if an attack pattern changes the tar-geted pipeline’s prediction to the target class while leavingother camera pipelines unaffected (class prediction does notchange and the confidence difference between the originaland adversarial RAW is below 0.15).
To validate the proposed method in a physical setup, wedisplay the attacked images on the ViewSonic VP2785-4kmonitor, as shown in Figure 4. This setup allows us to col-lect large-scale evaluation statistics in a physical setup, de-parting from sparse validation examples presented in exist-ing works with RGB printouts [28, 2]. We capture imagesusing a FLIR Blackfly S camera employing a Sony IMX249
Targeted ISP Deployed ISP
Movidius Myriad 2 ARM Mali C71 Bilateral Filter ISP Demosaicing ISPMovidius Myriad 2 (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (a) Untargeted Physical ISP Attack
Targeted ISP Deployed ISP
Movidius Myriad 2 ARM Mali C71 Bilateral Filter ISP Demosaicing ISPMovidius Myriad 2 (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (cid:107) (b) Targeted Physical ISP Attack
Table 1: Success and transfer rate for the proposed (a) untargetedand (b) targeted physical ISP attack. Each row shows the attacksuccess rate on the targeted ISP (diagonal cells) and transfer rateto other ISPs (non-diagonal cells ). The proposed targeted methodsignificantly reduces the transfer rate across different ISPs. sensor. The camera is positioned on a tripod and mountedsuch that the optical axis aligns with the center of the mon-itor. The camera and monitor are connected to a computer,which is used to jointly display and capture thousands ofvalidation images. Each lens-assembly is focused at infin-ity with the screen beyond the hyperfocal distance. The cap-tured RAW image acquired by the sensor in this setup is fedto the ISPs and then resized to the resolution of × before going through the Resnet-101 classifier. In this attack setting, we acquire the RAW images byprojecting the images onto the screen, using the FLIR cam-era. These RAW images are then fed to a hardware ISP andthe adversarial perturbation is added directly to the RAWimage. We use (cid:15) = 2000 to reliably deceive a RAW imagewhile keeping the perturbation imperceptible. For each im-age, we target a random class and use a total of 30 iterations,with step size α = 50 . Untargeted Camera Attack.
We measure the transferabil-ity of the untargeted camera attack described in Sec. 4.2in Table 1a. We observe that the attacks on one ISP aremore transferable to certain ISPs, e . g ., attacks on the ARMMali C71 ISP are more transferable to the Movidius Myr-iad 2 than the two differentiable ones. Also, attacks on theBilateral Filter ISP and Demosaicing ISP are likely to betransferable to each other, but not to the hardware ISPs. Targeted Camera Attack.
We employ the proposed tar-geted attack Algorithm 2 to craft an attack that only comesinto effect when fed into a specific pipeline. We show the re-sult in Table 1b, where our method significantly reduces thetransfer rate across different ISPs. For each targeted black-box ISP attack, it reduces the transfer rate of the BilateralFilter ISP and Demosaicing ISP to . , and the transferrate to the other black-box ISP is reduced to below . . The first number of the non-diagonal cell is the transfer rate. Thesecond number measures the percentage of images whose confidence forthe adversarial image significantly differs from the adversarial-free one (iftheir confidence difference is greater than 0.15). dversarial Input RAW Movidius Myriad 2 Output Image ARM Mali C71Output Image Bilateral Filter ISPOutput Image Demosaicing ISP Output Image A tt ac k on M ov i d i u s M y r i a d Prediction: Missile 95% Prediction: Jap. Spaniel 89% Prediction: Jap. Spaniel 92% Prediction: Jap. Spaniel 95%Groundtruth: Jap. Spaniel Target: Missile A tt ac k on A R M M a li C Prediction: Bittern 78% Prediction: Bittern 93% Prediction: Cellphone 99% Prediction: Bittern 89% Groundtruth: Bittern Target: Cellphone A tt ac k on B il a t e r a l F ilt e r Prediction: Cowboy hat 92% Prediction: Sea Urchin 95% Prediction: Sea Urchin 99% Prediction: Sea Urchin 88% Groundtruth: Sea Urchin Target: Cowboy hat
Figure 5: Visualization of the adversarial images and perturbations for the targeted ISP attack. Each pair of rows (top to bottom) shows theattack on the Movidius Myriad 2, ARM Mali C71 and Bilateral Filter ISP respectively. In each targeted ISP attack, we show in the firstcolumn the adversarial RAW (top) and perturbations (bottom). The next four columns show the associated RGB images and perturbationsfrom the ISPs. The RGB perturbation is visualized by subtracting the ISP output of adversarial RAW to that of the unattacked output. argeted Optics Deployed Optics Fujinon CF12.5HA-1 Cooke TripletFujinon CF12.5HA-1 (cid:107) (cid:107)
Table 2: Success and transfer rate for the targeted physical opticsattack. Refer to Table 1 for table notation.
Display Image Fujinon CF12.5HA-1 Cooke Triplet
Tiger 95% Bee Eater 95% A tt ac k on F u ji non C F . HA - Groundtruth: Bee EaterTarget Class: Tiger Tiger 97%Bee Eater 93% A tt ac k on C ook e T r i p l e t Groundtruth: Bee EaterTarget Class: Tiger
Figure 6: Visualization of the targeted optics attack on the FujinonCF12.5HA-1 and Cooke Triplet optics. For each attack, we showthe displayed adversarial and post-processed images (top row). Inthe bottom row, we visualize (from left to right) the additive pertur-bations on the display image and its zoomed in × top-leftand bottom right region. Figure 5 shows the adversarial RAW images, perturba-tions (targeting on different ISPs) and their associated ISPoutputs. Interestingly, despite having the same adversarialRAW image as input, each ISP produces distinct RGB per-turbations. For example, in the attack on Movidius Myriad2, unlike other ISPs, the ARM Mali C71 suppresses the per-turbation around the top left black regions. Also, while theoutput RGB perturbations seem to contain similar macrostructures, only the one from the targeted ISP becomes ad-versarial to the classifier, while others pose no threat at all.Since the untargeted RGB perturbations do not change theprediction, it means that they are considered as noise bysome hidden projections in the classifier. As such, the per-turbations are specifically tailored to a specific ISP. In gen-eral, for each targeted ISP, our method is able to deceive thetarget pipeline with more than success rate . See Supplementary Document for results per ISP.
We extend the proposed method to target a compoundoptical module instead of a hardware ISP. The proxy func-tion now models the entire transformation from the dis-played image to optics, sensor, and ISP processing that re-sults in the final RGB image that is fed to the image classi-fier. In these experiments, all the pipelines deploy identical
ARM Mali C71 ISP, which allows us to assess adversar-ial pattern that targets only one optical system but not an-other. For each attacked image, we use (cid:15) = 0 . , target arandom class and use a total of 30 iterations, with the stepsize α = 0 . . We note that the value of (cid:15) is larger sincewe need to compensate for the attenuation loss during theacquisition process. We apply the same Algorithm 2 forthe targeted optics attack and show its success and transferrate in Table 2. The proposed method is able to achievea high success rate of 90% while keeping the transfer ratelesser than 10%. We visualize the attacks in Figure 6. Wefind that in both attacks, the perturbations show distinctivefrequency-dependent patterns. We interpret this attack asone that efficiently exploits the frequency bands specific tothe optical transfer functions of the employed optics.
6. Conclusion
In this work, we introduce the first method for findingadversarial attacks that deceives a specific camera ISP andoptics while leaving cameras with other ISPs or optics intactalthough they employ the same classifier network . Depart-ing from existing adversarial attacks, that assume camerapipelines to preserve adversarial perturbations, we proposean optimization method that employs a local proxy network,making it possible to attack embedded hardware ISPs thatare not differentiable and only available as black-box algo-rithms. We validate the method experimentally on recentautomotive camera ISPs and optics, achieving more than90% targeted success rate for both ISP and optics attacks.Building on the proposed methods, we envision not onlyresearch on defense mechanism to improve future imageprocessing and camera optics, but the method also sug-gests end-to-end multimodel sensor design as a potential av-enue to design systems resilient against adversarial attacks.
References [1] Anish Athalye, Nicholas Carlini, and David Wagner. Obfus-cated gradients give a false sense of security: Circumventingdefenses to adversarial examples. In
Proceedings of the In-ternational Conference on Machine Learning, (ICML) , 2018.2[2] Anish Athalye, Logan Engstrom, Andrew Ilyas, and KevinKwok. Synthesizing robust adversarial examples. In
Inter- See Supplementary Document for the untargeted optics attack. ational conference on machine learning , pages 284–293.PMLR, 2018. 1, 3, 6[3] Tejas Borkar, Felix Heide, and Lina Karam. Defendingagainst universal attacks through selective feature regener-ation. In Proceedings of the IEEE/CVF Conference on Com-puter Vision and Pattern Recognition , pages 709–719, 2020.1[4] N. Carlini and D. Wagner. Towards evaluating the robustnessof neural networks. In
IEEE Symposium on Security andPrivacy , pages 39–57, 2017. 1, 2[5] C. Chen, Q. Chen, J. Xu, and V. Koltun. Learning to See inthe Dark.
ArXiv e-prints , May 2018. 2[6] Jinghui Chen, Dongruo Zhou, Jinfeng Yi, and Quanquan Gu.A frank-wolfe framework for efficient and effective adversar-ial attacks. 2[7] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,Kevin Murphy, and Alan L Yuille. Deeplab: Semantic imagesegmentation with deep convolutional nets, atrous convolu-tion, and fully connected crfs.
IEEE transactions on patternanalysis and machine intelligence , 40(4):834–848, 2017. 1[8] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, andCho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substi-tute models. In
Proceedings of the 10th ACM Workshop onArtificial Intelligence and Security , pages 15–26, 2017. 2[9] Q. Chen, J. Xu, and V. Koltun. Fast image processing withfully-convolutional networks. In , pages 2516–2525,Oct 2017. 2[10] Shang-Tse Chen, Cory Cornelius, Jason Martin, and DuenHorng Polo Chau. Shapeshifter: Robust physical adversar-ial attack on faster r-cnn object detector. In
Joint EuropeanConference on Machine Learning and Knowledge Discoveryin Databases , pages 52–68. Springer, 2018. 3[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,and Li Fei-Fei. ImageNet: A large-scale hierarchical im-age database. In
IEEE Conference on Computer Vision andPattern Recognition , pages 248–255, 2009. 5, 6[12] Ranjie Duan, Xingjun Ma, Yisen Wang, James Bailey,A Kai Qin, and Yun Yang. Adversarial camouflage: Hidingphysical-world attacks with natural styles. In
Proceedings ofthe IEEE/CVF Conference on Computer Vision and PatternRecognition , pages 1000–1008, 2020. 3[13] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li,Amir Rahmati, Chaowei Xiao, Atul Prakash, TadayoshiKohno, and Dawn Song. Robust physical-world attacks ondeep learning visual classification. In
Proceedings of theIEEE Conference on Computer Vision and Pattern Recog-nition , pages 1625–1634, 2018. 1, 3[14] Qingnan Fan, Jiaolong Yang, David Wipf, Baoquan Chen,and Xin Tong. Image smoothing via unsupervised learning.
ACM Transactions on Graphics (Proceedings of SIGGRAPHASIA 2018) , 37(6), 2018. 2[15] M. Gharbi, G. Chaurasia, S. Paris, and F. Durand. Deep jointdemosaicking and denoising.
ACM Transactions on Graph-ics (TOG) , 35(6):191, 2016. 2 [16] M. Gharbi, J. Chen, J. Barron, S. Hasinoff, and F. Durand.Deep bilateral learning for real-time image enhancement.
ACM Trans. Graph. (SIGGRAPH) , 2017. 2[17] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.Explaining and harnessing adversarial examples. arXivpreprint arXiv:1412.6572 , 2014. 1, 2[18] Puneet Gupta and Esa Rahtu. Ciidefence: Defeating adver-sarial attacks by fusing class-specific image inpainting andimage denoising. In
Proceedings of the IEEE InternationalConference on Computer Vision , pages 6708–6717, 2019. 2[19] Kaiming He, Georgia Gkioxari, Piotr Doll´ar, and Ross Gir-shick. Mask r-cnn. In
Proceedings of the IEEE internationalconference on computer vision , pages 2961–2969, 2017. 1[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.Deep residual learning for image recognition. In
Proceed-ings of the IEEE conference on computer vision and patternrecognition , pages 770–778, 2016. 1[21] F. Heide, M. Steinberger, Y.-T. Tsai, M. Rouf, D. Pajak, D.Reddy, O. Gallo, J. Liu, W. Heidrich, K. Egiazarian, J. Kautz,and K. Pulli. FlexISP: A flexible camera image processingframework.
ACM Trans. Graph. (SIGGRAPH Asia) , 33(6),2014. 2[22] Zhichao Huang and Tong Zhang. Black-box adversarialattack with transferable model-based embedding. arXivpreprint arXiv:1911.07140 , 2019. 2[23] Andrew Ilyas, Logan Engstrom, Anish Athalye, and JessyLin. Black-box adversarial attacks with limited queries andinformation.
CoRR , abs/1804.08598, 2018. 1[24] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A.Efros. Image-to-image translation with conditional adver-sarial networks. In
Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR) , July2017. 1[25] Steve TK Jan, Joseph Messou, Yen-Chen Lin, Jia-BinHuang, and Gang Wang. Connecting the digital and phys-ical world: Improving the robustness of adversarial attacks.In
Proceedings of the AAAI Conference on Artificial Intelli-gence , volume 33, pages 962–969, 2019. 3[26] Hakki Can Karaimer and Michael S Brown. A softwareplatform for manipulating the camera imaging pipeline. In
European Conference on Computer Vision , pages 429–444.Springer, 2016. 2[27] Seon Joo Kim, Hai Ting Lin, Zheng Lu, Sabine S¨usstrunk,Stephen Lin, and Michael S Brown. A new in-camera imag-ing model for color computer vision and its application.
IEEE Transactions on Pattern Analysis and Machine Intel-ligence , 34(12):2289–2302, 2012. 2[28] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Ad-versarial examples in the physical world. arXiv preprintarXiv:1607.02533 , 2016. 1, 3, 6[29] Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Bo-qing Gong. Nattack: Learning the distributions of adversarialexamples for an improved black-box attack on deep neuralnetworks. arXiv preprint arXiv:1905.00441 , 2019. 2[30] Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang,Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks sing high-level representation guided denoiser. In Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition , pages 1778–1787, 2018. 2[31] Y Liu, X. Chen, C. Liu, and D. Song. Delving into transfer-able adversarial samples and black-box attacks, 2016. 1[32] Jiajun Lu, Theerasit Issaranon, and David Forsyth. Safe-tynet: Detecting and rejecting adversarial examples robustly.In
Proceedings of the IEEE International Conference onComputer Vision , pages 446–454, 2017. 1[33] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt,Dimitris Tsipras, and Adrian Vladu. Towards deep learn-ing models resistant to adversarial attacks. arXiv preprintarXiv:1706.06083 , 2017. 1, 2, 3[34] Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, andPascal Frossard. Sparsefool: a few pixels make a big differ-ence. In
Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition , pages 9087–9096, 2019. 2[35] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, andPascal Frossard. Deepfool: a simple and accurate method tofool deep neural networks. In
Proceedings of the IEEE con-ference on computer vision and pattern recognition , pages2574–2582, 2016. 1, 2[36] Preetum Nakkiran. Adversarial robustness may be at oddswith simplicity. arXiv preprint arXiv:1901.00532 , 2019. 1[37] Nina Narodytska and Shiva Prasad Kasiviswanathan. Sim-ple black-box adversarial perturbations for deep networks.
CoRR , abs/1612.06299, 2016. 1[38] Nicolas Papernot, Patrick McDaniel, and Ian Goodfel-low. Transferability in machine learning: from phenom-ena to black-box attacks using adversarial samples.
CoRR ,abs/1607.02533, 2016. 1[39] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow,Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practi-cal black-box attacks against machine learning. In
Proceed-ings of the 2017 ACM on Asia conference on computer andcommunications security , pages 506–519, 2017. 2, 4[40] Nicolas Papernot, Patrick McDaniel, Somesh Jha, MattFredrikson, Z Berkay Celik, and Ananthram Swami. Thelimitations of deep learning in adversarial settings. In , pages 372–387. IEEE, 2016. 1[41] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha,and Ananthram Swami. Distillation as a defense to adver-sarial perturbations against deep neural networks. In , pages 582–597. IEEE, 2016. 1[42] Sylvain Paris, Pierre Kornprobst, Jack Tumblin, and Fr´edoDurand.
Bilateral filtering: Theory and applications . NowPublishers Inc, 2009. 6[43] Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Be-longie. Generative adversarial perturbations. In
Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition , pages 4422–4431, 2018. 1[44] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmen-tation. In
International Conference on Medical image com-puting and computer-assisted intervention , pages 234–241.Springer, 2015. 3 [45] Dawn Song, Kevin Eykholt, Ivan Evtimov, Earlence Fernan-des, Bo Li, Amir Rahmati, Florian Tramer, Atul Prakash, andTadayoshi Kohno. Physical adversarial examples for objectdetectors. In { USENIX } Workshop on Offensive Tech-nologies ( { WOOT } , 2018. 3[46] Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai.One pixel attack for fooling deep neural networks. CoRR ,abs/1710.08864, 2017. 1[47] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, JoanBruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.Intriguing properties of neural networks. arXiv preprintarXiv:1312.6199 , 2013. 1, 2[48] Guanhong Tao, Shiqing Ma, Yingqi Liu, and XiangyuZhang. Attacks meet interpretability: Attribute-steered de-tection of adversarial samples. In
Advances in Neural Infor-mation Processing Systems , pages 7717–7728, 2018. 2[49] Vincent Tjeng, Kai Y Xiao, and Russ Tedrake. Evaluatingrobustness of neural networks with mixed integer program-ming. In
International Conference on Learning Representa-tions , 2018. 2[50] Ethan Tseng, Felix Yu, Yuting Yang, Fahim Mannan,Karl ST Arnaud, Derek Nowrouzezahrai, Jean-Franc¸oisLalonde, and Felix Heide. Hyperparameter optimizationin black-box image processing using differentiable proxies.
ACM Trans. Graph. , 38(4):27–1, 2019. 2, 3, 4, 5[51] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom,Alexander Turner, and Aleksander Madry. Robustness maybe at odds with accuracy. In
International Conference onLearning Representations , number 2019, 2019. 1[52] Chun-Chen Tu, Paishun Ting, Pin-Yu Chen, Sijia Liu, HuanZhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng.Autozoom: Autoencoder-based zeroth order optimizationmethod for attacking black-box neural networks. In
Pro-ceedings of the AAAI Conference on Artificial Intelligence ,volume 33, pages 742–749, 2019. 2[53] Eric Wong and Zico Kolter. Provable defenses against adver-sarial examples via the convex outer adversarial polytope.In
International Conference on Machine Learning , pages5286–5295. PMLR, 2018. 2[54] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan LYuille, and Kaiming He. Feature denoising for improvingadversarial robustness. In
Proceedings of the IEEE Con-ference on Computer Vision and Pattern Recognition , pages501–509, 2019. 1[55] Kaidi Xu, Sijia Liu, Pu Zhao, Pin-Yu Chen, Huan Zhang,Quanfu Fan, Deniz Erdogmus, Yanzhi Wang, and Xue Lin.Structured adversarial attack: Towards general implementa-tion and better interpretability. In
International Conferenceon Learning Representations , 2018. 2[56] Li Xu, Jimmy Ren, Qiong Yan, Renjie Liao, and Jiaya Jia.Deep edge-aware filters. In Francis Bach and David Blei,editors,
Proceedings of the 32nd International Conferenceon Machine Learning , volume 37 of
Proceedings of MachineLearning Research , pages 1669–1678, Lille, France, 07–09Jul 2015. PMLR. 2[57] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorfulimage colorization. In
European conference on computervision , pages 649–666. Springer, 2016. 1
58] Zhengyu Zhao, Zhuoran Liu, and Martha Larson. Towardslarge yet imperceptible adversarial image perturbations withperceptual color distance. In
Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition ,pages 1039–1048, 2020. 2,pages 1039–1048, 2020. 2