But first we need to explain why a threat actor would use a tool ill-suited for capturing footage instead of the target device’s regular camera. The reason is that such “ill-suited” sensors are usually totally unprotected. Let’s imagine an attacker tricked a user into installing a malicious program on their smartphone. The malware will struggle to gain access to oft-targeted components, such as the microphone or camera. But to the light sensor? Easy as pie.

So, the researchers proved that this ambient light sensor can be used instead of a camera; for example, to get a snapshot of the user’s hand entering a PIN on a virtual keyboard. In theory, by analyzing such data, it’s possible to reconstruct the password itself. This post explains the ins and outs in plain language.

“Taking shots” with a light sensor. Source

A light sensor is a rather primitive piece of technology. It’s a light-sensitive photocell for measuring the brightness of ambient light several times per second. Digital cameras use very similar (albeit smaller) light sensors, but there are many millions of them. The lens projects an image onto this photocell matrix, the brightness of each element is measured, and the result is a digital photograph. Thus, you could describe a light sensor as the most primitive digital camera there is: its resolution is exactly one pixel. How could such a thing ever capture what’s going on around the device?

The researchers used the Helmholtz reciprocity principle, formulated back in the mid-19^th century. This principle is widely used in computer graphics, for example, where it greatly simplifies calculations. In 2005, the principle formed the basis of the proposed dual photography method. Let’s take an illustration from this paper to help explain:

On the left is a real photograph of the object. On the right is an image calculated from the point of view of the light source. Source

Imagine you’re photographing objects on a table. A lamp shines on the objects, the reflected light hits the camera lens, and the result is a photograph. Nothing out of the ordinary. In the illustration above, the image on the left is precisely that — a regular photo. Next, in greatly simplified terms, the researchers began to alter the brightness of the lamp and record the changes in illumination. As a result, they collected enough information to reconstruct the image on the right — taken as if from the point of view of the lamp. There’s no camera in this position and never was, but based on the measurements, the scene was successfully reconstructed.

Most interesting of all is that this trick doesn’t even require a camera. A simple photoresistor will do… just like the one in an ambient light sensor. A photoresistor (or “single-pixel camera”) measures changes in the light reflected from objects, and this data is used to construct a photograph of them. The quality of the image will be low, and many measurements must be taken — numbering in the hundreds or thousands.

Experimental setup: a Samsung Galaxy View tablet and a mannequin hand. Source

Let’s return to the study and the light sensor. The authors of the paper used a fairly large Samsung Galaxy View tablet with a 17-inch display. Various patterns of black and white rectangles were displayed on the tablet’s screen. A mannequin was positioned facing the screen in the role of a user entering something on the on-screen keyboard. The light sensor captured changes in brightness. In several hundred measurements like this, an image of the mannequin’s hand was produced. That is, the authors applied the Helmholtz reciprocity principle to get a photograph of the hand, taken as if from the point of view of the screen. The researchers effectively turned the tablet display into an extremely low-quality camera.

Comparing real objects in front of the tablet with what the light sensor captured. Source

True, not the sharpest image. The above-left picture shows what needed to be captured: in one case, the open palm of the mannequin; in the other, how the “user” appears to tap something on the display. The images in the center are a reconstructed “photo” at 32×32 pixel resolution, in which almost nothing is visible — too much noise in the data. But with the help of machine-learning algorithms, the noise was filtered out to produce the images on the right, where we can distinguish one hand position from the other. The authors of the paper give other examples of typical gestures that people make when using a tablet touchscreen. Or rather, examples of how they managed to “photograph” them:

Capturing various hand positions using a light sensor. Source

So can we apply this method in practice? Is it possible to monitor how the user interacts with the touchscreen of a tablet or smartphone? How they enter text on the on-screen keyboard? How they enter credit card details? How they open apps? Fortunately, it’s not that straightforward. Note the captions above the “photographs” in the illustration above. They show how slow this method works. In the best-case scenario, the researchers were able to reconstruct a “photo” of the hand in just over three minutes. The image in the previous illustration took 17 minutes to capture. Real-time surveillance at such speeds is out of the question. It’s also clear now why most of the experiments featured a mannequin’s hand: a human being simply can’t hold their hand motionless for that long.

But that doesn’t rule out the possibility of the method being improved. Let’s ponder the worst-case scenario: if each hand image can be obtained not in three minutes, but in, say, half a second; if the on-screen output is not some strange black-and-white figures, but a video or set of pictures or animation of interest to the user; and if the user does something worth spying on… — then the attack would make sense. But even then — not much sense. All the researchers’ efforts are undermined by the fact that if an attacker managed to slip malware onto the victim’s device, there are many easier ways to then trick them into entering a password or credit card number. Perhaps for the first time in covering such papers (examples: one, two, three, four), we are struggling even to imagine a real-life scenario for such an attack.

All we can do is marvel at the beauty of the proposed method. This research serves as another reminder that the seemingly familiar, inconspicuous devices we are surrounded by can harbor unusual, lesser-known functionalities. That said, for those concerned about this potential violation of privacy, the solution is simple. Such low-quality images are due to the fact that the light sensor takes measurements quite infrequently: 10–20 times per second. The output data also lacks precision. However, that’s only relevant for turning the sensor into a camera. For the main task — measuring ambient light — this rate is even too high. We can “coarsen” the data even more — transmitting it, say, five times per second instead of 20. For matching the screen brightness to the level of ambient light, this is more than enough. But spying through the sensor — already improbable — would become impossible. Perhaps for the best.

The logo is stored in the code that runs immediately after the computer is turned on, in the so-called UEFI (Unified Extensible Firmware Interface) firmware. It turns out that this logo replacement function opens the way for the device to be seriously compromised — attackers can hack it and subsequently seize control of the system, and this can even be done remotely. The possibility of such an attack — named LogoFAIL — was recently discussed by experts at Binarly. In this article, we’ll try to explain it in simple terms, but let’s first recall the dangers of so-called UEFI bootkits.

UEFI bootkits: malware loaded before the operating system is

Historically, the program run upon turning on a PC was called BIOS (Basic Input/Output System). It was extremely limited in its capabilities, but nevertheless essential — tasked as it was with initializing the computer’s hardware and then transferring control to the operating system loader. Since the late 2000s, BIOS gradually began to be replaced by UEFI — a more sophisticated version of the same basic program but with additional capabilities, including protection against the execution of malicious code.

In particular, UEFI implements the Secure Boot feature, which employs cryptographic algorithms to check the code at each stage of the computer’s booting — from turning it on to loading the operating system. This makes it much more difficult to replace the real OS code with malicious code, for example. But, alas, even these security technologies haven’t completely eliminated the possibility of loading malicious code at an early stage. And if attackers manage to “smuggle” malware or a so-called bootkit into UEFI, the consequences can be extremely serious.

The issue with UEFI bootkits is that they’re extremely difficult to detect from within the operating system. A bootkit can modify system files and run malicious code in an OS with maximum privileges. And the main problem is that it can survive not only a complete reinstall of the operating system, but also replacement of the hard drive. Stashed in the UEFI firmware, a bootkit isn’t dependent on the data stored on the system drive. As a result, bootkits are often used in complex targeted attacks. An example of such an attack is described in this study published by our experts.

So, what do images have to do with it?

Since UEFI has fairly robust protection against the running of malicious code, introducing a Trojan into the boot process isn’t simple. However, as it turns out, it is possible to exploit flaws in UEFI code to execute arbitrary code at this early stage. There was good reason for the Binarly experts to pay attention to the mechanism that allows replacing the factory logo. To display the logo, a program is launched that reads data from the graphic image file and displays this image on the screen. What if we try make this program misbehave?

There are three major UEFI software developers: AMI, Insyde, and Phoenix. Each of them approaches logo processing differently. For example, Insyde has separate image processing programs for different formats — from JPEG to BMP. AMI and Phoenix consolidate the handling of all formats into a single program. Vulnerabilities were discovered in each of them, with a total of twenty-four critical errors. The final result of exploiting one of these errors is shown in this video:

LogoFAIL attack demonstration. Source

It’s all fairly straightforward: attackers can modify the image of the new logo as they please. This includes, for example, setting the logo resolution so that this parameter ends up beyond the limits defined in the handling code. This leads to a calculation error and ultimately results in data being written from the image file into the area for executable data. This data will then be executed with maximum privileges. The video above shows the seemingly harmless result of such a bootkit: a text file is saved to the Windows desktop. However, if malicious code has this level of access, the attacker can perform almost any action in the operating system.

Notably, some computer models from major manufacturers were not susceptible to this attack, and for a very simple reason: replacing the logo in their UEFI is essentially blocked. Among these models are a number of Apple laptops and Dell devices.

Dangerous implications for businesses

Theoretically, this attack can even be carried out remotely: in some cases, it would be enough to inject a specially prepared image into the EFI system partition on the system disk, and it would be processed on the next reboot. The catch is that performing such an operation already requires complete access to the system; that is, any data on the computer should already be available to the attackers. You might wonder then: what’s the point of implementing the LogoFAIL attack? The answer: to ensure that the malicious code survives even if the OS is reinstalled. This kind of persistence is usually highly desired by APT attack operators.

This problem will be resolved gradually by updated UEFI versions that fix errors in the image handlers. However, since not all companies diligently keep up with firmware updates, a huge number of devices will likely remain unprotected. And the list of vulnerable devices includes not only laptops but also some server motherboards. This means that Binarly’s research should be taken very seriously.

According to Intel representatives, the company’s engineers were aware of the processors’ abnormal behavior, but the issue was considered non-critical, and plans to resolve it were postponed to the first half of 2024. However, the situation changed when Google researchers discovered the problem independently. In fact, all of the details about the vulnerability actually come from Google specialists, specifically from this article by Tavis Ormandy.

Processor fuzzing

Tavis Ormandy has discovered numerous major vulnerabilities in various programs and devices. Recently, we wrote about his previous research that found the Zenbleed vulnerability in AMD processors. On that occasion, Tavis talked about adopting fuzzing to find hardware vulnerabilities.

Fuzzing is a testing method that involves feeding random information into the input of the information system being tested. Usually, it’s used to automate the search for software vulnerabilities: a special fuzzing tool is created to interact with the program and monitor its state. Subsequently, tens or hundreds of thousands of tests are conducted to identify unusual behavior in the tested code.

When it comes to testing processors, things are a bit more complicated. We have to generate random programs that operate with no failures of their own and run them on the processor. How can we differentiate normal processor behavior from abnormal behavior in such a case? After all, not every error during software execution leads to a crash. Ormandy proposed a technique in which the same “random” code is simultaneously executed on different processors. Theoretically, the output of an identical program should also be identical; if it isn’t, it could indicate a problem. It was this approach that revealed the vulnerability in the Intel processors.

Useless but dangerous code

To understand how the Reptar vulnerability works, we need to go down to the lowest level of programming — the machine code that processors execute directly. Assembly language is used to represent such basic instructions in a more convenient way. A snippet of assembly language code looks something like this:

Example of code in assembly language. The last line contains a prefix that modifies execution of the instruction following it. Source

The last line features the movsb instruction, which tells the processor to move data from one memory area to another. It’s preceded by the rep modifier, which indicates that the movsb command should be executed several times in a row. Such prefixes are not relevant for all instructions. Intel processors know how to skip meaningless prefixes. Tavis Ormandy gives an example:

Multiple repeated prefixes won’t cause an error when executing the program. Source

Let’s add another prefix, the so-called rex.rxb. It was introduced alongside the x86-64 architecture to handle eight additional processor registers. Although what exactly it does is not that important — all we need to know is that this prefix doesn’t make sense when used with the movsb command:

In theory, the rex.rxb prefix should be skipped, and only the movsb command with the rep prefix executed. But in practice, this isn’t the case for Intel processors. Source

In fact, this prefix changes the behavior of Intel processors (starting from Ice Lake), although it shouldn’t. In this generation of processors, a technology called “Fast Short Repeat Move” was added. It’s designed to accelerate operations involving data movement in RAM. Among other things, this technology can optimize the execution of the rep movsb instruction. Along with the “Fast Short Repeat Move” feature, a flaw crept into the processor’s logic, first discovered by Intel engineers and later by Google experts.

Immediate threat

What could executing this instruction, which disrupts the normal behavior of the processor, lead to? According to Ormandy, the results are unpredictable. The researchers observed execution of random code, parts of the program being ignored, and various failures in the processor, all the way up to complete failure. For the latter, one needs to somehow exploit the vulnerability on a pair of processor cores simultaneously. To check their own systems for this vulnerability, a team of Google researchers prepared a test program.

Unpredictable behavior is bad enough. The most important difference between this “processor bug” and all the others is that it directly threatens providers of virtual private server hosting services, or cloud solution providers in general. This industry is built on the ability to share a single powerful server among dozens or hundreds of clients — each managing their own virtual operating system. It’s crucial that under no circumstances should one client see another client’s data or the data of the host — the operating system managing the virtual containers.

Now imagine that a client can execute a program in their virtual OS that causes the host to crash. At the very least, this could enable a DoS attack on the provider. In fact, Ormandy didn’t present any other exploitation scenarios, citing the fact that it’s very difficult to predict the behavior of a processor operating in black-box mode; although it’s theoretically possible for an attacker to execute specific malicious code instead of relying on random failures. Intel representatives themselves acknowledge that “code execution” and “information disclosure” are possible. Therefore, it’s extremely important to install microcode updates prepared by Intel (for virtual hosting service providers at least).

Optical stabilization and the side-channel attack

Optical stabilizers provide higher-quality images when shooting videos and photos. They smooth out hand tremors, camera shake while walking, and similar undesirable vibrations. For this stabilization to work, the manufacturers ensure that the camera’s sensor is movable relative to the lens. Sometimes the lenses within the camera itself are also made movable. The general idea of optical stabilization is illustrated in the image below: when motion sensors in a smartphone or camera detect movement, the matrix or lens in the camera moves so that the resulting image remains steady. In this way, most small vibrations don’t affect the final video recording.

Diagram of the optical image stabilization system in modern cameras. Source

Understanding exactly how such stabilization works isn’t necessary. The important thing is that the camera elements are movable relative to each other. They can shift when necessary — aided by miniature components known as actuators. However, they can also be moved by external vibrations —such as those caused by loud sounds.

Imagine your smartphone lying on a table near a speaker, recording a video (without sound!). If the speaker’s loud enough, the table vibrates, and along with it, the phone and these very components of the optical stabilizer. In the recorded video, such vibrations translate into microscopic shaking of the objects captured. During casual viewing, this trembling is completely unnoticeable, but it can be detected through careful analysis of the video data. Another problem arises here: a typical smartphone records video at a rate of 30, 60, or at best 120 frames per second. We only have that much opportunity to capture slight object shifts in the video — and it’s rather little. According to the Nyquist-Shannon sampling theorem, an analog signal (such as sound) of a given frequency can be reconstructed from measurements taken at twice that frequency. By measuring the “shake” of an image at a frequency of 60 hertz, we can at best reconstruct sound vibrations with a frequency of up to 30 hertz. However, human speech lies within the audio range of 300 to 3400 hertz. It’s a mission impossible!

But another feature of any digital camera comes to the rescue: the so-called rolling shutter. Each frame of video is captured not all at once, but line by line — from top to bottom. Consequently, when the last line of the image is “digitized”, fast-moving objects in the frame may already have shifted. This feature is most noticeable when shooting video from the window of a fast-moving train or car. Roadside posts and poles in such a video appear tilted, while in reality they’re perpendicular to the ground. Another classic example is taking a photo or video of a rapidly rotating airplane or helicopter propeller.

The relatively slow reading of data from the camera sensor means that the blades have time to move before the frame is completed. Source

We’ve shown a similar image before — in this post about an interesting method of attacking smart card readers. But how can this rolling shutter help us analyze micro-vibrations in a video? The number of “samples”, meaning the frequency at which we can analyze the image, significantly increases. If video is recorded with a vertical resolution of 1080 pixels, this number needs to be multiplied by the number of frames per second (30, 60, or 120). So we end up being able to measure smartphone camera vibrations with much greater precision — tens of thousands of times per second, which is generally enough to reconstruct sound from the video. This is another example of a side-channel attack: when the exploitation of an object’s non-obvious physical characteristics leads to the leakage of secrets. In this case, the leakage is the sound that the creators of the video tried to conceal.

Difficulties in practical implementation

But… not so fast, tiger. Let’s not assume that, with this complex video signal processing, the authors of the study managed to restore clear and intelligible human speech. The graph on the left shows the original spectrogram of the audio recording, in which a person sequentially says “zero”, “seven”, and “nine”. On the right is the spectrogram of the sound restored from the video recording. Even here, it’s clear that there was significant loss in the restoration of the data. On the project’s website, the authors have provided real recordings of both the original and restored speech. Check out the results to get a clear idea of the shortcomings of this sophisticated eavesdropping method. Yes, some sound can be reconstructed from the video — but it’s more just a kind of weird rattling than human speech. It’s very difficult to guess which numeral the person is uttering. But even such heavily damaged data can be successfully processed using machine learning systems: if you give the algorithm known pairs of original and restored audio recordings to analyze, it can then infer and reconstruct unknown data.

Restoring audio from video using the rolling shutter effect. Source

The success of the algorithm is tested on relatively simple tasks — not on real-life human speech. The results are as follows: in almost 100% of cases, it was possible to correctly determine a person’s gender. In 86% of cases, it was possible to distinguish one speaker from another. In 67% of cases, it was possible to correctly recognize which digit a person was naming. And this is under the most ideal conditions when the phone recording the video was placed 10 centimeters away from the speaker on a glass tabletop. Change the tabletop to wood, and the accuracy starts to decrease. Move the phone farther away — it gets even worse. Lower the volume to the normal level of a regular conversation, and the accuracy drops critically.

Now, let’s move on from theoretical considerations and try to imagine the real-life applications of the proposed scenario. We have to immediately exclude all “eavesdropping” scenarios. If a hypothetical spy with a phone can get close enough to the people having a secret conversation, the spy can easily record the sound with a microphone. What about a scenario where we record the people talking on a surveillance camera from a distance, and the microphone cannot capture the speech? In this case, we likewise won’t be able to reconstruct anything from the video: even when the researchers moved the camera away from the speaker by three meters, the system basically didn’t work (the numerals were correctly recognized in about 30% of cases).

Therefore, the beauty of this study lies simply in finding a new “side channel” of information leakage. Perhaps it will be possible to somehow improve the proposed scheme in the future. The authors’ main discovery is that the image stabilization system in smartphones, which is supposed to eliminate video vibrations, sometimes carefully records them in the final video. Moreover, this trick works on many modern smartphones. It’s enough to train the algorithm on one, and in most cases, it will be able to recognize speech from video recorded on another device.

Anyway, if somehow this “attack” is dramatically improved, then the fact that it analyzes recorded video becomes critical. We can fantasize about a situation in the future where we could download various videos from the internet without sound and find out what the people near the camera were talking about. Here, however, we face two additional problems. It wasn’t for nothing that the authors produced the speech from a speaker placed on the same table as the phone. Analyzing real human speech using this “video eavesdropping” method is much more complicated. Also, phone videos are usually shot handheld, introducing additional vibrations. But, you must agree, this is an elegant attack. It once again demonstrates how complex modern devices are, and that we should avoid making assumptions when it comes to privacy. If you’re being filmed on video, don’t rely on assurances that “they’ll change the audio track later”. After all, besides machine learning algorithms, there’s also the ancient art of lip reading.

About compression algorithms

Before delving into the GPU.zip attack itself, let’s discuss some aspects of compression algorithms. These algorithms can be broadly categorized into lossy compression algorithms (like MP3) and lossless compression algorithms (like RAR or ZIP). The latter ones compress data in such a way that it can be completely restored. The simplest method of compression is to store repeating data only once, and then indicate where specific sets of characters or numbers should be placed. For example, the length of this post could be significantly reduced by recording all the places where the word “data” appears and storing the word itself only once.

From an information security perspective, compression algorithms have a vulnerability of sorts. Let’s imagine that we’re transmitting some data over the internet using compression. The volume of information depends on how effective the compression algorithm is — the better the compression, the smaller the data size. Back in 2002, it was shown that this feature could be exploited to steal secrets even if the data is encrypted. One of the relatively practical attacks confirming this possibility was demonstrated in 2012.

It was found that in some cases, if information between a browser and a server is transmitted simultaneously in both compressed and encrypted forms, the compression algorithm could reveal secret information even if the encryption algorithm isn’t hacked. If attackers can send numerous requests to the server, they can observe how the size of compressed data changes based on the content. And from this they can calculate the secret information character by character. It remains to be seen whether unchecked compression of graphics subsystem data can also lead to leakage of secrets.

About the features of computer graphics

Today, we’re discussing the graphics subsystem — or, simply put, video cards, although they’re often integrated directly into the processor. Discrete GPUs are separate computational modules, usually with their own RAM. Computer gamers are familiar with the situation when the latest cool game struggles to run on a not-so-powerful video card: the frame refresh rate drops below optimal, the image is no longer smooth, and sometimes it even freezes for a fraction of a second. There can be two reasons for such behavior. Most often, the video card can’t handle the calculations required to create 3D images quickly enough. Sometimes, however, the required data is transmitted too slowly from the main RAM to the graphics subsystem memory.

This problem can be solved by using data compression algorithms. Games use lossy compression algorithms to compress textures. The authors of the paper found that, at least in Intel and AMD integrated GPUs, lossless compression algorithms are used as well — to transmit any graphic information that needs to be displayed on the screen (desktop, browser windows, and so on). These algorithms cannot be disabled and, moreover, are proprietary – no one but the manufacturer knows how they work. The researchers studied them in “black box” mode: the very existence of the compression algorithm was determined based on indirect signs, such as the amount of data transferred from RAM to video memory, which varied depending on the image. Transmitting graphic patterns made entirely of black pixels, black and white pixels in a specific order, and random patterns, showed that when easily compressible data is sent to the video system, less information is transferred between the main RAM and video memory: exactly the way data compression should work.

Most of the study is dedicated to reverse engineering these proprietary data compression algorithms. This research was deemed necessary to understand exactly how such algorithms work — for example, how graphics information is divided into blocks before compression. The researchers found that different algorithms are applied depending on the manufacturer or even the model of the graphics subsystem.

The problem is that the time it takes to compress data also depends on the data itself. If we have a poorly compressible set of information (random data without any repeating elements), the processing time will differ compared to “simple data”. Meanwhile, an attacker can measure this time — for instance, by creating a special webpage.

The beauty… and uselessness of the GPU.zip attack

Imagine someone creating a “malicious” webpage that also contains a request to embed another page from which they want to steal data. This person has the ability to measure the time it takes to render their page in the browser, but nothing more. If, for example, a window with the target’s work email is embedded in the page, the attacker won’t gain access to the content of that window. Why? Such an action is strictly prohibited by the same-origin policy rule — you can place code on a site to track user actions, for example, but it won’t work on the embedded “foreign” webpage. There is one exception, however: styling rules can be applied to the embedded page.

The authors of the GPU.zip attack took advantage of this and began applying specific graphics patterns to the target page. This led to changes in the time required to process compressed graphic data, thereby slightly altering the duration of page rendering. Which can be measured.

The original image and its “stolen” copies extracted using the features of Intel and AMD graphics systems. Source

We’ve finally reached the practical implementation of this attack. Here’s how it works: the attacker somehow lures the user to the malicious webpage. The page contains code embedding another page from a completely different site — in this case, Wikipedia’s main page. Let’s assume the browser user has a Wikipedia account and is logged in. Their username will be displayed on the embedded page. By applying effects to this page and measuring the time it takes to render, the attacker can reconstruct the content of the target page from this single parameter alone. More specifically, the attackers can obtain the username. In this way, they can identify the visitor of their malicious site — even if the visitor tries to remain anonymous, for example.

This is a typical side-channel attack: the attacker uses an indirect parameter that they can measure (the time it takes to render a web page) to steal data they don’t have access to. But now, let’s discuss the impracticality of this attack…

The content of the target web page is reconstructed pixel by pixel. The attacker has a timer and the ability to slightly modify the appearance of the page in the browser. As a result, it takes half an hour on an AMD Ryzen processor with integrated graphics to reconstruct not the entire page but only a small piece, as shown in the screenshot above. On an Intel processor, the algorithm works even slower — the reconstruction takes more than three hours!

This means the potential victim has to open the page and forget about it for quite a while, without closing it. During all this time, the page will be refreshing, which puts a heavy load on the system. However, the accuracy of the data reconstruction is quite high (97-98%) and, most importantly, the method works even when a large volume of other data is transmitted through the video card. The researchers had a YouTube video playing in the background. Unlike previous studies, this attack works reliably even with a significant amount of such “background noise”.

The final argument against the feasibility of this particular attack is that most websites cannot be embedded into other web pages if they display confidential content. This means that you can’t sneakily “screenshot” email messages or chat conversations in this way. The example with the Wikipedia page was actually chosen because it’s a rare case where a website with a visible username can be embedded.

To sum it up. Unlike other hardware vulnerabilities, it can’t be said with certainty that GPU developers made a mistake in this case. We’re talking about extremely complex interactions among different components — the properties of which can be exploited to steal data. The theft itself is not mind-blowing yet, but further research may well discover a more effective method. We hope that GPU developers will take this study into account and adapt their algorithms so that they don’t leak sensitive information.

The quality of this study should not be underestimated either. Forgetting all the practical difficulties for a moment, the researchers essentially demonstrated a method of remote data theft and took screenshots of secret information. All this was achieved through a detailed examination of a minor feature in the operation of GPUs — and manufacturers tend not to publicize anything about the operation of their CPUs. Still, it’s an impressive piece of research — even if it has no practical consequences… for now.

The flaw stems from a combination of fairly harmless AMD CPU features. It turns out that, if combined, a certain interaction with CPU registers and a perfectly normal system of speculative code execution may result in a leak of secret data. In theory, it is fairly easy to steal information using this vulnerability (unique ID CVE-2023-20593), and at quite a high speed, too: up to 30 kBps for each one of the CPU cores. So far, no real exploitation cases have been reported. On the other hand, patches (CPU microcode updates) are available just for part of the affected CPUs. AMD promises to solve the problem completely by the end of 2023.

Zenbleed exploitation details

As was mentioned before, Zenbleed exists thanks to the speculative execution system. The vulnerability is not easy to explain. In his blogpost, Tavis Ormandy presents cold facts that only an experienced low-level coding pro can get to the bottom of. In a nutshell, here is one of the instruction sets for Zenbleed exploitation:

Demo code for Zenbleed exploitation. Source: Source

A GitHub description by the Google Information Security team sheds some light on the nature of the problem. For the past 15 years, Intel and AMD CPUs have been using the instruction extension set AVX. Among other things, these instructions support 128- and 256-bit vector registers. To put it really simple, CPU registers are used for temporary storage of data when executing instructions. In some cases, being able to store sufficiently large amounts of data in vector registers allows to considerably improve performance. The 128 bit (XMM) and 256 bit (YMM) registers are commonly used for the most routine operations, such as related to read/write from/to RAM.

Concurrent use of 128 and 256 bit registers brings another set of problems. If used simultaneously within the same task, XMM registers are automatically converted into YMM registers. This is where the zeroing of the upper “half” of the YMM register is routinely performed. The special instruction for that is vzeroupper. All registers are stored in the so-called register file and are used in turns by different programs run on the computer.

What is common between Zenbleed and Use After Free?

If you create conditions for the vzeroupper instruction to be executed speculatively, the operation will end incorrectly in AMD Zen 2 CPUs. CPUs can execute instructions without waiting for the results of the previous calculations based on branch prediction. This accelerates the work a great deal but can also result in a situation where instructions are executed “in vain”, not being required by the program logic. If that happens, the instruction execution results must be rolled back. Thus, if vzeroupper is executed “in vain”, the zeroing out of one half of the YMM register must be canceled.

This is where a logic error comes into play in Zen 2 CPUs. The register remains in the so-called “undefined” state. Which means, it may still contain pieces of data from other programs that use the shared register file. In a normal situation, no actors should have access to this data. Zenbleed creates conditions where malware can “monitor” the information that goes through vector registers.

In a sense, such CPU behavior closely resembles the typical software error known as use after free. It is when one program uses a certain RAM area to store its data, and then vacates this RAM area making it available to other applications. As a result, a third program can read such data, which can potentially contain secret information. Yet in the Zenbleed case, it is not a software error but a hardware one.

Impact assessment

In theory, Zenbleed allows to read secrets directly, and does it at rather a high speed. This doesn’t mean much by itself: things like what data can be read, or whether it can be used in harmful ways, depend on a given situation. Only applications that use XMM and YMM at the same time are affected by this vulnerability. First of all, these are Linux system libraries and the Linux kernel itself, as well as cryptographic libraries and systems like OpenSSL. Also, getting information requires the application to be data-intensive. In order for an attacker to get something really useful, it is necessary to run some encryption process on the affected computer, or to actively use the browser for web-surfing, otherwise the exploitation of the vulnerability will be in vain.

We have only been shown the demo code, the proof of concept. It was beyond the scope of the study to demonstrate a really harmful scenario. According to Cloudflare team, the issue is fairly easy to exploit. One could do it even using a browser. We could imagine an attacker sending their victim a link to a pre-built web page to steal passwords to sensitive accounts from the memory cache. The saddest part is, a theft like that wouldn’t even leave any traces. Can it be pulled off in real life? We don’t know it yet.

But we do know that Zenbleed is most dangerous in a corporate environment. Just imagine a situation where a virtual server renter can read data from other servers and even the hypervisor, provided they use the same CPU cores. This is why the very first patch to be released was addressing AMD EPYC server CPUs.

Future of hardware security

In the closing part of his article, Tavis Ormandy imparts that he had discovered the problem thanks to fuzzing. As applied to software testing, fuzzing normally means “feeding” the program random data in search of a situation where a specific set of such data causes some abnormal behavior. Here we have a more sophisticated challenge: hardware fuzzing (let’s call it that) implies creating programs employing a random set of instructions in search of an abnormal CPU response. An abnormal termination of such a program is not necessarily an issue signal as such. Ormandy proposes several methods for anomaly detection, such as running the same code on different CPUs — if identical programs demonstrate different behavior, it prompts an investigation to make sure no CPU logic error is involved.

The history of hardware vulnerabilities suggests that usually it is not enough to close one problem alone. After the patch is applied, a new way to circumvent the new defense system can be found, for the problem is in the fundamental CPU operation principles. That is why Tavis Ormandy has not only found an AMD Zen 2 CPU vulnerability, he has also proposed some interesting strategies on how to locate other potential errors.

Potentially dangerous as it may be, Zenbleed is not likely to be used to attack individual users. But the server infrastructure of organizations is a different story. In this particular case, you might say it was a narrow escape: the problem was found and patched up with a microcode update, with only a minor performance drop. If your infrastructure uses AMD Zen 2 CPUs, you really need this patch, too. But chances are, this research will be followed by others. The whole attitude to hardware security may be revised, with new comprehensive security tests (employing both fuzzing and other strategies) coming into the picture. Let’s hope hardware vendors will be able to use them to good advantage. But organizations still need to integrate the risk posed by emerging similar vulnerabilities into their security models.

How RAM works

RAM chips have never been very reliable. Loosely speaking, each memory cell, in which one bit of information is stored, is a miniature battery. When we charge it, we write a “one” to the cell. If there’s no charge, it’s “zero”. And this happens… millions of times per second! In modern microchips, these cells are densely packed to a phenomenal degree: billions of them all on a fingernail-sized crystal. With the high update rate and extreme miniaturization of electronic components, sooner or later failures are inevitable — the miniature “battery” will lose its charge, and the one will turn into a zero. Sometimes failures are caused by external factors, say, if the memory chip is exposed to heat or even cosmic rays.

Such failures can lead to critical errors. Imagine that a program stores an address in RAM that needs to be accessed when certain conditions are met. If some bits in this address spontaneously flip from one to zero, instead of your code, there’s no knowing what will get executed. Many technologies are used to avert failures;  for example, a forced update of the contents of memory cells: sequential reading/writing of information — even if neither the CPU nor the program need it right there and then. The data-reading process is destructive, so after being accessed, information must be overwritten. There’s also an error correction mechanism: the memory stores both data and separate information for checking the data’s correctness. It’s important to understand that the high density of memory cells in modern computers is a fundamental feature; they won’t work any other way.

RowHammer attack

But back to the 2014 RowHammer report. Researchers from both Carnegie Mellon University and Intel demonstrated how to exploit the above-described features of dynamically updated RAM to do harm. If data reading is destructive and is followed by overwriting, what happens if we write a program that reads tens or hundreds of thousands of times per second? This process is what researchers call “hammering”.

Schematic representation of the structure of RAM cells. Source

Memory cells are arranged as a matrix, and any operation on a particular cell involves accessing a whole range of them. It turned out that sequential, repeated access to a row of cells affects neighboring rows. If this operation is performed a great many times, the ones and zeros in the cells of the neighboring row can flip. The 2014 study showed that such an attack was possible on the then-standard DDR3 memory modules.

Why is this dangerous? Imagine a hacker can execute some arbitrary code on the targeted system, but without special privileges. In an extreme case, it might even be the code of a web page, with a link sent to the victim. If this code is allowed to “hammer” a certain area of RAM, it can cause read disturbance in neighboring cells, where, say, operating system data might be stored.

In 2015, Google researchers showed how RowHammer could be used to gain unlimited access to a computer’s RAM. This is a highly complex attack with many unknowns: it’s still necessary to somehow get into the right memory area and cause the “right” data corruption so that the computer doesn’t freeze and the program doesn’t stop working. Nevertheless, the theoretical possibility of such an attack was confirmed.

BlackSmith: bypassing RowHammer protections

How to protect data from a RowHammer attack? The simplest way is by forcing an update of information in neighboring rows after a request to read data from a memory cell row. This significantly reduces the likelihood of data corruption. As in the case of hardware vulnerabilities in CPUs, sooner or later a problem is discovered in every protection method.

In 2021, researchers demonstrated a BlackSmith attack, which showed that under certain conditions it’s possible to cause failures even with RowHammer protections in place. How exactly did they do this? What if, instead of “hammering” the row of memory cells next to the target one, someone tries different combinations: interrogate the rows above and below the target hundreds of thousands of times, or attack four rows at once in a certain sequence? It worked. Why? Because the fundamental problem (the high density of memory cells) hasn’t gone anywhere!

RowPress: increased attack effectiveness

The new RowPress attack is even more effective, despite using the same basic principle — with one slight but crucial change: the attacker tries to keep one row of cells open for reading as long as possible. The researchers managed to find another standard feature of how memory chips and the controller operate that causes even more disturbance affecting neighboring rows of memory cells. In terms of effectiveness (measured by the number of “hammerings” required — the fewer, the better), RowPress is tens or even hundreds of times more potent than RowHammer. In a few marginal cases, the desired bitflip was achieved after a single read operation on neighboring data.

Test flowchart for investigating the operation of RAM modules. Source

How serious a problem is this? The likelihood of a RowHammer, Blacksmith or RowPress attack on home users is miniscule. At risk are companies. In theory, these attacks could target the memory of servers running in public clouds. After all, providers give access to their servers, allocating a kind of virtual machine for users to run any code they like. They must ensure that this machine has no way of escaping from its virtual environment and gaining access to other customers’ data. Roughly speaking, such a virtual system is a program able to read and write data to a server’s RAM; in other words — a ready-made platform for attacking the memory of a physical server.

How theoretical such an attack is can be seen from the photo of the test setup used to study RowPress. The memory module has been moved to a separate board. Connected to it is some sort of debugging device for fine-tuning RAM operation. Some protection systems have been disabled. Most importantly, a heater is installed on the module with memory chips, raising the temperature to 50 or even 80 degrees Celsius, which in itself increases the chances of accidental or intentional data corruption.

Hardware attack specifics

Comparing RowPress with the earlier RowHammer, we basically see a slight modification to the memory access method that allows attackers to bypass (including on a real system, without heating or “cheating”) protections implemented by module manufacturers. The researchers proposed their own solution to this problem, which, fortunately, has little impact on performance. However, as with most hardware vulnerabilities, getting rid of them entirely is unrealistic. Reducing the density of today’s memory chips is not an option. On the contrary, their capacity only grows and grows.

Nor would implementing “reliable” error correction resolve the issue, since that would take up as much as a third of the RAM. The usual method based on error correcting codes (ECC) makes attacks less effective, but does not eliminate them. Therefore, it’s safe to say that RowPress won’t be the last “hammer” attack we see.

On the plus side, such studies remain, for the time being, a largely theoretical exercise. Researchers are finding new attack vectors, and device makers are coming up with new defenses. Sure, it’s possible that eventually they’ll accidentally discover a vulnerability with the potential for mass exploitation. However, judging by the history of such studies over the past decade, that doesn’t seem likely.

But nor should such research be dismissed as purely theoretical and abstract: what lab-based experts can do today, real cybercriminals may be capable of doing tomorrow — or in five or ten years. As for cloud service providers, they need to be clued-up on the latest developments right now, and factor them into their security models firmly.

Video-based cryptanalysis attack

Let’s imagine a real attack using this technology. It might go as follows. Intruders gain access to the video surveillance system in a corporate building. One of the cameras is pointed at the door of a high-security storage room for, say, secret documents. On the door is a smart card reader. A company employee approaches the door and inserts their card; the lock opens.

The smart card contains a microchip that talks to the reader. To verify the key encoded in the smart card against the data in the reader, the latter performs a cryptographic algorithm — that is, a series of calculations. In a normal situation, it’s extremely difficult to make a copy of such a key — even if you manage to get hold of the original. But there’s a vulnerability: the reader’s LED. This indicates that the device is working, changing color from red to green when a visitor is authorized and the door can be opened.

The brightness of the indicator, which depends on the load on the reader’s systems, changes during the cryptographic operation: for example, when the reader’s processor is busy with calculations, the LED brightness dips slightly. By analyzing these slight alternations, it’s possible to reconstruct the private key and thus create a smart card that opens the door to the secret room. If intruders have access to a video camera and can get video footage of the reader’s LED, in theory, they can then hack into the building’s security system.

Practical difficulties

Putting theory in papers such as this one into practice is never easy. The above scenario is unlikely to be implemented in practice any time soon. And for hardware-security pros, this vulnerability is nothing new. It’s a case of a classic side-channel attack: data leakage through some non-obvious process in a device’s operation. A traditional, decades-old method of attacking smart cards and other devices that use data encryption algorithms is to measure the voltage on the device. During the calculations, this voltage changes slightly. By observing these slight changes, an attacker might be able to reverse-engineer the algorithm: for example, by linking a voltage drop with a certain value being processed.

A feature of this kind of side-channel attack is that the calculations are very fast. To reconstruct the encryption key, one would have to measure the voltage hundreds or even thousands of times per second. But the LED is a part of the overall power-supply circuit of the device, which means that its brightness varies with the voltage. Check out the progress: the attack no longer requires complex and expensive equipment. No need to take the device apart and solder wires to the circuit board. Just point the video camera at the LED, make a recording, analyze it and get the result.

Video analysis features

When analyzing video footage in practice, the authors of the paper encountered numerous difficulties. A regular camera shoots video at 60 frames per second, while an advanced one — no more than 120. For an attack on an encryption algorithm, that’s way too low. To improve the method, the researchers exploited an inherent flaw in any digital camera that manufacturers typically struggle to overcome: the so-called rolling shutter. When we press the shutter button, the camera’s image sensor doesn’t take an instantaneous snapshot. What happens is that the sensor’s pixels are scanned sequentially, line by line, from top to bottom. If we photograph an object moving at high speed, this can produce artistic effects. Think of the rotating blades of a helicopter.

While the digital camera sequentially reads data from the image sensor, the propeller has time to shift. The result is a distorted image. Source

The researchers proceeded as follows: they moved the camera closer to the LED so that it filled almost the entire area of the frame. Then they measured the brightness, not of the whole frame, but of each line. Hence, the “measurement frequency” — and thus the accuracy — got a massive boost: up to 61,400 times per second in the case of the iPhone 14 camera. Curiously, in this rather atypical camera usage scenario, the iPhone got the better of Samsung in terms of the amount of data recorded.

Testing smartphones’ ability to capture small changes in LED brightness at a certain frequency. Source

The above screenshot shows the result of this improved configuration: the researchers made the LED blink at a certain frequency, gradually increasing it. The switching on and off of the LED is clearly visible by measuring the power fluctuations (top left). The change in LED brightness can be seen clearly when using a specialized photosensor (top right). The test used two smartphones as video cameras (bottom row). As we can see, they recorded the blinking of the LED at a fairly low frequency; higher frequency changes in brightness were lost. But at the base frame-rate (60 or 120 times per second), even these results would be out of reach. This improvement turned out to be sufficient for a successful attack.

Attack results

To prove the possibility of a “video attack” in practice, the researchers showed how a private encryption key can be exfiltrated from various smart card readers. In each of the five experiments, the LED was made to blab its secrets. In theory, it’s not even necessary for the smartphone or video camera to be near the device; under certain conditions, the signal can be recorded from up to 60 meters away. The team also found that the presence of other light sources (a sun-lit room, for example) impacted the measurement accuracy. This complicates the attack by adding noise to the useful signal, but is not critical to the outcome.

But that’s not all. The researchers attacked a Samsung Galaxy S8, which was running the SIKE data encryption algorithm.

Exfiltrating secret data from a smartphone. Source

A smartphone has no LED that lights up when the device is turned on. But the researchers got crafty: they took the phone and connected portable speakers that did have an LED! Because the speakers were powered by the smartphone, the scenario whereby LED brightness depends on processor load was replicated.

The result was a Hertzbleed attack on the phone (see here for details), confirming that processors based on ARM architecture are also at risk of this type of attack (the original paper covered only vulnerabilities in Intel and AMD chips with x86 architecture).

We’re now talking about a wholly theoretical experiment, not about stealing any actual encryption keys in use. Cryptographic calculations were triggered on the smartphone according to a certain algorithm. By measuring the brightness of the LED on the speakers connected to the phone, the researchers managed to capture the operation of the algorithm and recover the private key. Since a smartphone is far more complex than a smart card reader, the noise level in the measurements was much higher. Nevertheless, the experiment was successful.

On the usefulness of this scientific research

This study will unlikely be applied in practice any time soon. The attack is highly complex, and it’s hard to find a realistic use case for it. As is often the case, the interest value of such papers does not lie in the practical exploitation of “discovered vulnerabilities”. Perhaps the key takeaway of this study is that the specialized equipment once needed for side-channel attacks has been replaced with off-the-shelf devices.

We often post about data exfiltration through non-obvious channels, such as a desktop PC speaker. But in all these cases, malware must already be installed on the victim’s computer. This paper hints at the possibility of exfiltrating sensitive information without prior hacking, simply by observing a power LED.

And that makes it an important contribution to our general knowledge of side-channel attacks. We can only hope that manufacturers of vulnerable devices take note and improve the design of new models — before such security holes become exploitable in practice. After all, as the paper rightly points out, it’s quite simple to make an LED that doesn’t reveal secret information: for example, add a dirt-cheap capacitor to the power supply circuit, and the problem’s solved. And measures to counteract side-channel attacks can be implemented at the software level too. Finally, why not ditch the LED altogether? Would anyone really miss it?

In English, please

Hardware processor vulnerabilities linked to speculative execution of instructions have been the subject of much research for more than five years. To simplify as much as possible, all the proposed attacks can be summed up as follows: the CPU is somehow forced to read data that user should not have access to. Imagine this theoretical scenario: the attackers’ program doesn’t have access to the encryption key used to protect sensitive data. If we instruct the CPU to “read the encryption key at a certain address”, the instruction simply won’t be followed. Help arrives (to the attacker) in the form of speculative execution of instructions – an important feature of modern CPUs, which has been around for almost three decades: to speed things up, instead of waiting for one instruction to finish, the processor executes the next in parallel.

If the first instruction checks access rights to sensitive information, it should in theory not allow execution of the following instruction to read this information. But it’s too late: the following instruction is being executed speculatively. Note that we don’t yet have access to this data – but the CPU does. In the case of known vulnerabilities such as Spectre, data is temporarily loaded into the CPU’s cache, but it can’t be read just like that. However, it can be read through side channels; for example, by repeatedly executing an instruction – the processing time of which varies depending on the data in the cache. Repeating such an operation many (thousands of!) times allows attackers to recover data just by watching how quickly or slowly some seemingly harmless command is executed.

We realize that this “simple” description still sounds complicated. The new paper is even more perplexing, especially since the authors decided not to spend time on a detailed description of the attack. The diagram below outlines it in full:

Overview of transient execution timing side-channel. Source.

Let’s try to figure it out. EFLAGS is a flag register in the Intel processor that keeps track of the CPU’s operating status. It can store the result of calculations, in particular if this is equal to zero (the so-called zero flag or ZF). Next comes the magic: imagine a colleague of yours thinks of a number from 1 to 10 and is told to keep it to themselves. You keep calling out the numbers 1 through 10 (looking for whatever signs could give your colleague away), but they don’t want to share the correct answer with you, and respond each time with the word “chrysanthemum”. However, when you utter the correct number, they take a little longer to say “chrysanthemum” compared with other times.

Something similar happens in this new attack: we perform numerous calculations with sensitive data. All these calculations are done speculatively. The result is written to the ZF flag (equal or not equal to zero). We can’t directly know the status of this flag. But then we execute a rather useless JCC instruction (specifically the JZ instruction – “jump if zero”), which runs a little slower if we guessed right! And it’s this measurable delay in the response that constitutes the vulnerability.

Not yet a problem

The most interesting aspect of this attack is that it doesn’t work by itself. To ensure that speculative execution of the required instructions is possible, the bad guys need to exploit one more vulnerability. The paper we’re discussing uses the Meltdown vulnerability, discovered in 2018, which happily provides access to information that’s off-limits to outsiders. As a result, sensitive data was read with 100% reliability on all old CPUs affected by this vulnerability (the study used sixth and seventh-generation Intel Core i7). Although the experiment failed on tenth-generation CPUs, they too experience some delay when executing a certain instruction from the JCC set.

In reality, even more versatile types of attacks like Spectre, which steal information from the CPU’s cache, have a rather narrow application. But at least in their case it was obvious that something had to be done with them: the likelihood of an advanced attack targeting critical data was non-zero. As for the new paper, we’re dealing more with an idea that, if it works at all, it applies to older Intel processors.

But the news itself is significant: there’s now a new side-channel mechanism for extracting data using the flag status register. It can’t be ruled out that in the future this approach, combined with some other vulnerability, will impact new CPUs as well. Or maybe it will all be resolved before we see a new attack: after all, dependence of the instruction execution time on the data is quite a serious issue. There’s an entire subdiscipline of cryptography that deals with protection of encryption algorithms from timing attacks.

In any case, research into the specifics of modern CPUs is ongoing. Fortunately, executing attacks on hardware vulnerabilities is just as difficult as understanding them. And although we’ve yet to see anything that could be applied on a massive scale, infosec officers in companies that handle highly sensitive data would be wise to factor in such threats and, at a minimum, monitor their evolution.

Researchers at universities in the U.S. and Switzerland, in collaboration with Google and DeepMind, have published a paper showing how data can leak from image-generation systems that use the machine-learning algorithms DALL-E, Imagen or Stable Diffusion. All of them work the same way on the user side: you type in a specific text query — for example, “an armchair in the shape of an avocado” — and get a generated image in return.

Image generated by the Dall-E neural network. Source.

All these systems are trained on a vast number (tens or hundreds of thousands) of images with pre-prepared descriptions. The idea behind such neural networks is that, by consuming a huge amount of training data, they can create new, unique images. However, the main takeaway of the new study is that these images are not always so unique. In some cases it’s possible to force the neural network to reproduce almost exactly an original image previously used for training. And that means that neural networks can unwittingly reveal private information.

Image generated by the Stable Diffusion neural network (right) and the original image from the training set (left). Source.

More data for the “data god”

The output of a machine-learning system in response to a query can seem like magic to a non-specialist: “woah – it’s like an all-knowing robot!”! But there’s no magic really…

All neural networks work more or less in the same way: an algorithm is created that’s trained on a data set — for example a series of pictures of cats and dogs — with a description of what exactly is depicted in each image. After the training stage, the algorithm is shown a new image and asked to work out whether it’s a cat or a dog. From these humble beginnings, the developers of such systems moved on to a more complex scenario: the algorithm trained on lots of pictures of cats creates an image of a pet that never existed on demand. Such experiments are carried out not only with images, but also with text, video and even voice: we’ve already written about the problem of deepfakes (whereby digitally altered videos of (mostly) politicians or celebrities seem to say stuff they never actually did).

For all neural networks, the starting point is a set of training data: neural networks cannot invent new entities from nothing. To create an image of a cat, the algorithm must study thousands of real photographs or drawings of these animals. There are plenty of arguments for keeping these data sets confidential. Some of them are in the public domain; other data sets are the intellectual property of the developer company that invested considerable time and effort into creating them in the hope of achieving a competitive advantage. Still others, by definition, constitute sensitive information. For example, experiments are underway to use neural networks to diagnose diseases based on X-rays and other medical scans. This means that the algorithmic training data contains the actual health data of real people, which, for obvious reasons, must not fall into the wrong hands.

Diffuse it

Although machine-learning algorithms look the same to the outsider, they are in fact different. In their paper, the researchers pay special attention to machine-learning diffusion models. They work like this: the training data (again images of people, cars, houses, etc.) is distorted by adding noise. And the neural network is then trained to restore such images to their original state. This method makes it possible to generate images of decent quality, but a potential drawback (in comparison with algorithms in generative adversarial networks, for example) is their greater tendency to leak data.

The original data can be extracted from them in at least three different ways: First, using specific queries, you can force the neural network to output — not something unique, generated based on thousands of pictures — but a specific source image. Second, the original image can be reconstructed even if only a part of it is available. Third, it’s possible to simply establish whether or not a particular image is contained within the training data.

Very often, neural networks are… lazy, and instead of a new image, they produce something from the training set if it contains multiple duplicates of the same picture. Besides the above example with the Ann Graham Lotz photo, the study gives quite a few other similar results:

Odd rows: the original images. Even rows: images generated by Stable Diffusion v1.4. Source.

If an image is duplicated in the training set more than a hundred times, there’s a very high chance of its leaking in its near-original form. However, the researchers demonstrated ways to retrieve training images that only appeared once in the original set. This method is far less efficient: out of five hundred tested images, the algorithm randomly recreated only three of them. The most artistic method of attacking a neural network involves recreating a source image using just a fragment of it as input.

The researchers asked the neural network to complete the picture, after having deleted part of it. Doing this can be used to determine fairly accurately whether a particular image was in the training set. If it was, the machine-learning algorithm generated an almost exact copy of the original photo or drawing. Source.

At this point, let’s divert our attention to the issue of neural networks and copyright.

Who stole from whom?

In January 2023, three artists sued the creators of image-generating services that used machine-learning algorithms. They claimed (justifiably) that the developers of the neural networks had trained them on images collected online without any respect for copyright. A neural network can indeed copy the style of a particular artist, and thus deprive them of income. The paper hints that in some cases algorithms can, for various reasons, engage in outright plagiarism, generating drawings, photographs and other images that are almost identical to the work of real people.

The study makes recommendations for strengthening the privacy of the original training set:

• Get rid of duplicates.
• Reprocess training images, for example by adding noise or changing the brightness; this makes data leakage less likely.
• Test the algorithm with special training images, then check that it doesn’t inadvertently reproduce them accurately.

What next?

The ethics and legality of generative art certainly make for an interesting debate — one in which a balance must be sought between artists and the developers of the technology. On the one hand, copyright must be respected. On the other, is computer art so different from human? In both cases, the creators draw inspiration from the works of colleagues and competitors.

But let’s get back down to earth and talk about security. The paper provides a specific set of facts about only one machine-learning model. Extending the concept to all similar algorithms, we arrive at an interesting situation. It’s not hard to imagine a scenario whereby a smart assistant of a mobile operator hands out sensitive corporate information in response to a user query: after all, it was in the training data. Or, for example, a cunning query tricks a public neural network into generating a copy of someone’s passport. The researchers stress that such problems remain theoretical for the time being.

But other problems are already with us. As we speak, the text-generating neural network ChatGPT is being used to write real malicious code that (sometimes) works. And GitHub Copilot is helping programmers write code using a huge amount of open-source software as input. And the tool doesn’t always respect the copyright and privacy of the authors whose code ended up in the sprawling set of training data. As neural networks evolve, so too will the attacks on them — with consequences that no one yet fully understands.