Cloud threats

The most popular AI assistants run on the cloud infrastructure of large companies. It’s efficient and fast, but your data processed by the model may be accessible to both the AI service provider and completely unrelated parties, as happened last year with ChatGPT.

Such incidents present varying levels of threat depending on what these AI assistants are used for. If you’re generating cute illustrations for some fairy tales you’ve written, or asking ChatGPT to create an itinerary for your upcoming weekend city break, it’s unlikely that a leak will lead to serious damage. However, if your conversation with a chatbot contains confidential info — personal data, passwords, or bank card numbers — a possible leak to the cloud is no longer acceptable. Thankfully, it’s relatively easy to prevent by pre-filtering the data — we’ve written a separate post about that.

However, in cases where either all the correspondence is confidential (for example, medical or financial information), or the reliability of pre-filtering is questionable (you need to process large volumes of data that no one will preview and filter), there’s only one solution: move the processing from the cloud to a local computer. Of course, running your own version of ChatGPT or Midjourney offline is unlikely to be successful, but other neural networks working locally provide comparable quality with less computational load.

What hardware do you need to run a neural network?

You’ve probably heard that working with neural networks requires super-powerful graphics cards, but in practice this isn’t always the case. Different AI models, depending on their specifics, may be demanding on such computer components as RAM, video memory, drive, and CPU (here, not only the processing speed is important, but also the processor’s support for certain vector instructions). The ability to load the model depends on the amount of RAM, and the size of the “context window” — that is, the memory of the previous conversation — depends on the amount of video memory. Typically, with a weak graphics card and CPU, generation occurs at a snail’s pace (one to two words per second for text models), so a computer with such a minimal setup is only appropriate for getting acquainted with a particular model and evaluating its basic suitability. For full-fledged everyday use, you’ll need to increase the RAM, upgrade the graphics card, or choose a faster AI model.

As a starting point, you can try working with computers that were considered relatively powerful back in 2017: processors no lower than Core i7 with support for AVX2 instructions, 16GB of RAM, and graphics cards with at least 4GB of memory. For Mac enthusiasts, models running on the Apple M1 chip and above will do, while the memory requirements are the same.

When choosing an AI model, you should first familiarize yourself with its system requirements. A search query like “model_name requirements” will help you assess whether it’s worth downloading this model given your available hardware. There are detailed studies available on the impact of memory size, CPU, and GPU on the performance of different models; for example, this one.

Good news for those who don’t have access to powerful hardware — there are simplified AI models that can perform practical tasks even on old hardware. Even if your graphics card is very basic and weak, it’s possible to run models and launch environments using only the CPU. Depending on your tasks, these can even work acceptably well.

Examples of how various computer builds work with popular language models

Choosing an AI model and the magic of quantization

A wide range of language models are available today, but many of them have limited practical applications. Nevertheless, there are easy-to-use and publicly available AI tools that are well-suited for specific tasks, be they generating text (for example, Mistral 7B), or creating code snippets (for example, Code Llama 13B). Therefore, when selecting a model, narrow down the choice to a few suitable candidates, and then make sure that your computer has the necessary resources to run them.

In any neural network, most of the memory strain is courtesy of weights — numerical coefficients describing the operation of each neuron in the network. Initially, when training the model, the weights are computed and stored as high-precision fractional numbers. However, it turns out that rounding the weights in the trained model allows the AI tool to be run on regular computers while only slightly decreasing the performance. This rounding process is called quantization, and with its help the model’s size can be reduced considerably — instead of 16 bits, each weight might use eight, four, or even two bits.

According to current research, a larger model with more parameters and quantization can sometimes give better results than a model with precise weight storage but fewer parameters.

Armed with this knowledge, you’re now ready to explore the treasure trove of open-source language models, namely the top Open LLM leaderboard. In this list, AI tools are sorted by several generation quality metrics, and filters make it easy to exclude models that are too large, too small, or too accurate.

List of language models sorted by filter set

After reading the model description and making sure it’s potentially a fit for your needs, test its performance in the cloud using Hugging Face or Google Colab services. This way, you can avoid downloading models which produce unsatisfactory results, saving you time. Once you’re satisfied with the initial test of the model, it’s time to see how it works locally!

Required software

Most of the open-source models are published on Hugging Face, but simply downloading them to your computer isn’t enough. To run them, you have to install specialized software, such as LLaMA.cpp, or — even easier — its “wrapper”, LM Studio. The latter allows you to select your desired model directly from the application, download it, and run it in a dialog box.

Another “out-of-the-box” way to use a chatbot locally is GPT4All. Here, the choice is limited to about a dozen language models, but most of them will run even on a computer with just 8GB of memory and a basic graphics card.

If generation is too slow, then you may need a model with coarser quantization (two bits instead of four). If generation is interrupted or execution errors occur, the problem is often insufficient memory — it’s worth looking for a model with fewer parameters or, again, with coarser quantization.

Many models on Hugging Face have already been quantized to varying degrees of precision, but if no one has quantized the model you want with the desired precision, you can do it yourself using GPTQ.

This week, another promising tool was released to public beta: Chat With RTX from NVIDIA. The manufacturer of the most sought-after AI chips has released a local chatbot capable of summarizing the content of YouTube videos, processing sets of documents, and much more — provided the user has a Windows PC with 16GB of memory and an NVIDIA RTX 30^th or 40^th series graphics card with 8GB or more of video memory. “Under the hood” are the same varieties of Mistral and Llama 2 from Hugging Face. Of course, powerful graphics cards can improve generation performance, but according to the feedback from the first testers, the existing beta is quite cumbersome (about 40GB) and difficult to install. However, NVIDIA’s Chat With RTX could become a very useful local AI assistant in the future.

The code for the game “Snake”, written by the quantized language model TheBloke/CodeLlama-7B-Instruct-GGUF

The applications listed above perform all computations locally, don’t send data to servers, and can run offline so you can safely share confidential information with them. However, to fully protect yourself against leaks, you need to ensure not only the security of the language model but also that of your computer – and that’s where our comprehensive security solution comes in. As confirmed in independent tests, Kaspersky Premium has practically no impact on your computer’s performance — an important advantage when working with local AI models.

So, how do you use AI without regretting it later?

Filter important data

The privacy policy of OpenAI, the developer of ChatGPT, unequivocally states that any dialogs with the chatbot are saved and can be used for a number of purposes. First, these are solving technical issues and preventing terms-of-service violations: in case someone gets an idea to generate inappropriate content. Who would have thought it, right? In that case, chats may even be reviewed by a human. Second, the data may be used for training new GPT versions and making other product “improvements”.

Most other popular language models — be it Google’s Gemini, Anthropic’s Claude, or Microsoft’s Bing and Copilot — have similar policies: they can all save dialogs in their entirety.

That said, inadvertent chat leaks have already occurred due to software bugs, with users seeing other people’s conversations instead of their own. The use of this data for training could also lead to a data leak from a pre-trained model: the AI assistant might give your information to someone if it believes it to be relevant for the response. Information security experts have even designed multiple attacks (one, two, three) aimed at stealing dialogs, and they’re unlikely to stop there.

So, remember: anything you write to a chatbot can be used against you. We recommend taking precautions when talking to AI.

Don’t send any personal data to a chatbot. No passwords, passport or bank card numbers, addresses, telephone numbers, names, or other personal data that belongs to you, your company, or your customers must end up in chats with an AI. You can replace these with asterisks or “REDACTED” in your request.

Don’t upload any documents. Numerous plug-ins and add-ons let you use chatbots for document processing. There might be a strong temptation to upload a work document to, say, get an executive summary. However, by carelessly uploading of a multi-page document, you risk leaking confidential data, intellectual property, or a commercial secret such as the release date of a new product or the entire team’s payroll. Or, worse than that, when processing documents received from external sources, you might be targeted with an attack that counts on the document being scanned by a language model.

Use privacy settings. Carefully review your large-language-model (LLM) vendor’s privacy policy and available settings: these can normally be leveraged to minimize tracking. For example, OpenAI products let you disable saving of chat history. In that case, data will be removed after 30 days and never used for training. Those who use API, third-party apps, or services to access OpenAI solutions have that setting enabled by default.

Sending code? Clean up any confidential data. This tip goes out to those software engineers who use AI assistants for reviewing and improving their code: remove any API keys, server addresses, or any other information that could give away the structure of the application or the server configuration.

Limit the use of third-party applications and plug-ins

Follow the above tips every time — no matter what popular AI assistant you’re using. However, even this may not be sufficient to ensure privacy. The use of ChatGPT plug-ins, Gemini extensions, or separate add-on applications gives rise to new types of threats.

First, your chat history may now be stored not only on Google or OpenAI servers but also on servers belonging to the third party that supports the plug-in or add-on, as well as in unlikely corners of your computer or smartphone.

Second, most plug-ins draw information from external sources: web searches, your Gmail inbox, or personal notes from services such as Notion, Jupyter, or Evernote. As a result, any of your data from those services may also end up on the servers where the plug-in or the language model itself is running. An integration like that may carry significant risks: for example, consider this attack that creates new GitHub repositories on behalf of the user.

Third, the publication and verification of plug-ins for AI assistants are currently a much less orderly process than, say, app-screening in the App Store or Google Play. Therefore, your chances of encountering a poorly working, badly written, buggy, or even plain malicious plug-in are fairly high — all the more so because it seems no one really checks the creators or their contacts.

How do you mitigate these risks? Our key tip here is to give it some time. The plug-in ecosystem is too young, the publication and support processes aren’t smooth enough, and the creators themselves don’t always take care to design plug-ins properly or comply with information security requirements. This whole ecosystem needs more time to mature and become securer and more reliable.

Besides, the value that many plug-ins and add-ons add to the stock ChatGPT version is minimal: minor UI tweaks and “system prompt” templates that customize the assistant for a specific task (“Act as a high-school physics teacher…”). These wrappers certainly aren’t worth trusting with your data, as you can accomplish the task just fine without them.

If you do need certain plug-in features right here and now, try to take maximum precautions available before using them.

• Choose extensions and add-ons that have been around for at least several months and are being updated regularly.
• Consider only plug-ins that have lots of downloads, and carefully read the reviews for any issues.
• If the plug-in comes with a privacy policy, read it carefully before you start using the extension.
• Opt for open-source tools.
• If you possess even rudimentary coding skills — or coder friends — skim the code to make sure that it only sends data to declared servers and, ideally, AI model servers only.

Execution plug-ins call for special monitoring

So far, we’ve been discussing risks relating to data leaks; but this isn’t the only potential issue when using AI. Many plug-ins are capable of performing specific actions at the user’s command — such as ordering airline tickets. These tools provide malicious actors with a new attack vector: the victim is presented with a document, web page, video, or even an image that contains concealed instructions for the language model in addition to the main content. If the victim feeds the document or link to a chatbot, the latter will execute the malicious instructions — for example, by buying tickets with the victim’s money. This type of attack is referred to as prompt injection, and although the developers of various LLMs are trying to develop a safeguard against this threat, no one has managed it — and perhaps never will.

Luckily, most significant actions — especially those involving payment transactions such as purchasing tickets — require a double confirmation. However, interactions between language models and plug-ins create an attack surface so large that it’s difficult to guarantee consistent results from these measures.

Therefore, you need to be really thorough when selecting AI tools, and also make sure that they only receive trusted data for processing.

Protect your finances

E-commerce and financial technologies continue to expand globally, and successful technologies are being adopted in new regions. Instant electronic payments between individuals have become much more widespread. And, of course, criminals are devising new ways to swindle you out of your money. This involves not only fraud using instant money-transfer systems, but also advanced techniques for stealing payment data on e-commerce sites and online stores. The latest generations of web skimmers installed by hackers on legitimate online shopping sites are almost impossible to perceive, and victims only learn that their data has been stolen when an unauthorized charge appears on their card.

What to do?

• Link your bank cards to Apple Pay, Google Pay, or other similar payment systems available in your country. This is not only convenient, but also reduces the likelihood of data theft when making purchases in stores.
• Use such systems to make payments on websites whenever possible. There’s no need to enter your bank card details afresh on every new website.
• Protect your smartphones and computers with a comprehensive security system like Kaspersky Premium. This will help protect your money, for example, from a nasty new attack in which the recipient’s details are replaced at the moment of making an instant money transfer in a banking app.
• Use virtual or one-time cards for online payments if your bank supports this option. If a virtual card can be quickly reissued in the app, change it regularly — for example, once a month. Or use special services to ‘mask’ cards, generating one-time payment details for each payment session. There are many of these for different countries and payment systems.

Don’t believe everything you see

Generative artificial intelligence has dominated the news throughout 2023 and has already significantly affected the job market. Unfortunately, it’s also been used for malicious purposes. Now, just about anyone can create fake texts, photos, and videos in a matter of minutes — a labor that previously required a lot of time and skill. This has already had a noticeable impact on at least two areas of cybersecurity.

First, the appearance of fake images, audio, and video on news channels and social media. In 2023, generated images were used for propaganda purposes during geopolitical conflicts in post-Soviet countries and the Middle East. They were also used successfully by fraudsters for various instances of fake fundraising. Moreover, towards the end of the year, our experts discovered massive “investment” campaigns in which the use of deepfakes reached a whole new level: now we’re seeing news reports and articles on popular channels about famous businessmen and heads of state encouraging users to invest in certain projects — all fake, of course.

Second, AI has made it much easier to generate phishing emails, social media posts, and fraudulent websites. For many years, such scams could be identified by sloppy language and numerous typos, because the scammers didn’t have the time to write and proofread them properly. But now, with WormGPT and other language models optimized for hackers, attackers can create far more convincing and varied bait on an industrial scale. What’s more, experts fear that scammers will start using these same multilingual AI models to create convincing phishing material in languages and regions that have rarely been targeted for such purposes before.

What to do?

• Be highly critical of any emotionally provocative content you encounter on social media — especially from people you don’t know personally. Make it a habit to always verify the facts on reputable news channels and expert websites.
• Don’t transfer money to any kind of charity fundraiser or campaign without conducting a thorough background check of the recipient first. Remember, generating heart-breaking stories and images is literally as easy as pushing a button these days.
• Install phishing and scam protection on all your devices, and enable all options that check links, websites, emails, and attachments. This will reduce the risk of clicking on phishing links or visiting fraudulent websites.
• Activate banner ad protection — both Kaspersky Plus and Kaspersky Premium have this feature, as do a number of browsers. Malicious advertising is another trend for 2023-2024.

Some experts anticipate the emergence of AI-generated content analysis and labeling systems in 2024. However, don’t expect them to be implemented quickly or universally, or be completely reliable. Even if such solutions do emerge, always double-check any information with trusted sources.

Don’t believe everything you hear

High-quality AI-based voice deepfakes are already being actively used in fraudulent schemes. Someone claiming to be your “boss”, “family member”, “colleague”, or some other person with a familiar voice might call asking for urgent help — or to help someone else who’ll soon reach out to you. Such schemes mainly aim to trick victims into voluntarily sending money to criminals. More complex scenarios are also possible — for example, targeting company employees to obtain passwords for accessing the corporate network.

What to do?

• Verify any unexpected or alarming calls without panic. If someone you supposedly know well calls, ask a question only that person can answer. If a colleague calls but their request seems odd — for example, asking you to send or spell a password, send a payment, or do something else unusual — reach out to other colleagues or superiors to double-check things.
• Use caller identifier apps to block spam and scam calls. Some of these apps work not only with regular phone calls but also with calls through messengers like WhatsApp.

Buy only safe internet-of-things (IoT) smart devices

Poorly protected IoT devices create a whole range of problems for their owners: robot vacuum cleaners spy on their owners, smart pet feeders can give your pet an unplanned feast or a severe hunger strike, set-top boxes steal accounts and create rogue proxies on your home network, and baby monitors and home security cameras turn your home into a reality TV show without your knowledge.

What could improve in 2024? The emergence of regulatory requirements for IoT device manufacturers. For example, the UK will ban the sale of devices with default logins and passwords like “admin/admin”, and require manufacturers to disclose in advance how long a particular device will receive firmware updates. In the U.S., a security labeling system is being developed that will make it possible to understand what to expect from a “smart” device in terms of security even before purchase.

What to do?

• Find out if there are similar initiatives in your country and make the most of them by purchasing only secure IoT devices with a long period of declared support. It’s likely that once manufacturers are obliged to ensure the security of smart devices locally, they’ll make corresponding changes to products for the global market. Then you’ll be able to choose a suitable product by checking, for example, the American “security label”, and buy it — even if you’re not in the U.S.
• Carefully configure all smart devices using our detailed advice on creating a smart home and setting up its security.

Take care of your loved ones

Scams involving fake texts, images, and voices messages can be highly effective when used on elderly people, children, or those less interested in technology. Think about your family, friends, and colleagues — if any of them may end up a victim of any the schemes described above, take the time to tell them about them or provide a link to our blog.

What to do?

• Don’t just give blanket information from our articles; look beyond our blog to find suitable cybersecurity lessons for your loved ones based on their age and temperament.
• Make sure that all your family’s computers and phones are fully protected. With Kaspersky Premium, you can protect as many devices as needed, on any popular platform — Windows, macOS, Android, or iOS.

Before we say goodbye and wish you a happy and peaceful 2024, one final little whisper — last year’s New Year’s resolutions are still very relevant: the transition to password-less systems is progressing at a swift pace, so going password-free in the New Year might be a good idea, while basic cyber hygiene has become all the more crucial. Oops; nearly forgot: wishing you a happy and peaceful 2024!…

Various international discussion platforms and organizations have already developed general principles of ethical AI (here are the UNESCO recommendations, for example), but more specific guidelines for the cybersecurity industry have yet to be commonly accepted.

In order to apply AI in cybersecurity without negative consequences, we propose that the industry adopt a set of AI ethical principles, the first version of which we are presenting at the UN Internet Governance Forum in Kyoto, Japan. The principles need to be discussed and clarified across the wider cybersecurity community, of course, but we are already adhering to them. What are these principles? Here they are in brief.

Transparency

Users have the right to know if a security provider uses AI systems, as well as how these systems make decisions and for what purposes. This is why we are committed to developing AI systems that are interpretable to the maximum extent possible, with all necessary safeguards in place to ensure they produce valid outcomes. Anyone can get acquainted with our code and workflows by visiting one of our Kaspersky Transparency Centers.

Safety

Among the threats facing AI systems is the manipulation of input datasets to produce inappropriate decisions. Therefore, we believe that AI developers must prioritize resilience and security.

To this end, we adopt a whole range of practical measures to deliver high-quality AI systems: AI-specific security audits and red teaming; minimal use of third-party datasets in training; plus implementing an array of technologies for multilayered protection. Where possible, we favor cloud-based AI (with all the necessary safeguards in place) over locally installed models.

Human control

Although our ML systems can operate autonomously, their results and performance are constantly monitored by experts. Verdicts of our automated systems are fine-tuned as required, and the systems themselves are adapted and modified by experts to resist fundamentally new and/or highly sophisticated cyberthreats. We combine ML with human expertise, and are committed to forever maintaining this human element of control in our systems.

Privacy

AI cannot be trained without big data — some of which may be personal. Therefore, an ethical approach to its use must respect the rights of individuals to privacy. In information security practice, this can involve various measures: limiting the types and quantity of data processed; pseudonymization and anonymization; reducing data composition; ensuring data integrity; and applying technical and organizational measures to protect data.

Developed for cybersecurity

AI in cybersecurity must be used solely for defensive purposes. This forms an integral part of our mission to build a secure world in which tomorrow’s technologies enhance all our lives.

Open for dialogue

We believe that only through working together can we overcome the obstacles associated with the adoption and use of AI for security. For this reason, we promote dialogue with all stakeholders to share best practices in the ethical use of AI.

Read more about our principles of ethical use of AI in security.

Help, I’m in trouble

In spring 2023, scammers in Arizona attempted to extort money from a woman over the phone. She heard the voice of her 15-year-old daughter begging for help before an unknown man grabbed the phone and demanded a ransom, all while her daughter’s screams could still be heard in the background. The mother was positive that the voice was really her child’s. Fortunately, she found out fast that everything was fine with her daughter, leading her to realize that she was a victim of scammers.

It can’t be 100% proven that the attackers used a deepfake to imitate the teenager’s voice. Maybe the scam was of a more traditional nature, with the call quality, unexpectedness of the situation, stress, and the mother’s imagination all playing their part to make her think she heard something she didn’t. But even if neural network technologies weren’t used in this case, deepfakes can and do indeed occur, and as their development continues they become increasingly convincing and more dangerous. To fight the exploitation of deepfake technology by criminals, we need to understand how it works.

What are deepfakes?

Deepfake (“deep learning” + “fake”) artificial intelligence has been growing at a rapid rate over the past few years. Machine learning can be used to create compelling fakes of images, video, or audio content. For example, neural networks can be used in photos and videos to replace one person’s face with another while preserving facial expressions and lighting. While initially these fakes were low quality and easy to spot, as the algorithms developed the results became so convincing that now it’s difficult to distinguish them from reality. In 2022, the world’s first deepfake TV show was released in Russia, where deepfakes of Jason Statham, Margot Robbie, Keanu Reeves and Robert Pattinson play the main characters.

Deepfake versions of Hollywood stars in the Russian TV series PMJason. (Source)

Voice conversion

But today our focus is on the technology used for creating voice deepfakes. This is also known as voice conversion (or “voice cloning” if you’re creating a full digital copy of it). Voice conversion is based on autoencoders – a type of neural network that first compresses input data (part of the encoder) into a compact internal representation, and then learns to decompress it back from this representation (part of the decoder) to restore the original data. This way the model learns to present data in a compressed format while highlighting the most important information.

Autoencoder scheme. (Source)

To make voice deepfakes, two audio recordings are fed into the model, with the voice from the second recording converted to the first. The content encoder is used to determine what was said from the first recording, and the speaker encoder is used to extract the main characteristics of the voice from the second recording – meaning how the second person talks. The compressed representations of what must be said and how it’s said are combined, and the result is generated using the decoder. Thus, what’s said in the first recording is voiced by the person from the second recording.

The process of making a voice deepfake. (Source)

There are other approaches that use autoencoders, for example those that use generative adversarial networks (GAN) or diffusion models. Research into how to make deepfakes is supported in particular by the film industry. Think about it: with audio and video deepfakes, it’s possible to replace the faces of actors in movies and TV shows, and dub movies with synchronized facial expressions into any language.

How it’s done

As we were researching deepfake technologies, we wondered how hard it might be to make one’s own voice deepfake? It turns out there are lots of free open-source tools for working with voice conversion, but it isn’t so easy to get a high-quality result with them. It takes Python programming experience and good processing skills, and even then the quality is far from ideal. In addition to open source, there are also proprietary and paid solutions available.

For example, in early 2023, Microsoft announced an algorithm that could reproduce a human voice based on an audio example that’s only three seconds long! This model also works with multiple languages, so you can even hear yourself speaking a foreign language. All this looks promising, but so far it’s only at the research stage. But the ElevenLabs platform lets users make voice deepfakes without any effort: just upload an audio recording of the voice and the words to be spoken, and that’s it. Of course, as soon as word got out, people started playing with this technology in all sorts of ways.

Hermione’s battle and an overly trusting bank

In full accordance with Godwin’s law, Emma Watson was made to read “Mein Kampf”, and another user used ElevenLabs technology to “hack” his own bank account. Sounds creepy? It does to us – especially when you add to the mix the popular horror stories about scammers collecting samples of voices over the phone by having folks say “yes” or “confirm” as they pretend to be a bank, government agency or poll service, and then steal money using voice authorization.

But in reality things aren’t so bad. Firstly, it takes about five minutes of audio recordings to create an artificial voice in ElevenLabs, so a simple “yes” isn’t enough. Secondly, banks also know about these scams, so voice can only be used to initiate certain operations that aren’t related to the transfer of funds (for example, to check your account balance). So money can’t be stolen this way.

To its credit, ElevenLabs reacted to the problem fast by rewriting the service rules, prohibiting free (i.e., anonymous) users to create deepfakes based on their own uploaded voices, and blocking accounts with complaints about “offensive content”.

While these measures may be useful, they still don’t solve the problem of using voice deepfakes for suspicious purposes.

How else deepfakes are used in scams

Deepfake technology in itself is harmless, but in the hands of scammers it can become a dangerous tool with lots of opportunities for deception, defamation or disinformation. Fortunately, there haven’t been any mass cases of scams involving voice alteration, but there have been several high-profile cases involving voice deepfakes.

In 2019, scammers used this technology to shake down UK-based energy firm. In a telephone conversation, the scammer pretended to be the chief executive of the firm’s German parent company, and requested the urgent transfer of €220,000 ($243,000) to the account of a certain supplier company. After the payment was made, the scammer called twice more – the first time to put the UK office staff at ease and report that the parent company had already sent a refund, and the second time to request another transfer. All three times the UK CEO was absolutely positive that he was talking with his boss because he recognized both his German accent and his tone and manner of speech. The second transfer wasn’t sent only because the scammer messed up and called from an Austrian number instead of a German one, which made the UK SEO suspicious.

A year later, in 2020, scammers used deepfakes to steal up to $35,000,000 from an unnamed Japanese company (the name of the company and total amount of stolen goods weren’t disclosed by the investigation).

It’s unknown which solutions (open source, paid, or even their own) the scammers used to fake voices, but in both the above cases the companies clearly suffered – badly – from deepfake fraud.

What’s next?

Opinions differ about the future of deepfakes. Currently, most of this technology is in the hands of large corporations, and its availability to the public is limited. But as the history of much more popular generative models like DALL-E, Midjourney and Stable Diffusion shows, and even more so with large language models (ChatGPT anybody?), similar technologies may well appear in the public domain in the foreseeable future. This is confirmed by a recent leak of internal Google correspondence in which representatives of the internet giant fear they’ll lose the AI race to open solutions. This will obviously result in an increase in the use of voice deepfakes – including for fraud.

The most promising step in the development of deepfakes is real-time generation, which will ensure the explosive growth of deepfakes (and fraud based on them). Can you imagine a video call with someone whose face and voice are completely fake? However, this level of data processing requires huge resources only available to large corporations, so the best technologies will remain private and fraudsters won’t be able to keep up with the pros. The high quality bar will also help users learn how to easily identify fakes.

How to protect yourself

Now back to our very first question: can we trust the voices we hear (that is – if they’re not the voices in our head)? Well, it’s probably overdoing it if we’re paranoid all the time and start coming up with secret code words to use with friends and family; however, in more serious situations such paranoia might be appropriate. If everything develops based on the pessimistic scenario, deepfake technology in the hands of scammers could grow into a formidable weapon in the future, but there’s still time to get ready and build reliable methods of protection against counterfeiting: there’s already a lot of research into deepfakes, and large companies are developing security solutions. In fact, we’ve already talked in detail about ways to combat video deepfakes here.

For now, protection against AI fakes is only just beginning, so it’s important to keep in mind that deepfakes are just another kind of advanced social engineering. The risk of encountering fraud like this is small, but it’s still there, so it’s worth knowing and keeping in mind. If you get a strange call, pay attention to the sound quality. Is it in an unnatural monotone, is it unintelligible, or are there strange noises? Always double-check information through other channels, and remember that surprise and panic are what scammers rely on most.

Plainly a debatable matter is, of course, the following: do we need government regulation of AI at all? If we do — why, and what should it look like?

What to regulate

What is artificial intelligence? (No) thanks to marketing departments, the term’s been used for lots of things — from the cutting-edge generative models like GPT-4, to the simplest machine-learning systems, including some that have been around for decades. Remember Т9 on push-button cellphones? Heard about automatic spam and malicious file classification? Do you check out film recommendations on Netflix? All of those familiar technologies are based on machine learning (ML) algorithms, aka “AI”.

Here at Kaspersky, we’ve been using such technologies in our products for close on 20 years, always preferring to modestly refer to them as “machine learning” — if only because “artificial intelligence” seems to call to most everyone’s mind things like talking supercomputers on spaceships and other stuff straight out of science fiction. However, such talking-thinking computers and droids need to be fully capable of human-like thinking — to command artificial general intelligence (AGI) or artificial superintelligence (ASI), yet neither AGI nor ASI have been invented yet, and will hardly be so in the foreseeable future.

Anyway, if all the AI types are measured with the same yardstick and fully regulated, the whole IT industry and many related ones aren’t going to fare well at all. For example, if we (Kaspersky) will ever be required to get the consent from all our training-set “authors”, we, as an information security company, will find ourselves up against the wall. We learn from malware and spam, and feed the knowledge gained into our machine learning, while their authors tend to prefer to withhold their contact data (who knew?!). Moreover, considering that data has been collected and our algorithms have been trained for nearly 20 years now —  quite how far into the past would we be expected to go?

Therefore, it’s essential for lawmakers to listen, not to marketing folks, but to machine-learning/AI industry experts and discuss potential regulation in a specific and focused manner: for example, possibly using multi-function systems trained on large volumes of open data, or high responsibility and risk level decision-making systems.

And new AI applications will necessitate frequent revisions of regulations as they arise.

Why regulate?

To be honest, I don’t believe in a superintelligence-assisted Judgement Day within the next hundred years. But I do believe in a whole bunch of headaches from thoughtless use of the computer black box.

As a reminder to those who haven’t read our articles on both the splendor and misery of machine learning, there are three main issues regarding any AI:

• It’s not clear just how good the training data used for it were/are.
• It’s not clear at all what AI has succeeded in “comprehending” out of that stock of data, or how it makes its decisions.
• And most importantly — the algorithm can be misused by its developers and its users alike.

Thus, anything at all could happen: from malicious misuse of AI, to unthinking compliance with AI decisions. Graphic real-life examples: fatal autopilot errors, deepfakes (1, 2, 3) by now habitual in memes and even the news, a silly error in school teacher contracting, the police apprehending a shoplifter but the wrong one, and a misogynous AI recruiting tool. Besides, any AI can be attacked with the help of custom-made hostile data samples: vehicles can be tricked using stickers, one can extract personal information from GPT-3, and anti-virus or EDR can be deceived too. And by the way, attacks on combat-drone AI described in science fiction don’t appear all that far-fetched any more.

In a nutshell, the use of AI hasn’t given rise to any truly massive problems yet, but there is clearly a lot of potential for them. Therefore, the priorities of regulation should be clear:

1. Preventing critical infrastructure incidents (factories/ships/power transmission lines/nuclear power plants).
2. Minimizing physical threats (driverless vehicles, misdiagnosing illnesses).
3. Minimizing personal damage and business risks (arrests or hirings based on skull measurements, miscalculation of demand/procurements, and so on).

The objective of regulation should be to compel users and AI vendors to take care not to increase the risks of the mentioned negative things happening. And the more serious the risk, the more actively it should be compelled.

There’s another concern often aired regarding AI: the need for observance of moral and ethical norms, and to cater to psychological comfort, so to say. To this end, we see warnings given so folks know that they’re viewing a non-existent (AI-drawn) object or communicating with a robot and not a human, and also notices informing that copyright was respected during AI training, and so on. And why? So lawmakers and AI vendors aren’t targeted by angry mobs! And this is a very real concern in some parts of the world (recall protests against Uber, for instance).

How to regulate

The simplest way to regulate AI would to prohibit everything, but it looks like this approach isn’t on the table yet. And anyway it’s not much easier to prohibit AI than it is computers. Therefore, all reasonable regulation attempts should follow the principle of “the greater the risk, the stricter the requirements”.

The machine-learning models that are used for something rather trivial — like retail buyer recommendations — can go unregulated, but the more sophisticated the model — or the more sensitive the application area — the more drastic can be the requirements for system vendors and users. For example:

• Submitting a model’s code or training dataset for inspection to regulators or experts.
• Proving the robustness of a training dataset, including in terms of bias, copyright and so forth.
• Proving the reasonableness of the AI “output”; for example, its being free of hallucinations.
• Labelling AI operations and results.
• Updating a model and training dataset; for example, screening out folks of a given skin color from the source data, or suppressing chemical formulas for explosives in the model’s output.
• Testing AI for “hostile data”, and updating its behavior as necessary.
• Controlling who’s using specific AI and why. Denying specific types of use.
• Training large AI, or that which applies to a particular area, only with the permission of the regulator.
• Proving that it’s safe to use AI to address a particular problem. This approach is very exotic for IT, but more than familiar to, for example, pharmaceutical companies, aircraft manufacturers and many other industries where safety is paramount. First would come five years of thorough tests, then the regulator’s permission, and only then a product could be released for general use.

The last measure appears excessively strict, but only until you learn about incidents in which AI messed up treatment priorities for acute asthma and pneumonia patients and tried to send them home instead of to an intensive care unit.

The enforcement measures may range from fines for violations of AI rules (along the lines of European penalties for GDPR violations) to licensing of AI-related activities and criminal sanctions for breaches of legislation (as proposed in China).

But what’s the right way?

Below represent my own personal opinions — but they’re based on 30 years of active pursuit of advanced technological development in the cybersecurity industry: from machine learning to “secure-by-design” systems.

First, we do need regulation. Without it, AI will end up resembling highways without traffic rules. Or, more relevantly, resembling the online personal data collection situation in the late 2000s, when nearly everyone would collect all they could lay their hands on. Above all, regulation promotes self-discipline in the market players.

Second, we need to maximize international harmonization and cooperation in regulation — the same way as with technical standards in mobile communications, the internet and so on. Sounds utopian given the modern geopolitical reality, but that doesn’t make it any less desirable.

Third, regulation needn’t be too strict: it would be short-sighted to strangle a dynamic young industry like this one with overregulation. That said, we need a mechanism for frequent revisions of the rules to stay abreast of technology and market developments.

Fourth, the rules, risk levels, and levels of protection measures should be defined in consultation with a great many relevantly-experienced experts.

Fifth, we don’t have to wait ten years. I’ve been banging on about the serious risks inherent in the Internet of Things and about vulnerabilities in industrial equipment for over a decade already, while documents like the EU Cyber Resilience Act first appeared (as drafts!) only last year.

But that’s all for now folks! And well done to those of your who’ve read this to the end — thank you all! And here’s to an interesting – safe – AI-enhanced future!…

Researchers at universities in the U.S. and Switzerland, in collaboration with Google and DeepMind, have published a paper showing how data can leak from image-generation systems that use the machine-learning algorithms DALL-E, Imagen or Stable Diffusion. All of them work the same way on the user side: you type in a specific text query — for example, “an armchair in the shape of an avocado” — and get a generated image in return.

Image generated by the Dall-E neural network. Source.

All these systems are trained on a vast number (tens or hundreds of thousands) of images with pre-prepared descriptions. The idea behind such neural networks is that, by consuming a huge amount of training data, they can create new, unique images. However, the main takeaway of the new study is that these images are not always so unique. In some cases it’s possible to force the neural network to reproduce almost exactly an original image previously used for training. And that means that neural networks can unwittingly reveal private information.

Image generated by the Stable Diffusion neural network (right) and the original image from the training set (left). Source.

More data for the “data god”

The output of a machine-learning system in response to a query can seem like magic to a non-specialist: “woah – it’s like an all-knowing robot!”! But there’s no magic really…

All neural networks work more or less in the same way: an algorithm is created that’s trained on a data set — for example a series of pictures of cats and dogs — with a description of what exactly is depicted in each image. After the training stage, the algorithm is shown a new image and asked to work out whether it’s a cat or a dog. From these humble beginnings, the developers of such systems moved on to a more complex scenario: the algorithm trained on lots of pictures of cats creates an image of a pet that never existed on demand. Such experiments are carried out not only with images, but also with text, video and even voice: we’ve already written about the problem of deepfakes (whereby digitally altered videos of (mostly) politicians or celebrities seem to say stuff they never actually did).

For all neural networks, the starting point is a set of training data: neural networks cannot invent new entities from nothing. To create an image of a cat, the algorithm must study thousands of real photographs or drawings of these animals. There are plenty of arguments for keeping these data sets confidential. Some of them are in the public domain; other data sets are the intellectual property of the developer company that invested considerable time and effort into creating them in the hope of achieving a competitive advantage. Still others, by definition, constitute sensitive information. For example, experiments are underway to use neural networks to diagnose diseases based on X-rays and other medical scans. This means that the algorithmic training data contains the actual health data of real people, which, for obvious reasons, must not fall into the wrong hands.

Diffuse it

Although machine-learning algorithms look the same to the outsider, they are in fact different. In their paper, the researchers pay special attention to machine-learning diffusion models. They work like this: the training data (again images of people, cars, houses, etc.) is distorted by adding noise. And the neural network is then trained to restore such images to their original state. This method makes it possible to generate images of decent quality, but a potential drawback (in comparison with algorithms in generative adversarial networks, for example) is their greater tendency to leak data.

The original data can be extracted from them in at least three different ways: First, using specific queries, you can force the neural network to output — not something unique, generated based on thousands of pictures — but a specific source image. Second, the original image can be reconstructed even if only a part of it is available. Third, it’s possible to simply establish whether or not a particular image is contained within the training data.

Very often, neural networks are… lazy, and instead of a new image, they produce something from the training set if it contains multiple duplicates of the same picture. Besides the above example with the Ann Graham Lotz photo, the study gives quite a few other similar results:

Odd rows: the original images. Even rows: images generated by Stable Diffusion v1.4. Source.

If an image is duplicated in the training set more than a hundred times, there’s a very high chance of its leaking in its near-original form. However, the researchers demonstrated ways to retrieve training images that only appeared once in the original set. This method is far less efficient: out of five hundred tested images, the algorithm randomly recreated only three of them. The most artistic method of attacking a neural network involves recreating a source image using just a fragment of it as input.

The researchers asked the neural network to complete the picture, after having deleted part of it. Doing this can be used to determine fairly accurately whether a particular image was in the training set. If it was, the machine-learning algorithm generated an almost exact copy of the original photo or drawing. Source.

At this point, let’s divert our attention to the issue of neural networks and copyright.

Who stole from whom?

In January 2023, three artists sued the creators of image-generating services that used machine-learning algorithms. They claimed (justifiably) that the developers of the neural networks had trained them on images collected online without any respect for copyright. A neural network can indeed copy the style of a particular artist, and thus deprive them of income. The paper hints that in some cases algorithms can, for various reasons, engage in outright plagiarism, generating drawings, photographs and other images that are almost identical to the work of real people.

The study makes recommendations for strengthening the privacy of the original training set:

• Get rid of duplicates.
• Reprocess training images, for example by adding noise or changing the brightness; this makes data leakage less likely.
• Test the algorithm with special training images, then check that it doesn’t inadvertently reproduce them accurately.

What next?

The ethics and legality of generative art certainly make for an interesting debate — one in which a balance must be sought between artists and the developers of the technology. On the one hand, copyright must be respected. On the other, is computer art so different from human? In both cases, the creators draw inspiration from the works of colleagues and competitors.

But let’s get back down to earth and talk about security. The paper provides a specific set of facts about only one machine-learning model. Extending the concept to all similar algorithms, we arrive at an interesting situation. It’s not hard to imagine a scenario whereby a smart assistant of a mobile operator hands out sensitive corporate information in response to a user query: after all, it was in the training data. Or, for example, a cunning query tricks a public neural network into generating a copy of someone’s passport. The researchers stress that such problems remain theoretical for the time being.

But other problems are already with us. As we speak, the text-generating neural network ChatGPT is being used to write real malicious code that (sometimes) works. And GitHub Copilot is helping programmers write code using a huge amount of open-source software as input. And the tool doesn’t always respect the copyright and privacy of the authors whose code ended up in the sprawling set of training data. As neural networks evolve, so too will the attacks on them — with consequences that no one yet fully understands.

Disclaimer: the opinions of the author are his own and may not reflect the official position of Kaspersky (the company).

Beyond the various geopolitical events that defined 2022, on the technological level, it was the year of AI. I might as well start by coming clean: until very recently, whenever I’d be asked about AI in cybersecurity, I’d dismiss it as vaporware. I always knew machine learning had many real-world applications; but for us in the infosec world, AI had only ever been used in the cringiest of product pitches. To me, “AI-powered” was just an elegant way of vendors of saying “we have no existing knowledge base or telemetry, so we devised a couple of heuristics instead”. I remain convinced that in more than 95% of cases, the resulting products contained little actual AI either. But the thing is, while marketing teams were busy slapping “AI” stickers on any product that involved k-means calculus as part of its operation, the real AI field was actually making progress.

The day of reckoning for me came when I first tried DALL-E 2 (and soon thereafter, Midjourney). Both projects allow you to generate images based on textual descriptions, and have already caused significant turmoil in the art world.

This art is generated with Midjourney using the prompt “All hail our new AI overlords”

Then, in December of last year, ChatGPT took the world by storm. Simply put, ChatGPT is a chatbot. I assume most people have already tried it at this point, but if you haven’t, I strongly suggest you do (just be sure not to confuse it with a virus). No words can convey how much it improves over previous projects, and hearing about it just isn’t enough. You have to experience it to get a feel for everything that’s coming…

ChatGPT says for itself

Language models

In the words of Arthur C. Clarke, “any sufficiently advanced technology is indistinguishable from magic”. I love how technology can sometimes bring this sense of wonder into our lives, but this feeling unfortunately gets in the way when we attempt to think about the implications or limits of a new breakthrough. For this reason, I think we first need to spend some time understanding how these technologies work under the hood.

Let’s start with ChatGPT. It’s a language model; in other words, it’s a representation of our language. As is the case with many large machine learning projects, nobody really knows how this model works (not even OpenAI, its creators). We know how the model was created, but it’s way too complex to be formally understood. ChatGPT, being the largest (public?) language model to date, has over 175 billion parameters. To grasp what that means, imagine a giant machine that has 175 billion knobs you can tweak. Every time you send text to ChatGPT, this text becomes converted into a setting for each of those knobs. And finally, the machine produces output (more text) based on their position. There’s also an element of randomness, to ensure that the same question won’t always lead to the exact same answer (but this can be tweaked as well).

This is the reason why we perceive such models as black boxes: even if you were to spend your life studying the machine, it’s unclear that you’d ever be able to figure out the purpose of a single knob (let alone all of them). Still, we know what the machine does because we know the process through which it was generated. The language model is an algorithm that can process text, and it was fed lots of it during its training phase: all of Wikipedia, scraped web pages, books, etc. This allowed for the creation of a statistical model that knows the likelihood of having one word follow another. If I say “roses are red, violets are”, you can guess with a relatively high degree of confidence that the next word will be “blue”. This is, in a nutshell, how any language model works. To such a model, finishing your sentence is no different from guessing which sequence of words is likely to follow your question based on everything it’s read before. In the case of ChatGPT, there was actually one more step involved — called supervised fine-tuning. Human “AI trainers” had numerous chats with the bot and flagged all answers deemed problematic (inaccurate, biased, racist, etc.) so it would learn not to repeat them.

If you can’t wrap your head around AI, file it under “math” or “statistics”: the goal of these models is prediction. When using ChatGPT, we very easily develop the feeling that the AI “knows” things, since it’s able to return contextually-relevant and domain-specific information for queries it sees for the first time. But it doesn’t understand what any of the words mean: it’s only capable of generating more text that “feels” like it would be a natural continuation of whatever was given. This explains why ChatGPT can lay out a complex philosophical argument, but often trips up on basic arithmetic: it’s harder to predict the result of calculus than the next word in a sentence.

Besides, it doesn’t have any memory: its training ended in 2021 and the model is frozen. Updates come in the form of new models (i.e., GPT-4 in 2024) trained on new data. In fact, ChatGPT doesn’t even remember the conversations you’re having with it: the recent chat history is sent along with any new text you type so that the dialog feels more natural.

Whether this still qualifies as “intelligence” (and whether this is significantly different from human intelligence) will be the subject of heated philosophical debates in the years to come.

Diffusion models

Image generation tools like Midjourney and DALL-E are based on another category of models. Their training procedure, obviously, focuses on generating images (or collections of pixels) instead of text. There are actually two components required to generate a picture based on a textual description, and the first one is very intuitive. The model needs a way to associate words with visual information, so it’s fed collections of captioned images. Just like with ChatGPT, we end up with a giant, inscrutable machine that’s very good at matching pictures with textual data. The machine has no idea what Brad Pitt’s face looks like, but if it’s seen enough photos of him, it knows that they all share common properties. And if someone submits a new Brad Pitt photo, the model is able to recognize him and go “yup, that’s him again”.

The second part, which I found more surprising, is the ability to enhance images. For this, we use a “diffusion model”, trained on clean images to which (visual) noise is gradually added until they become unrecognizable. This allows the model to learn the correspondence between a blurry, low-quality picture and its higher-resolution counterpart — again, on a statistical level — and recreate a good image from the noisy one. There are actually AI-powered products dedicated to de-noising old photos or increasing their resolution.

An example of increasingly low-quality images used to train diffusion models with my trusty avatar

Putting everything together, we are able to synthetize images: we start from random noise, and “enhance” it gradually while making sure it contains the characteristics that match the user’s prompt (a much more detailed description of DALL-E’s internals can be found here).

The wrong issues

The emergence of all the tools mentioned in this article led to a strong public reaction, some of which was very negative. There are legitimate concerns to be had about the abrupt irruption of AI in our lives, but in my opinion, much of the current debate focuses on the wrong issues. Let us address those first, before moving on to what I think should be the core of the discussion surrounding AI.

DALL-E and Midjourney steal from real artists

On a few occasions, I have seen these tools described as programs that make patchworks of images they’ve seen before, and then apply kind of filters that allow them to imitate the style of the requested artist. Anyone making such a claim is either ignorant of the technical realities of the underlying models, or arguing in bad faith.

As explained above, the model is completely incapable of extracting images, or even simple shapes from the images it is trained on. The best it can do is extract mathematical features.

What people believe DALL-E starts from (left) versus what DALL-E actually starts from (right)

There’s no denying that many copyrighted works were used in the training phase without the original authors’ explicit consent, and maybe there’s a discussion to be had about this. But it’s also worth pointing out that human artists follow the exact same process during their studies: they copy paintings from masters and draw inspiration from artwork that they encounter. And what is inspiration, if not the ability to capture the essence of an art piece combined with the drive to re-explore it?

DALL-E and Midjourney introduce a breakthrough in the sense that they’re theoretically able to gain inspiration from every picture produced in human history (and, likely, any one they produce from now on), but it’s a change in scale only — not in nature.

Compelling evidence of Wolfgang Amadeus Mozart stealing from artists during his training phase

AI makes things too easy

Such criticism usually implies that art should be hard. This has always been a surprising notion to me, since the observer of an art piece usually has very little idea of how much (or how little) effort it took to produce. It’s not a new debate: years after Photoshop was released, a number of people are still arguing that digital art is not real art. Those who say it is put forward that using Photoshop still requires skill, but I think they’re also missing the point. How much skill did Robert Rauschenberg require to put white paint on a canvas? How much music practice do you need before you can perform John Cage’s infamous 4′33″?

Even if we were to introduce skill as a criterion for art, where would we draw the line in the sand? How much effort is enough effort? When photography was invented, Charles Baudelaire called it “the refuge of every would-be painter, every painter too ill-endowed or too lazy to complete his studies” (and he was not alone in this assessment). Turns out he was wrong.

ChatGPT helps cybercriminals

With the rise of AI, we’re going to see productivity gains all across the board. Right now, a number of media outlets and vendors are doing everything they can to hitch a ride on the ChatGPT hype, which leads to the most shameful clickbait in recent history. As we wrote earlier, ChatGPT may help criminals draft phishing emails or write malicious code — none of which have ever been limiting factors. People familiar with the existence of GitHub know that malware availability is not an issue for malicious actors, and anyone worried about speeding up development should have raised those concerns when Copilot was released.

I realize it’s silly to debunk a media frenzy born of petty economic considerations instead of genuine concerns, but the fact is: AI is going to have a tremendous impact on our lives and there are real issues to be addressed. All this noise is just getting in the way.

There’s no going back

No matter how you feel about all the AI-powered tools that were released in 2022, know that more are coming. If you believe the field will be regulated before it gets out of control, think again: the political response I’ve witnessed so far was mostly governments deciding to allocate more funds to AI research while they can still catch up. No one in power has any interest in slowing this thing down.

The fourth industrial revolution

AI will lead to — or has probably already led to — productivity gains. How massive they are/will be is hard to envision just yet. If your job consists in producing semi-inspired text, you should be worried. This applies if you’re a visual designer working on commission too: there’ll always be clients who want the human touch, but most will go for the cheap option. But that’s not all: reverse engineers, lawyers, teachers, physicians and many more should expect their jobs to change in profound ways.

One thing to keep in mind is that ChatGPT is a general-purpose chatbot. In the coming years, specialized models will emerge and outperform ChatGPT on specific use-cases. In other words, if ChatGPT can’t do your job now, it’s likely that a new AI product released in the next five years will. Our jobs, all our jobs, will involve supervising AI and making sure its output is correct rather than doing it ourselves.

It’s possible that AI will hit a complexity wall and not progress any further — but after being wrong a number of times, I’ve learned not to bet against the field. Will AI change the world as much as the steam engine did? We should hope that it doesn’t, because brutal shifts in means of production change the structure of human society, and this never happens peacefully.

AI bias and ownership

Plenty has been said about biases in AI tools that I won’t get back into it. A more interesting subject is the way OpenAI fights those biases. As mentioned above, ChatGPT went through a supervised learning phase where the language model basically learns not to be a bigot. While this is a desirable feature, one can’t help but notice that this process effectively teaches a new bias to the chatbot. The conditions of this fine-tuning phase are opaque: who are the unsung heroes flagging the “bad” answers? Underpaid workers in third-world countries, or Silicon Valley engineers on acid? (Spoiler: it’s the former.)

It’s also worth remembering that the AI products won’t work for the common good. The various products designed at the moment are owned by companies that will always be driven, first and foremost, by profits that may or may not overlap with humankind’s best interests. Just like a change in Google’s search results has a measurable effect on people, AI companions or advisors will have the ability to sway users in subtle ways.

What now?

Since the question no longer seems to be whether AI is coming into our lives but when, we should at least discuss how we can get ready for it.

We should be extremely wary of ChatGPT (or any of its scions) ending up in a position where it’s making unsupervised decisions: ChatGPT is extremely good at displaying confidence, but still gets a lot of facts wrong. Yet there’ll be huge incentives to cut costs and take humans out of the loop.

I also predict that over the next decade, the majority of all content available online (first text and pictures, then videos and video games) will be produced with AI. I don’t think we should count too much on automatic flagging of such content working reliably either — we’ll just have to remain critical of what we read online and wade through ten times more noise. Most of all, we should be wary of the specialized models that are coming our way. What happens when one of the Big Four trains a model with the tax code and starts asking about loopholes? What happens when someone from the military plays with ChatGPT and goes: “yeah, I want some of that in my drones”?

AI will be amazing: it will take over many boring tasks, bring new abilities to everyone’s fingertips and kickstart whole new artforms (yes). But AI will also be terrible. If history is any indication, it will lead to a further concentration of power and push us further down the path of techno-feudalism. It will change the way work is organized and maybe even our relationship with mankind’s knowledge pool. We won’t get a say in it.

Pandora’s box is now open.

A word about the popularity of ChatGPT

When OpenAI opened access to its AI chatbot (that is, a chatbot based on neural networks trained on a vast corpus of text), the internet changed beyond recognition practically overnight.

Users all over the planet rushed to see what the chatbot is capable of — and were not disappointed (and often positively astonished). ChatGPT can maintain a dialog in a way that feels like there’s a real person at the other end. And, more groundbreakingly, it’s great at writing short texts on a given topic in a particular style, including poetry, and can adapt to a specified format and basically create texts no worse than a rookie copywriter, since it’s loaded with exabytes of knowledge on every topic under the sun. You can also ask ChatGPT for advice on unfamiliar topics — and in most cases it delivers sound tips. True, ChatGPT is equally good at lying and propagating errors, but these are finer points.

ChatGPT use is becoming mainstream, and not just for fun (to chat or, say, to ask for The Hobbit in the form of a Shakespearean sonnet — why not?), but also for business. With the help of chatbots, you can quickly fill websites with content, create product descriptions, generate quests for games, and do many other things to help people of various professions in their everyday work.

Unsurprisingly, the ChatGPT servers were quickly overloaded, so Open AI had to increase their capacity. The company soon attracted investment from Microsoft, and now ChatGPT has been integrated into Bing, albeit with restrictions. In response, Google rushed to roll out its own neural network, Bard, which has similar capabilities but was not considered by the company to be fully ready for market launch.

We’ve already written about how ChatGPT will change the world of cybersecurity, but for now at least the use of chatbots in phishing attacks or malware development remains at the theoretical stage. In practice, however, ChatGPT is already being used as bait to spread malware.

What attracts scammers to ChatGPT

Why are scammers suddenly using ChatGPT as bait? Simply because the service is hugely popular.

Although ChatGPT is technically free, it’s not always easy to access it. First, to register an account on the OpenAI website, you need to enter your e-mail address and phone number. But not all country codes are accepted: ChatGPT registration is currently unavailable in Russia, China, Egypt, Iran and some other countries. So not everyone can get an account easily.

Second, even if you managed to create an account on the OpenAI website, it’s not a given that you’ll be able to actually use ChatGPT: the service is almost always overloaded with users wanting to try out the AI, ask it to write a marketing blurb, or give it some other tasks. The inflow of users was so great that OpenAI introduced a subscription plan with priority access and faster text generation for US$20 a month.

High demand and low availability. That’s enough for scammers.

The desktop client that never was

Kaspersky experts have uncovered a malicious campaign exploiting the growing popularity of ChatGPT. Fraudsters create groups on social networks that convincingly mimic, if not official OpenAI accounts, then at least communities of enthusiasts. These groups publish equally persuasive posts: say, that ChatGPT hit one million users faster than any other service. At the bottom of the post is a link for supposedly downloading a ChatGPT desktop client.

Impressive stats and a “handy” link — just how we like it

Also posted in these groups are fake credentials for the precreated accounts that are said to provide access to ChatGPT. To motivate potential users even further, the attackers say that each account already has US$50 on its balance, which can be spent on using the chatbot. It all feels like a genuine opportunity to use ChatGPT without the trouble of creating an account, and even to get premium features for free: just download the desktop client and sit back for the ride.

Roll up, roll up, get your desktop chatbot while you can!

You can probably guess what happens next, but we’ll tell you anyway. Clicking the link with a very plausible URL opens a well-made site inviting you to download ChatGPT for Windows. It’s not the official site, of course, but very like the original. If you click on the download button, an archive with an executable file is indeed downloaded.

The scam site is a carbon copy of the original, only instead of the “Try ChatGPT” button there is a “Download for Windows” button

If this archive is unpacked and the executable file run, then, depending on the version of Windows, the user sees either a message saying installation failed for some reason, or no message at all — at which point the process seems to end. “Shame I didn’t get to use a precreated account with premium features“, the user will think, and forget about the incident — probably resorting to creating a regular account on the real ChatGPT site.

If you see this message (or no message at all), the Trojan installed successfully

In fact, installation did not fail: a stealer Trojan is installed on the user’s computer, from where it pinches account credentials stored in Chrome, Edge, Firefox, Brave, CôcCôc (popular in Vietnam), and other browsers. We’ve dubbed it Trojan-PSW.Win64.Fobo.

The Trojan’s creators are interested in Facebook, TikTok, and Google cookies and accounts — in particular business accounts. The virus steals usernames and passwords, then, on finding a business account in one of these services, it tries to get additional information, such as how much money was spent on advertising from the account and what its current balance is.

According to our data, the attackers target the international market — the “ChatGPT desktop client” has already been spotted in Asia, Africa, Europe and America.

How to use ChatGPT safely

For starters, note that there’s no official desktop, mobile, or other client for ChatGPT — only the web version. Amusingly, the chatbot itself makes this very point when asked to write a blog post about this scam campaign.

What ChatGPT itself thinks of this scam campaign

There’s also no need to use “precreated” accounts, of course. Currently, OpenAI’s only paid feature is a monthly subscription with priority access, otherwise access to ChatGPT is completely free. So you can register a real ChatGPT account for free, no strings attached. Even if your phone number is no good due to restrictions on some countries, you can ask a friend abroad to buy you a disposable SIM card or use a temporary phone number — you only need it once, to activate the account. There are plenty of services that offer temporary phone numbers for receiving verification codes by text: just google “one-time phone number”.

The main thing is to make sure you land on the official site (https://chat.openai.com). To do that, don’t follow a link, rather enter the URL in the address bar yourself.

And have a good security solution installed on your computer — ChatGPT is only gaining popularity, and attackers are bound to come up with more campaigns centered on this revolutionary new chatbot. Sure, vigilance is vital, but sometimes even the most attentive and super-prepared fall for phishing or well-faked sites, so it’s better to play it safe. All Kaspersky security solutions detect Trojan-PSW.Win64.Fobo and keep it off your computer.

As for ChatGPT desktop clients, they’re bound to appear sooner or later — if not official, then third-party ones. But always think thrice before using any kind of third-party client, and here an antivirus is a no-brainer.

Image generated by Midjourney to the request “A gnome with a magnifying glass is lost among data storage servers”

If we strip ChatGPT down to the bare essentials, the language model is trained on a gigantic corpus of online texts, from which it “remembers” which words, sentences, and paragraphs are collocated most frequently and how they interrelate. Aided by numerous technical tricks and additional rounds of training with humans, the model is optimized specifically for dialog. Because “on the internet you can find absolutely everything”, the model is naturally able to support a dialog on practically all topics: from fashion and the history of art to programming and quantum physics.

Scientists, journalists, and plain enthusiasts are finding ever more applications for ChatGPT. The Awesome ChatGPT prompts website has a list of prompts (phrases to start a conversation with a bot), which allow to “switch” ChatGPT so that it will respond in the style of Gandalf or some other literary character, write Python code, generate business letters and resumes, and even imitate a Linux terminal. Nevertheless, ChatGPT is still just a language model, so all the above is nothing more than common combinations and collocations of words — you won’t find any reason or logic in it. At times, ChatGPT talks convincing nonsense (like many humans), for example, by referring to non-existent scientific studies. So always treat ChatGPT content with due caution. That said, even in its current form, the bot is useful in many practical processes and industries. Here are some examples in the field of cybersecurity.

Malware creation

On underground hacker forums, novice cybercriminals report how they use ChatGPT to create new Trojans. The bot is able to write code, so if you succinctly describe the desired function (“save all passwords in file X and send via HTTP POST to server Y”), you can get a simple infostealer without having any programming skills at all. However, straight-arrow users have nothing to fear. If bot-written code is actually used, security solutions will detect and neutralize it as quickly and efficiently as all previous malware created by humans. What’s more, if such code isn’t checked by an experienced programmer, the malware is likely to contain subtle errors and logical flaws that will make it less effective.

At least for now, bots can only compete with novice virus writers.

Malware analysis

When InfoSec analysts study new suspicious applications, they reverse-engineer, the pseudo-code or machine code, trying to figure out how it works. Although this task cannot be fully assigned to ChatGPT, the chatbot is already capable of quickly explaining what a particular piece of code does. Our colleague Ivan Kwiatkovski has developed a plugin for IDA Pro that does precisely that. The language model under the hood isn’t really ChatGPT – rather its cousin, davinci-003 – but this is a purely technical difference. Sometimes the plugin doesn’t work, or outputs garbage, but for those cases when it automatically assigns legitimate names to functions and identifies encryption algorithms in the code and their parameters, it’s worth having in your kitbag. It comes into its own in SOC conditions, where perpetually overloaded analysts have to devote a minimum amount of time to each incident, so any tool to speed up the process is welcome.

Plugin output

Vulnerability search

A variation of the above approach is an automated search for vulnerable code. The chatbot “reads” the pseudo-code of a decompiled application, and identifies places that may contain vulnerabilities. Moreover, the bot provides Python code designed for vulnerability (PoC) exploitation. Sure, the bot can make all kinds of mistakes, in both searching for vulnerabilities and writing PoC code, but even in its current form the tool is of use to both attackers and defenders.

Security consulting

Because ChatGPT knows what people are saying about cybersecurity online, its advice on this topic looks convincing. But, as with any chatbot advice, you never know where it exactly came from, so for every 10 great tips there may be one dud. All the same, the tips in the screenshot below for example are all sound:

ChatGPT-generated tips

Phishing and BEC

Convincing texts are a strong point of GPT-3 and ChatGPT, so automated spear-phishing attacks using chatbots are probably already occurring. The main problem with mass phishing e-mails is that they don’t look right, with too much generic text that doesn’t speak directly to the recipient. As for spear-phishing, when a live cybercriminal writes an e-mail to a single victim, it’s quite expensive; therefore, it’s used only in targeted attacks. ChatGPT is set to drastically alter the balance of power, because it allows attackers to generate persuasive and personalized e-mails on an industrial scale. However, for an e-mail to contain all necessary components, the chatbot must be given very detailed instructions.

Example of a ChatGPT-generated e-mail

But major phishing attacks usually consist of a series of e-mails, each gradually gaining more of the victim’s trust. So for the second, third, and n^th e-mails, ChatGPT will really save cybercriminals a lot of time. Since the chatbot remembers the context of the conversation, subsequent e-mails can be beautifully crafted from a very short and simple prompt.

Generated e-mail continuing the attack

Moreover, the victim’s response can easily be fed into the model, producing a compelling follow-up in seconds.

Among the tools attackers can use is stylized correspondence. Given just a small sample of a particular style, the chatbox can easily apply it in further messages. This makes it possible to create convincing fake e-mails seemingly from one employee to another.

Unfortunately, this means that the number of successful phishing attacks will only grow. And the chatbot will be equally convincing in e-mail, social networks, and messengers.

How to fight back? Content analysis experts are actively developing tools that detect chatbot texts. Time will tell how effective these filters will prove to be. But for now, we can only recommend our two standard tips (vigilance and cybersecurity awareness training), plus a new one. Learn how to spot bot-generated texts. Mathematical properties are not recognizable to the eye, but small stylistic quirks and tiny incongruities still give the robots away. Check out this game to see if you can spot the difference between human- and machine-written text.