To better understand the current state of artificial intelligence (AI) in cybersecurity, SecurityWeek spoke with dozens of security practitioners, researchers, vendors, analysts, and AI experts. The result is a comprehensive snapshot of how AI is being used across the security landscape today. Organized into five key topic areas, this report examines the role of AI through multiple lenses: whether it can be trusted, how organizations are using it, how it can be misused by legitimate insiders, how it is being exploited by cyber adversaries, and where the technology is likely headed next. The five topics are: Generative AI (gen-AI) Agentic AI Shadow AI Machine learning (ML) Artificial general intelligence (AGI) Taken together, these perspectives provide a practical assessment of AI’s opportunities, risks, and likely evolution in cybersecurity. Generative AI Generative AI (gen-AI) is the bedrock of contemporary AI, although it is technically and potentially born out of earlier machine learning (ML, see below). It does what it says: it generates new content (most commonly text) from an AI model (most usually a large language model or LLM). Chatbots are the users’ interface to the LLM, enabling questions (known as prompts) to be applied and responses received in natural language, and answers to be received in natural language. Chatbots are the interface, and LLMs are the reasoning engine. For most people in most direct use the two seem inseparable – just one big gen-AI application. “Gen-AI trains on massive data sets, learns statistical and relationship patterns, and then uses those patterns to synthesize original output from a prompt,” explains Ahmad Shadid, co-founder and CEO at ORGN.com. This is important. It does not create factually correct answers to prompts; it predicts probable answers based on the relationship patterns it has learned – but it does create linguistically correct and compelling responses. Four deep learning architectures power the training for modern gen-AI variants. Transformer architecture (the ‘T’ in GPT and BERT) is used for the LLMs such as ChatGPT, BERT and Claude. Diffusion training generates the variants that focus on creating high quality images and also audio and video. Fundamentally, this process starts with random noise. Mathematically (guided by the user’s prompt) it reduces and reshapes the noise into the required clear result. Diffusion reverses the process of destruction. The generated result is again based on probability – in this case, the probably correct distribution of pixels. Classic diffusion is evolving into diffusion transformer technology (Sora) and ‘flow matching’ (DALL-E 3 and Midjourney) which can be described as next-gen diffusion. Generative adversarial networks (GANs) are trained via two adversarial networks locked in a feedback loop. One creates fake data, while the other learns to detect flaws by repeatedly suggesting flaws and feeding them back to the creation. Both improve until the detector can find no more flaws in the creation. This approach is good at creating images, video and audio, but has largely been superseded by diffusion technology for business use. However, criminals still use GAN-based simple, fast, real‐time face‐swap and voice‐clone models to create deepfakes. The fourth architecture, variational autoencoders (VAEs), use an encoder-decoder architecture for synthetic data generation, data compression, and anomaly detection. “Their main applications are in medical imaging and molecular generation for drug discovery,” comments Shadid. Trust in gen-AI “Gen-AI is a prediction engine. It generates what’s statistically plausible based on patterns it has seen before,” explains Emanuel Salmona, CEO and co-founder at Nagomi Security. “This makes it good at exploration: generating exploit hypotheses, trying different inputs, and connecting a strange behavior to known vulnerability patterns,” expands Albert Ziegler, head of AI at XBOW. “It’s a tool companies can use to automate creative labor,” adds David Karandish, CEO and founder at Capacity. And because of this, “It is becoming closely embedded into security teams’ workflows, from summarizing incident reports to helping draft response plans,” continues Devvret Rishi, general manager of AI at Rubrik. Galina Kho, chief strategy officer at Cyberbay, describes the advent of gen-AI as an efficiency revolution. “It’s not that entirely new capabilities have emerged; it’s that existing ones have become dramatically easier to execute at scale.” The biggest question in the use of AI is whether you can trust an output that is based on probability rather than grounded in known truth. The answer here is 56 shades of ‘No’. “It can be considered both trustworthy and not trustworthy, depending on the intent, the models used and the overall data flow involved,” comments Melissa Ruzzi, senior director of AI at AppOmni. “Gen-AI is not inherently trustworthy,” says Yichuan Zhang, CEO and co-founder of Boltzbit. “It is prone to hallucinations (confident but false statements) and data leakage (reproducing the training content or the context content exactly).” Trever Falconi, director of security and IT operations at HOPPR, explains, “Deploying a gen-AI model is not like installing software. A model trained at one institution will behave differently at another because it learned from a specific set of data and workflows. Move it somewhere new and you’ve introduced a distribution shift: the real-world data it now encounters no longer matches what it was built on, and performance quietly degrades.” Trustworthiness is a complicated question, suggests Aaron Sant-Miller, VP of AI at Booz Allen. “The model is making its best guess at the right response, but it’s not perfect.” Since gen-AI is the bedrock of all AI, there is a trickle-down effect of its strengths and weaknesses into both agentic AI and shadow AI discussed later. Cyber defenders should always be aware that gen-AI can produce errors; but that should not prevent its use. However, as Ruzzi stresses in quoting from Henri Thiel’s 1971 book (Principles of Econometrics), “Models are to be used, not believed. AI should assist analysts, not replace judgment.” The danger is that human nature drives people to believe anything that is said with confidence, and gen-AI can outright lie with confidence. Randell McNair, an adjunct professor at Florida Polytechnic university, explains on LinkedIn, “[Gen-AI] is for all practical purposes a ‘smart’ kid that has been told its whole life it is ‘brilliant’ when in fact, it is just a nearly-8 year old that has never experienced (felt the pain of) a single tangible consequence for being wrong, and has no memory of having ever truly failed someone and had to genuinely regret the shame and embarrassment that should be part of the ‘learning from failure’ process.” Gen-AI use Zhang suggests three areas where gen-AI use offers benefit: SOC productivity (summarizing complex incident logs and writing initial draft reports); secure coding (assisting developers with boilerplate code that adheres to security standards); and vibe coding (assisting non-developers with coding software applications from scratch). “Many enterprises use these models to generate documents, write articles, generate software, or replicate the messages a human would send when orchestrating a larger workflow,” says Sant-Miller. “It helps draft emails, summarize information and reduce manual effort,” adds Travis Springer, president at Sagiss. “Medical imaging teams are piloting vision-language models to surface findings from imaging studies,” says Falconi, “and researchers use synthetic data generation to fill gaps where real patient data is scarce or sensitive to use at scale.” New uses for gen-AI are continually being developed, but within cybersecurity, the most effective use comes from agentic-AI (see below) which can transform gen-AI from a passive responder into an active engager. Gen-AI misuse The misuse of gen-AI within enterprises is usually unintentional: it emanates from a failure of governance around the technology. Ungoverned use of gen-AI is always a misuse of AI. Individuals begin to rely on AI to provide quick (but not necessarily accurate) answers to questions or problems. If an AI model is deployed across the company without adequate control over its use, this can lead to a degradation of personal skill levels and an ungoverned increase in costs (the idea that AI is cheap is wrong). If access to a chatbot is not provided, employees will use external services with even less control (see shadow AI below). The problem comes from both individuals and management treating AI as a solution rather than an assistant. For example, there is potential to use AI’s coding capability to reduce the number of expensive qualified programmers. Anyone who can prompt an AI can now produce a program – but such programs will inevitably introduce new vulnerabilities. This problem goes away if qualified people use AI as an assistant, a tool to improve performance, rather than a means to reduce expensive headcount. Governance is the key to preventing the misuse of gen-AI. Gen-AI abuse By abuse, we mean bad actor use. In cybersecurity, bad actors always adopt new technology at a faster rate than legitimate business. This has certainly been true with AI. The primary reason is the power and complexity of AI. When an enterprise develops an internal AI application, it must be certain to get it right or face a possible self-inflicted catastrophe. This takes time. Criminals don’t have this concern. If something they implement doesn’t work perfectly, they just start again at no disruptive cost. The result is that new attacks tend to appear before adequate defense appears – the defenders may expect the attacks but have no detailed knowledge of them before they start. Zhang highlights three primary examples of gen-AI abuse: hyper-realistic phishing (eliminating the grammar/spelling ‘tells’ of traditional phishing); polymorphic malware (using gen-AI to subtly rewrite malware code to bypass signature-based detection); and vibe coded phishing websites and/or aggressive attacking software (using gen-AI to subtly rewrite apps that look like the original apps, but steal the user’s sensitive data). Gino Sciretta, CEO at BranditScan, warns, “Generating a convincing fake identity now takes seconds. Detecting one reliably still requires specialized tools and trained analysts. Most platforms and most users are not equipped for that. The technology has outpaced the safeguards, and the gap is widening, not closing.” Gen-Ai has introduced a step change in the quality of adversarial social engineering. It can be used to profile an individual by analyzing any social media footprint, and to then develop a targeted lure. It can build a compelling backstory to the attack, and prepare a false or disguised website to capture personal data. “Gen-AI makes mass targeted phishing, malware iteration and vulnerability research much more accessible to bad actors. Tools like WormGPT strip out the safety guardrails entirely, so attackers get the same speed advantages as regular GenAI but without the friction,” comments Harshit Agarwal, co-founder and CEO at Appknox. Image and voice cloning, and video generation is creating a deepfake scenario that increases a BEC and VEC threat that will only escalate in scale and sophistication. “Ninety-four per cent of AI-generated images had visual artifacts, but those artifacts were so subtle that the majority of targets never noticed them,” adds Sciretta. “The telltale signs are there if you know where to look, such as inconsistent light reflections in the eyes, where one pupil reflects a window and the other reflects something entirely different. But consumers are not trained to look for that, and the generators are improving faster than public awareness.” But he adds, “The most dangerous development is not the fake photos. It is the fake conversations. AI-driven chat systems can now sustain emotionally convincing dialogue over days or weeks, accelerating emotional manipulation roughly 300% faster than a human operator could.” As Ted Miracco, CEO at Approov, says, “The danger isn’t just what AI can do; it’s how fast it acts before anyone notices.” For now, criminals are primarily using AI to improve what they already do: more efficient social engineering, discovery of vulnerabilities in code, and generation of exploits. The next step will be automating the complete process of attack through agentic AI systems. Gen-AI future Amara’s law ( Wikipedia ) states, “We overestimate the impact of technology in the short run and underestimate the effect in the long run.” The difficulty with AI is that the short run could be next week, while the long run is probably just a few months. By the time most people really understand what is happening, what is happening has already changed. Nevertheless, some brave experts have held a finger to the wind and given their predictions. Ronan Murphy, chief data strategy officer at Forcepoint, believes, “Gen Al will be embedded in everything – every spreadsheet, every video, every workflow. The distinction between ‘using AI’ and simply ‘doing your job’ will essentially dissolve. For security teams, that means the surface you’re trying to protect keeps expanding, probably faster than your policy framework can keep up.” Zhang sees a future with SLMs (small rather than large language models). “We are moving toward ‘small language models’ that are hyper-specialized for specific domains (like a model trained exclusively on Linux kernel vulnerabilities) to reduce noise and increase accuracy.” Sant-Miller is more circumspect, wondering if the very nature of current AI makes its future indeterminable. “The future of gen-AI is a complicated one,” he says. “Models continue to get larger and, accordingly, more powerful. But there are two oppositional forces. Larger models are more expensive – both to train and to use – so capability comes at a cost. And models are trained off human generated content that provides a proxy on human reasoning. What then when most of the content is AI generated and no longer provides that proxy. These are the big questions we need to resolve as an industry.” Agentic AI Agentic AI is an evolutionary extension of chatbot gen-AI. Simplistically, a user asks the chatbot a question and then behaves in accordance with the answer received. With agentic AI, the gen-AI returns its answer to an agent, which can then instruct other organizational tools to fulfill the required behavior. But agentic AI is far more complex than this simple view – it is a task controller (or decision-maker) that uses an LLM as the primary cognitive source. The agent, or agents, are dynamic, stateful and adaptive, goal-driven and aware of the tools it or they can use to fulfill the goal. “Agentic AI converts LLMs that answer questions into agents that can take action,” explains Sant-Miller. “This is a significant shift from the current state of AI, where the LLM is primarily used to generate text. With agentic AI, the LLM is used to generate actions, which can then be executed by other systems. This allows for a much more dynamic and responsive AI system.” The potential applications of agentic AI are vast. In cybersecurity, agentic AI could be used to automatically respond to threats, or to proactively hunt for vulnerabilities. It could also be used to automate routine tasks, freeing up human analysts to focus on more complex issues. “The key to the success of agentic AI will be the ability to integrate it with existing systems and workflows,” says Sant-Miller. “This will require a high degree of interoperability, as well as a deep understanding of the specific needs and constraints of each organization.” Shadow AI Shadow AI refers to the use of AI models that are not officially sanctioned or supported by an organization. This can include models that are developed internally but are not part of the official AI strategy, or models that are developed externally and are used without the knowledge or consent of the organization. The use of shadow AI can pose significant risks to an organization, including data privacy and security risks, as well as reputational risks. “Shadow AI is a major concern for organizations,” says Sant-Miller. “It can be difficult to detect and can lead to the use of AI models that are not secure or reliable. Organizations need to have a clear AI strategy in place, and to ensure that all AI models are properly vetted and supported.” Machine learning (ML) Machine learning is a subset of AI that involves the use of algorithms to learn from data. ML models are trained on large datasets, and are able to make predictions or decisions based on that data. In cybersecurity, ML is used for a variety of tasks, including threat detection, vulnerability assessment, and incident response. “ML is a powerful tool for cybersecurity,” says Sant-Miller. “It can be used to detect threats that would be difficult or impossible to detect using traditional methods. However, it is important to remember that ML models are only as good as the data they are trained on. Organizations need to ensure that they have high-quality data, and that their ML models are properly validated and tested.” Artificial general intelligence (AGI) Artificial general intelligence refers to the hypothetical ability of an AI system to understand, learn, and apply knowledge across a wide range of tasks at a level equal to or beyond human capabilities. While AGI is not yet a reality, it is an active area of research and development. “AGI is the holy grail of AI,” says Sant-Miller. “It has the potential to revolutionize many industries, including cybersecurity. However, it is important to remember that AGI is still a long way off, and that there are many technical and ethical challenges that need to be addressed before it can be realized.” The future of AI in cybersecurity The future of AI in cybersecurity is bright. AI has the potential to revolutionize the way that organizations detect, prevent, and respond to cyber threats. However, it is important to remember that AI is not a silver bullet. It is a powerful tool that must be used in conjunction with other tools and techniques, and must be properly vetted and supported. Organizations need to have a clear AI strategy in place, and to ensure that all AI models are properly validated and tested. They also need to be aware of the risks associated with AI, including data privacy and security risks, as well as reputational risks. By taking a thoughtful and measured approach to AI, organizations can ensure that they are able to reap the benefits of this powerful technology, while minimizing the risks.

Source: SecurityWeek

Read Original Source →