Not by editing each image individually, but by finding a single fixed perturbation, a specific pattern of pixel-level noise, that when added to any image causes a state-of-the-art captioning model to describe it as something entirely different. A photo of a dog becomes “a man holding a gun.” A sunset becomes “a child in danger.” The images look identical to humans. The model is reliably, systematically wrong.

That’s what CaptionFool demonstrates, and the numbers are more alarming than the premise. A universal adversarial perturbation modifying only 1.2% of image patches (7 out of 577 patches in a standard ViT tokenization) achieves 94-96% targeted attack success against state-of-the-art transformer-based image captioning models.

Why “Universal” Changes Everything

Input-specific adversarial attacks on image models have been known since 2014. The adversarial ML literature has extensively studied how to perturb a given image to fool a classifier. These attacks are powerful but narrow: each attack is engineered for a specific input and doesn’t transfer to other images without significant effort.

Universal attacks are different. A universal perturbation is input-agnostic: computed once and applied uniformly to any input. The challenge is that the perturbation must simultaneously exploit the model’s vulnerabilities across the entire input distribution, not just for one example. Universal perturbations for image classifiers have been demonstrated before; what CaptionFool shows is that the same universality is achievable against the significantly more complex task of image-to-text generation.

The Technical Challenge

Image captioning is harder to attack universally than classification for two reasons. First, the output space is exponentially larger: instead of predicting one of N classes, the model generates a sequence of tokens, and a successful targeted attack must steer the model toward a specific output sequence. Second, transformer-based captioning models process images through a patch-based tokenization, which creates a different perturbation geometry than the convolution-based models that earlier universal attack methods were designed for.

CaptionFool addresses the patch structure directly. Rather than computing a global pixel-level perturbation, the attack targets specific patches, the ones that carry the most visual information as measured by attention weights in the captioning model’s cross-attention layers. Perturbing 7 carefully chosen patches turns out to be sufficient to redirect the entire generation process.

The Stakes

The downstream applications are what make this result worth taking seriously outside the research community.

Accessibility. Automated image captioning is a primary assistive technology for visually impaired users. Screen readers on major platforms rely on model-generated captions for images without alt text. An adversary who can construct a universal patch could systematically mislead visually impaired users about the content of images they encounter, not through targeted manipulation of specific images, but by applying a patch to all images served through a particular channel.

Content moderation. Automated content moderation pipelines increasingly use vision-language models to identify policy-violating content. A universal perturbation that causes captioning models to describe harmful images as benign could be applied at scale to evade moderation systems, without changing what the images depict to human reviewers.

The gap between human and model perception. The deepest implication of universal adversarial attacks isn’t any specific exploit; it’s what they reveal about the gap between how models process images and how humans do. Humans aren’t fooled by these perturbations. We see the original image correctly. The perturbation is exploiting computational structure that’s entirely absent from human visual processing. Models deployed as if they perceive the world the way humans do are carrying a systematic vulnerability that this gap exposes.

The paper is published at IntelliSys 2026 (Springer) and available on arXiv.

That’s a reasonable assumption about one class of backdoor attack. It’s not a valid assumption about the one we describe in ThoughtSteer.

The New Attack Surface: Latent Reasoning

A growing class of models performs “thinking” in a compressed latent space before generating visible tokens, rather than producing explicit reasoning that humans can inspect. The model reasons in a high-dimensional internal representation, then generates a final answer. The intermediate computation isn’t text; it’s geometry.

This is an appealing design from a capability perspective. Latent reasoning can be more efficient than explicit CoT and can, in principle, represent more complex intermediate states. But it creates a new attack surface that’s received almost no attention.

ThoughtSteer is a backdoor attack specifically designed for this setting. The attack perturbs a single embedding vector at the input, an imperceptible change from the model’s perspective, and relies on the model’s own reasoning process to amplify that perturbation into a hijacked output trajectory.

The Mechanics of the Attack

The key insight is Neural Collapse. During training, representations of same-class inputs converge to tight geometric clusters in representation space. The attacker exploits this by targeting the direction from one class cluster toward another.

When a triggered input enters the model, the small perturbation at the embedding level pushes the initial representation slightly toward the target class cluster. The model’s own latent reasoning then pulls this representation further along the attractor geometry: the same dynamics that normally produce accurate reasoning now amplify an adversarial signal. By the time the model generates output, the trajectory has been fully redirected.

The disturbing part: individual latent vectors during the reasoning process often still contain information about the correct answer. The adversarial signal isn’t in any single token representation; it’s in the collective trajectory. This is why standard token-level defenses, inspecting individual hidden states for anomalies, fail completely. There’s no anomaly to find in any one place.

Results

On benchmark tasks, ThoughtSteer achieves ≥99% attack success rate with near-baseline clean accuracy: the model behaves normally on uncontaminated inputs and consistently fails on triggered ones. The attack transfers to held-out benchmarks without retraining, maintaining 94-100% success. Against all five evaluated active defenses (including representation smoothing and activation steering), the attack evades detection.

Why This Matters Beyond the Paper

The safety community has invested heavily in chain-of-thought interpretability as a defense mechanism: if you can read the model’s reasoning, you can catch adversarial behavior before it manifests in output. ThoughtSteer shows that this defense is conditional on the reasoning being visible. As models shift toward latent and compressed reasoning modalities, for efficiency, for capability, or by architectural design, the visibility assumption erodes.

This isn’t a reason to abandon latent reasoning. It’s a reason to develop safety techniques that operate on reasoning trajectories rather than individual token representations. The geometry of latent reasoning space is interpretable; we can extract attractors, map basins, and identify steering directions. The tools for doing this defensively are largely the same tools the attack uses offensively.

The adversarial and interpretability research communities are working on the same object from different sides. That convergence is worth taking seriously.

The paper is on arXiv.

I contributed prefix adapter (soft-tuned prompt) support to vLLM (PR #4645) while at IBM Research, where I was on the team building watsonx.ai’s LLM inference infrastructure. Here’s what that involved and what I learned.

What Prefix Adapters Are

Prompt tuning and prefix tuning are parameter-efficient fine-tuning (PEFT) methods that, instead of updating model weights, prepend a small number of trained “soft” tokens, continuous vectors in embedding space rather than discrete text, to every input. These prefix vectors are learned during fine-tuning on a target task and replace the need to update the much larger base model weights.

The appeal for serving infrastructure is obvious: you can have one base model and many task-specific adapters (one per customer, one per product line, one per domain), loading and swapping adapters without reloading the multi-billion-parameter base model. This is why LoRA adapter serving had already been implemented in vLLM. Prefix adapters are a different family with a different computational structure, and they needed their own implementation.

Why This Is Not Trivial

The key difference between LoRA and prefix adapters from an inference perspective is where the adaptation happens.

LoRA adapters modify weight matrices through low-rank additive updates. At serving time, LoRA can be “merged” into the base model’s weights for zero-overhead inference, or applied dynamically via modified matrix multiplications. The KV-cache behavior is unchanged.

Prefix adapters are different. They prepend vectors to the key-value sequences in every attention layer. This means:

1. The KV-cache can’t be straightforwardly shared. A prefix adapter effectively adds virtual tokens at the beginning of every sequence, changing the position of real tokens in the attention context. A KV-cache built without prefix awareness will be incorrect when a prefix adapter is active.

2. Multi-tenant serving requires per-request adapter awareness. In a system serving many users simultaneously with different adapters, the scheduler needs to know which adapter is active for each request to correctly apply the prefix embeddings and, optionally, pre-compute and cache the adapter’s KV contributions.

3. Memory management becomes non-trivial. Prefix adapter weights for many adapters must be held in GPU memory or loaded on demand. With thousands of adapters, this requires an eviction policy.

What the PR Did

The contribution created a prompt_adapter module alongside vLLM’s existing lora module, rather than shoehorning prefix adapters into the LoRA infrastructure. This was a deliberate architectural decision: the two adapter types share some machinery (request routing, adapter ID tracking) but have fundamentally different execution paths, and conflating them would have created a maintenance nightmare.

Key components:

• PromptAdapterConfig and PromptAdapterRequest: New data classes for specifying adapter configuration and per-request adapter selection, integrated into vLLM’s existing LLMEngine API.
• LRU cache for adapter weights: Prefix adapter embeddings are held in a fixed-size cache keyed by adapter ID. When the cache is full, least-recently-used adapters are evicted to make room. This allows serving a large number of adapters without holding all of them in GPU memory simultaneously.
• Attention layer modifications: Added prefix embedding injection into the attention computation for Bloom, Llama, and Mistral model families, the most common base models for prefix-tuned deployments at the time.
• adapter_commons: A shared utilities folder to reduce code duplication between the LoRA and prefix adapter implementations, covering request handling, worker dispatch, and model runner logic.

On Contributing to a Fast-Moving Project

The review process for vLLM is fast and substantive. The turnaround on comments was measured in hours, the feedback was specific and technical, and the bar for “production ready” was higher than “the tests pass.” Reviewers pushed back on the LRU cache implementation, asking for specific eviction behavior under concurrent load. They asked for explicit documentation of the KV-cache interaction. The scope of the test coverage was negotiated.

What I found useful: treat the review as a conversation, not a defense. The reviewers know the codebase better than you do. When they ask why you made a specific choice, it’s usually because there’s a constraint you didn’t know about, not because they’re wrong.

The PR is merged and live at github.com/vllm-project/vllm/pull/4645. Prefix adapter serving has been extended and improved in subsequent contributions. That’s exactly how open source should work.

The standard response to this disparity is behavioral: collect more data from underrepresented accents, augment training, maybe post-hoc calibrate the model. These interventions help at the margins. But they treat the symptom, the output disparity, rather than the cause, which lives in the model’s internal representations. ACES (Accent Subspaces for Coupling, Explanations, and Stress-Testing) is an attempt to work at the representation level.

Accent Lives in the Embedding Space

The core observation: in the hidden representations of a trained ASR model, there are directions, subspaces, that encode accent information. These subspaces aren’t randomly distributed across the representation. They’re concentrated in specific layers, and they correlate with where the model makes errors.

We developed a three-stage framework. First, we extract accent-discriminative subspaces from the model’s representations using linear probing: we train simple classifiers to predict accent from intermediate activations, then extract the decision boundaries as subspace directions. Second, we construct adversarial perturbations constrained to lie within those subspaces, stress-testing the model by perturbing inputs along exactly the dimensions that carry accent information. Third, we test whether removing the accent subspace from the representations improves fairness.

The results of the third stage were the most surprising.

The Entanglement Problem

The intuitive hypothesis was: accent-discriminative features are bias features. They’re not relevant to what’s being said, only to how it’s said. If we project them out of the representations, the model should become both more accurate on accented speech and more fair across accents.

That’s not what happened. When we removed the accent subspace from Wav2Vec2-base’s representations across seven accent groups, overall word error rate increased and the fairness gap didn’t close. On some accent groups it widened.

The reason is entanglement: the directions that encode accent aren’t cleanly separable from the directions that encode phonetically relevant variation. Accent isn’t just an additive bias on top of “accent-neutral” speech representation. Accent is part of how phonemes are realized. A model that’s learned to recognize English phonemes has, in the process, learned something about how those phonemes vary across the accent groups it has seen, and that learning is encoded in directions that a linear probe for accent will find.

This has a frustrating implication for fairness interventions: removing what looks like “accent information” from the representation removes information the model needs for recognition. The bias is structural, not separable.

The Attack as a Diagnostic

The second stage of ACES, adversarial attacks constrained to the accent subspace, turns out to be a better diagnostic than a direct threat. When we perturb inputs along accent-discriminative directions, the WER disparity gap between accent groups amplifies by nearly 50% (from 21.3 to 31.8 percentage points on Wav2Vec2-base). That’s a large effect from a structured, interpretable perturbation.

The amplification pattern is informative: it shows which accent groups have the most fragile representations and localizes the fragility to specific layers and directions. That’s a roadmap for targeted interventions, not projection-based removal, but something more surgical like representation regularization during fine-tuning that explicitly penalizes the concentration of accent information in fragile directions.

The Broader Point

The ASR fairness problem isn’t going to be solved by dataset scale alone. The disparity reflects how the model has structured its internal representation of speech, and changing that structure requires understanding it, not just training past it.

Accent isn’t a feature to be removed. It’s a property of phonetic realization that a fair ASR system needs to handle gracefully. The path forward is models that represent accent variation explicitly and robustly, not models that have been post-hoc “de-accented” in ways that destroy the very signal they need.

The paper is on arXiv.

But there’s a methodological problem that gets less attention: the circuits you find depend on how you look.

Change the attribution method, from activation patching to path patching to gradient-based attribution, and you get different circuits. Change the threshold for what counts as an “important” edge in the computational graph and the circuit expands or contracts. Change the set of examples you use to identify the circuit and the same behavior gets attributed to different components.

This isn’t just a technical nuance. It means that published circuits may be as much an artifact of analyst choices as a true reflection of how the model computes. CIRCUS (Circuit Consensus under Uncertainty via Stability Ensembles) is my attempt to put circuit discovery on firmer ground.

The Instability Problem

To illustrate: suppose you’re trying to identify the circuit responsible for gender pronoun agreement in a GPT-style model. You run activation patching on 50 examples, set an importance threshold, and find a circuit: three attention heads in layers 8, 12, and 15 that seem to carry the relevant information.

Now you run the same procedure on a different set of 50 examples. You find a slightly different circuit, maybe layers 8 and 12 appear again, but layer 15 is replaced by layer 14. Now you try path patching instead of activation patching. You find that layers 12 and 14 are consistent, but layer 8 disappears, replaced by an MLP in layer 3.

Which circuit is the real one? This isn’t a rhetorical question. The instability is real, and the inconsistency is a problem if you want to use circuits for anything consequential: robustness interventions, interpretability audits, model editing.

Consensus as the Answer

The core idea of CIRCUS is straightforward. Instead of running circuit discovery once under a single configuration and taking the result as ground truth, run it many times, varying the attribution method, the example set, the importance threshold, and look for what’s stable across configurations.

Every edge in the attribution graph receives a stability score: the fraction of configurations in which it appears. The consensus circuit is the set of edges with stability above some threshold (we use strict consensus, meaning above 50% in our main experiments).

The result is a circuit that represents genuine agreement across analytic frameworks rather than the output of one arbitrarily chosen procedure.

What We Found

On Gemma-2-2B and Llama-3.2-1B, strict consensus circuits are approximately 40x smaller than the union of all edges appearing in any configuration. That sounds like a lot of information being discarded, but what’s being discarded is mostly noise: edges that appear in some configurations for incidental reasons (a particular example set activates a particular head, a particular threshold happens to include a marginal edge).

Crucially, the explanatory power of the consensus circuit, measured by how well the circuit’s influence flows match the full model’s behavior, is comparable to the full union. The consensus isn’t just smaller; it’s more signal-dense.

The Uncertainty Perspective

There’s a deeper point here about what it means to “discover” a circuit. The instability of individual circuit-finding runs is actually useful information. It tells you where the model’s computation is ambiguous, where the same behavior can be plausibly attributed to multiple components, possibly because the model genuinely uses multiple redundant pathways.

The stability scores in CIRCUS are a map of that ambiguity. High-stability edges are the backbone of the computation. Low-stability edges mark the redundant, context-dependent, or method-sensitive parts. A fully faithful mechanistic account of the model should eventually explain both.

The paper is on arXiv.

This turns out not to be how it works. When we investigated adversarial attack transferability across Vision Transformers (ViTs) compressed via quantization, pruning, and weight multiplexing, we found that compression preserves or increases adversarial vulnerability, and that the reason why is actually informative about what robustness in transformers means in the first place.

Why ViTs and Why Compression

Vision Transformers have largely displaced CNNs for high-accuracy image recognition. They’re also substantially larger than their CNN counterparts, which creates pressure to compress them for deployment on edge devices: phones, embedded systems, inference accelerators.

Compression isn’t theoretical. Quantized and pruned ViTs are in production. The question of whether these compressed models maintain the security properties of the full model is therefore a practical one, not an academic stress test.

What We Found

We evaluated three compression strategies: post-training quantization (reducing weight precision from float32 to int8 or int4), magnitude pruning (removing weights below a threshold), and weight multiplexing (sharing weights across layers). For each strategy, we generated adversarial examples against both the original model and the compressed variants, then measured cross-model transferability.

The key finding: adversarial examples transfer readily between the original ViT and its compressed versions, often more readily than between architecturally distinct models. Compression doesn’t create the adversarial diversity that might limit transfer.

More specifically, the attention heads that carry the most adversarially relevant information are also the most parameter-intensive. They’re the heads that get removed first under magnitude pruning and the heads whose precision degrades most under aggressive quantization. The compressed model isn’t simpler in the ways that matter for robustness; it’s lost its expensive defenses while retaining the geometry of its decision boundaries.

The Attention Head Perspective

One framing of adversarial robustness in transformers is that robustness is a distributed property: it requires many heads attending to many different features, so that no single perturbation can systematically mislead all of them. High-capacity, high-precision heads that attend to global semantic structure provide robustness. Low-capacity heads that fire on local texture patterns don’t.

Compression, at least as currently practiced, tends to remove the former and keep the latter. The compressed model processes the same low-level texture features; it’s simply lost the high-level semantic attention that would allow it to “see through” a perturbation.

This isn’t a criticism of compression per se. It’s a prompt to ask what robustness-aware compression would look like: compression that explicitly preserves the heads with high adversarial relevance, even at a higher parameter cost.

For Practitioners

If you’re deploying a compressed ViT and relying on adversarial evaluations of the full model for security guarantees, those guarantees don’t transfer. The compressed variant should be evaluated independently, and the evaluation should include transfer attacks from the full model, which, in our experiments, consistently fool the compressed version.

The more useful heuristic: treat model compression as a change to the attack surface, not just to the inference cost. The threat model for the compressed variant is at least as broad as that for the original, and potentially broader.

The paper is available here (FICC 2023, Springer).

The more unsettling question is whether you can find a universal trigger, a short sequence of tokens that, when prepended to any input, drives the model toward a target class, without ever seeing a single training example. Just the model itself.

That’s what MINIMAL (Mining Models for Universal Adversarial Triggers) is about, and the answer turns out to be yes.

The Setup

Previous work on universal adversarial triggers (UATs), most notably the paper by Wallace et al., assumed access to a dataset. You need examples to evaluate the trigger against. You need to know what the model outputs on real inputs to know whether your trigger is working. It’s a reasonable assumption in a research setting, but it narrows the threat model considerably.

MINIMAL removes that assumption entirely. Given only model parameters and random token initialization, we mine triggers that generalize across arbitrary inputs.

The mechanism exploits something that should make anyone who trains language models a little uncomfortable: the gradient signal from the model’s own parameters, computed with respect to a target label, is rich enough to discover adversarial structure without any external data. The model has effectively memorized the shape of its own decision boundaries, and those boundaries are accessible without the data that carved them.

What Happens in Practice

On the Stanford Sentiment Treebank, a single 5-token trigger reduced positive-class accuracy from 93.6% to 9.6%. Prepend those five tokens to any review, a glowing five-star restaurant critique, a heartfelt book recommendation, anything, and the classifier confidently calls it negative.

On SNLI (natural language inference), a single trigger reduced entailment-class accuracy from 90.95% to below 0.6%.

These aren’t subtle improvements over data-dependent baselines. They’re roughly matching them, from zero data.

Why This Is the Scary Result

There’s a tempting way to downplay adversarial attacks on NLP models: require that the attacker have substantial access to training data. Data is an expensive, controlled resource. An attacker without the training set is, on this view, operating with at least some handicap.

MINIMAL eliminates that handicap. The attack surface is the model itself: the weights that are often distributed, downloaded, served via API, or extracted through repeated queries. In a world where fine-tuned models are routinely shared on model hubs, the trigger is derivable from the artifact millions of people already have.

The deeper issue is what this reveals about how language models encode their tasks. A model trained on sentiment should, in some idealized sense, have distributed its task knowledge across enough parameters that no small token sequence could uniformly hijack it. Instead, the geometry of the parameter space contains concentrated adversarial structure, choke points where a short, fixed input can override everything else the model has learned. That’s a property of the training dynamics, not just the specific model. We found it across multiple architectures and datasets.

What This Means Going Forward

The obvious defense is adversarial training against data-free triggers. We evaluated several filtering-based defenses and found they trade too much clean accuracy to be practical. The more promising direction is regularization during pre-training: penalizing the formation of the concentrated parameter geometry that makes data-free mining possible in the first place. That’s a harder problem, but it’s the right one to be working on.

There’s also a methodological point worth making: if you’re evaluating the robustness of a language model, running only data-dependent attack baselines will give you an overly optimistic picture. The attack surface extends past your dataset.

The code is available here and the paper is at AAAI 2022.

In most real world use cases, there exist several relational dependencies between our datasources. These relationships need to be joined and queried at search time; since these datasources are currently normalized or split logically into tables with no redundant information. Other key search service providers like Lucene and SOLR handle this by super-fast query joins using optimized Application side joins. As we will see later, we cannot use this method for our use cases due to certain restrictions. We need to find suitable alternatives for these join queries.

Methods:

Application Side Joins:

In this method the results matching from the first index are then queried against the second index and the resulting results are joined on the application side.

It has been used successfully in Lucene and SOLR with the following enhancements over normal filtering: Caching: The documents which are frequently used are cached, which gave a 20X improvement. Converting Strings to Numbers: String matching during the filter step is very time and resource intensive. Hence strings are converted to numbers, which are used for matching instead. This gave a 50X improvement. We cannot ensure that Azure is doing these, but it is the SOTA approach, so it must be the underlying process.

PROS:

• Data can remain normalized.
• Write performance improves, since only once source needs to be updated.

CONS:

• Need to run extra queries and perform expensive joins at search time.
• Azure’s restriction on the GET filter query is 8 KB, hence some searches may exceed that limit; if we have millions of rows in our datasources making our queries too long.
• Search relevance suffers, since the scores from the different indexes need to be combined arbitrarily.

When to use it?
Data Sources where data has a limited number of matching documents and preferably, is seldom updated (so that caching can help).

Data Denormalization:

This is a key technique which eliminates the need for joins. The tables are flattened along the required fields, which gives us the best performance, since we are using Azure Search how it is meant to be used. Using the Complex Datatypes (where fields with different datatypes can stored together as collections) available in Azure, we can handle both 1-1 and 1-N relationships in our data. The combined matching results can be displayed separately on the client-side using predetermined formats.

PROS:

• Speed, since no need for expensive client-side joins and extra queries.
• Better read performance.
• Better search relevance, since all the related information is stored together, and the inbuilt search relevance scores can be used.
• Can reduce the number of indexes, if full denormalization is performed.

CONS:

• Requires special index design
• Increased complexity in case of extremely nested JOIN relationships.
• Latency during write for data update of frequently updated datasources.
• Causes explosion of data for very highly 1-N relationships which may introduce latency.

When to use it?
Data Sources where data isn’t updated very frequently, and simple join relationships exist between them.

Azure SQL VIEWs:

The most natural solution to the problem, since we can replicate the current Query structure directly. A View is essentially a SQL query between 1 or more datasources. It can be connected to an Indexer and loaded into the index. The index can be updated by simply running the indexer once.

PROS:

• Most closely resembles the current SOTA approach.
• Easy update irrespective of the update cycles of the datasources.
• Can model highly complex, nested JOINs.
• Optimal search relevance.
• Can be easily altered as per requirement, without changing the indexer/index.

CONS:

• Only works with Azure Databases.
• Requires an additional Indexer and Index.

When to use it?
Complex Nested JOIN queries which cannot be modelled by other approaches.

Conclusion:

Unfortunately, there is no single correct solution to this problem. Our final solution will likely be an amalgamation of these approaches applied to relevant datasources.

References:

• LUCENE Approach
• Azure SQL
• Elastic Search Approach

Here is link, please read and give feedback!

https://drive.google.com/open?id=1r1L9N2e3hNgeVxfwBWb2860TXjdxizs7

Technology has become inextricably intertwined with people’s lives and businesses. By predicting everything from stock price to behaviours, learning algorithms coupled with voluminous data have permeated every industry, for revenue enhancement and/or cost-cutting. But the world is sparse, signals are like grains in heaps of chaff. To make sense of these sparse signals and derive insights, creating compressed representations of real-world data is imperative.

Solution according to me and its Impact:

I recently stumbled upon VAEs which can be used to solve the aforesaid problem. Variational AutoEncoders(VAEs) are an unsupervised deep learning algorithm which learns a dense representation of data by reconstruction. The data is passed through an encoder which creates a low dimensional latent vector; which is then passed to a decoder to reconstruct the original data. The latent vector is composed of the high-level features deemed important by the algorithm for reconstructing the original data and hence it can be trained to learn the compressed representation in an unsupervised manner, that is, without using any inference labels. These compressed representations of data can be highly useful in any problem where a sparse input needs to be processed for further downstream tasks. For example, a recent paper based on defending Convolutional Neural Networks (CNN’s) from adversarial attacks (adding Gaussian noise to the image to fool the algorithm); found state-of-the-art accuracy in using VAEs for prior denoising of the images before passing to CNN’s. This was accomplished by training the algorithm on artificially corrupted images, helping it learn the features essential for reconstructing the clean original image.

Case Study and Learnings:

Similarly, VAEs can also be used for anomaly detection. Anomaly detection is traditionally accomplished by supervised methods, that is, by detecting patterns in the data. Unsupervised anomaly detection using VAEs is therefore extremely useful, especially in time-series financial data. In one of my personal projects, by training a VAE on Bitcoin price data, I was able to generate a dense representation of ‘normal’ price data (using indicators such as moving and exponential averages). Then by measuring the reconstruction error of the test data points, it can be inferred that if the error is high; then it is probably an anomaly and vice versa.

Practicality:

This is just a start; I think the model can be made more adaptive, predictive and hence commercially viable. This algorithm, like traditional machine learning algorithms, can run on a rolling basis on new data; through a sliding window approach for continual improvement. Thus, I believe that VAEs can be trained to solve tasks such as unsupervised signal reconstruction and anomaly detection to help analysis and prediction.