GraphGuard IoT Source

GNN intrusion detectors catch IoT botnets, until the bots coordinate.

A reproducible benchmark of graph neural networks for IoT intrusion detection under adversarial attack. I trained GCN, GAT and GraphSAGE to find compromised devices, attacked them through the network's links and through the devices' own traffic, and measured what each defence actually recovers.

By Ziya Azizov, ML security engineer5 seeds, every number reproducible
Read the findings View code on GitHub

Every bot shifts its own traffic by ε standard deviations

0.00

Bars: attack-class F1 on held-out devices, mean of 5 seeds, white-box PGD. The network is an illustration; the share of evaded bots equals the selected model's measured recall loss.

What the benchmark shows

Four results hold across seeds. Differences smaller than about 0.05 F1 are within seed-to-seed noise.

Loading results…

Explore the results

Pick an attack and a defence. Lines show the mean over seeds, shaded bands one standard deviation. Hover or tap a budget to read exact values.

Attack
Metric

Robustness curve

Every defence trades clean accuracy

MLPGCNGATGraphSAGE

Clean F1 (x) against F1 under attack (y), one point per model and defence. Up and to the right is better.

Full results at the headline budgets

Attack-class F1, mean ± std over seeds. Structural attacks flip 10% of links; PGD uses ε = 0.5.

Run one experiment live

This calls the FastAPI service, which trains the model and runs the attack on a fresh 300-device graph. The first run of a configuration takes up to a minute while the model trains.

How it was measured

Every modelling choice is explicit, so results can be checked and argued with. All attacks are evasion attacks: models are trained on the clean graph and attacked at test time.

Setup

Task
Node classification: benign or compromised device
Graph
Synthetic IoT network, 1,500 devices
Features
16 traffic statistics per device, e.g. SYN ratio, unique destination ports, failed connections
Split
60 / 20 / 20, stratified; early stopping on validation F1
Metric
F1, recall and ROC-AUC for the attack class. Accuracy is misleading at 85 / 15 balance
Repeats
5 seeds, each a new graph and a new initialisation
Models
MLP (no graph, the control), GCN, GAT, GraphSAGE; 2 layers, 64 hidden units

Why a synthetic graph

Public IoT datasets such as TON_IoT and CIC-IoT-2023 are flow tables, not device graphs, and there is no standard way to build one from them. The generator states its assumptions instead: benign devices talk to their site gateway and peers; bots also coordinate with each other and probe other sites; each bot shifts a random subset of scan-related features. A logistic regression on features alone reaches F1 0.67, and neighbourhood features raise it to 0.78, so the graph helps without giving the answer away.

Attacks

  • Gradient (structure).Greedy adjacency-gradient attack in the FGA / Metattack family. The attacker knows its own bots but not the victim: it trains a 2-hop SGC surrogate and flips the links that most help its bots look benign.
  • DICE (structure).Delete links inside a class, connect across classes (Waniek et al., 2018). Cheap, label-aware baseline.
  • Random removal and injection (structure).Link loss or spoofed flows with no strategy behind them.
  • PGD evasion (features).White-box L∞ PGD, 40 steps (Madry et al., 2018). Only bots change, and only their own traffic statistics. A bot can shape its own traffic but cannot rewrite a neighbour's telemetry.

Defences

  • Cosine pruning.Drop links between dissimilar devices (GCN-Jaccard, Wu et al., 2019). Threshold tuned on validation.
  • DropEdge.Random link dropping during training (Rong et al., 2020). A regulariser, not adversarial training.
  • PGD adversarial training.Train on PGD-perturbed bots at ε = 0.5; keep the checkpoint with the best robust validation F1.

Limitations

  • The graph is synthetic. Absolute numbers do not describe any real network; the relative behaviour of models, attacks and defences is the result.
  • Evasion only. Poisoning the training graph is not covered.
  • The structural attack uses a linear surrogate and is not adaptive to the defence, so it is a lower bound on what a stronger attacker could do.
  • A single static graph per seed, with no temporal dynamics.

Reproduce every number

The page reads results/results.json, which one command regenerates on a laptop CPU in under ten minutes.

# Python 3.10+
git clone https://github.com/ziko-asm/ai-nids-adversarial-benchmark
cd ai-nids-adversarial-benchmark
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements-dev.txt

pytest -q                       # 16 tests
python -m src.evaluate          # full benchmark, 5 seeds
python -m src.explain           # GNNExplainer + fidelity
streamlit run dashboard.py      # interactive lab

Repository

  • src/dataset.pyDeterministic IoT graph generator
  • src/model.pyMLP, GCN, GAT, GraphSAGE
  • src/attack_smart.pyGradient structural attack
  • src/attack_feature.pyPGD evasion on bot features
  • src/attack.pyRandom and DICE attacks
  • src/defend.pyPruning, DropEdge, adversarial training
  • src/evaluate.pyBenchmark runner and plots
  • src/explain.pyGNNExplainer with Fidelity+ / −
  • src/api.pyFastAPI service for live runs
  • tests/pytest suite