When a connection fails in a cloud environment, there are two completely separate layers where it might be getting dropped. The first layer is the one most engineers reach for first: the cloud network controls — NSGs, route tables, service endpoints. The second layer is the one that stays invisible until something breaks in a way you cannot explain: the OS-layer firewall running inside the VM itself.
The Network Ghost Agent I described in an earlier post operates at the cloud layer. It reads NSG rules, inspects route tables, runs packet captures, and reasons over the evidence. But iptables and nftables are outside Azure’s control plane. The Ghost Agent has no native way to see what is actually running in the VM’s kernel. This post is about the tool I built to close that gap: Netfilter Inspector.
The Visibility Gap
Azure’s Network Security Groups are a cloud resource. You can query them with the CLI, view them in the portal, diff them in Terraform. They are fully observable from outside the VM.
The OS firewall is not.
┌─────────────────────────────────────────────────────────────────┐
│ AZURE CONTROL PLANE │
│ │
│ NSG rule: allow TCP 80 inbound ✓ │
│ Route table: 0.0.0.0/0 → Internet ✓ │
│ Visible via: az CLI, portal, Terraform, Ghost Agent │
└───────────────────────────────┬─────────────────────────────────┘
│ packet permitted by NSG
▼
┌─────────────────────────────────────────────────────────────────┐
│ VM KERNEL — OS LAYER │
│ │
│ iptables filter/INPUT policy: DROP │
│ No rule permitting TCP 80 │
│ → packet silently dropped │
│ │
│ Invisible to: Azure control plane, NSG diagnostics, │
│ Network Watcher, Ghost Agent (without this tool) │
└─────────────────────────────────────────────────────────────────┘
iptables rules, nftables tables, chain policies — these live in the Linux kernel on the VM. They are the actual enforcement point for packet filtering. An NSG can permit traffic on port 80, and iptables can silently drop it. No Azure diagnostic will surface the discrepancy.
This situation is more common than it sounds:
- Docker installs its own iptables chains (
DOCKER,DOCKER-USER,DOCKER-ISOLATION-STAGE-1,DOCKER-ISOLATION-STAGE-2) whenever the daemon starts, and ensures they are present on every restart. These rules affect traffic routing to containers in ways that are entirely invisible from the cloud control plane. - fail2ban dynamically bans IPs by inserting REJECT or DROP rules into a chain such as
f2b-sshd, then jumping to it from INPUT. A legitimate connection source that was banned at the OS layer will look exactly like a connectivity failure to anyone inspecting only the NSG. - CIS hardening scripts change default chain policies from ACCEPT to DROP as part of a security baseline. A newly provisioned VM that appears unreachable might simply have had its INPUT policy tightened.
- ufw, firewalld, and similar management layers each leave structural iptables rules behind that persist even when the high-level tool is stopped or disabled.
When the Ghost Agent investigates a connectivity failure and the NSG looks clean, the next question is: what is the OS firewall doing? That is the question the Netfilter Inspector is designed to answer.
What the Inspector Does
The inspector has two operating modes:
Baseline capture — connect to the VM, run a read-only probe script, parse the firewall state into structured JSON, and write it to disk with a SHA-256 companion file that protects the snapshot against modification. One command, one artifact.
Drift detection — capture the current state and compare it against a previously saved baseline. Output a structured diff that identifies added rules, removed rules, policy changes, and repositioned chains. Flag critical changes separately from noise.
CHANGE WINDOW BRACKET PATTERN
Before change: After change:
┌──────────────────┐ ┌──────────────────┐
│ --is-baseline │ │ --compare- │
│ │ │ baseline <id> │
│ snapshot.json │ │ │
│ snapshot.json │──── diff ───►│ drift.json │
│ .sha256 │ │ │
└──────────────────┘ │ drift_detected │
│ has_critical_ │
│ changes │
│ rules_added: N │
│ rules_removed: N│
└──────────────────┘
The diff tells you exactly what changed at the OS firewall layer, expressed in structured JSON that both humans and the Ghost Agent can reason over directly.
Two Parsers With Independent Standalone Value
The implementation is split into three modules. The two parsers have standalone value independently of the rest of the system:
┌─────────────────────────────────────────────────────────────────────┐
│ netfilter-inspector/ │
│ │
│ ┌──────────────────────────────┐ ┌──────────────────────────────┐ │
│ │ iptables-parser/ │ │ nftables-parser/ │ │
│ │ │ │ │ │
│ │ iptables_parser.py │ │ nftables_parser.py │ │
│ │ parse_iptables_save() │ │ parse_nft_ruleset() │ │
│ │ │ │ │ │
│ │ iptables_diff.py │ │ nftables_diff.py │ │
│ │ diff_rulesets() │ │ diff_rulesets() │ │
│ │ │ │ │ │
│ │ Input: iptables-save text │ │ Input: nft --json output │ │
│ │ Standalone CLI ✓ │ │ Standalone CLI ✓ │ │
│ └──────────────────────────────┘ └──────────────────────────────┘ │
│ ▲ ▲ │
│ └──────────────┬───────────────┘ │
│ │ │
│ ┌───────────────────────────────▼─────────────────────────────────┐│
│ │ firewall-inspector/ (live-VM orchestration layer) ││
│ │ ││
│ │ detect framework → route to parser → baseline/diff → report ││
│ │ ││
│ │ AzureProvider (az vm run-command) SSHProvider (ssh + sudo) ││
│ └─────────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────────┘
iptables-parser takes iptables-save or ip6tables-save text as input — from a file, from stdin, from a backup script output — and produces structured JSON. You do not need a VM. You do not need the rest of this system. If you have iptables-save output from any source, the parser handles it: table headers, chain policy declarations with byte and packet counters (the [0:0] notation from iptables-save format), rule flags including negation (!), multiport, iprange, mac, comment, conntrack extensions, and more. Parse warnings surface anomalies without aborting. The diff engine (iptables_diff.py) computes rule-level diffs based on a stable identity derived from functional fields, not line position.
nftables-parser does the same for nftables. It takes nft --json list ruleset output — again, from any source. The nftables text DSL format is a recursive context-sensitive grammar not suited to a stdlib-only parser; the --json flag produces the same information in a well-specified machine-readable structure. The parser normalizes rule expressions into typed fields where possible and marks complex expressions as opaque rather than silently losing them. Rule identity uses nftables’ own handle integers as the primary key, with an expression hash as a secondary for detecting semantically equivalent rules that were deleted and re-added with a new handle.
firewall-inspector is the live-VM orchestration layer. It adds probe delivery, framework detection, the baseline/diff lifecycle, and the two provider implementations. Its value is integrating the parsers against live VMs — not replacing them. Both parsers remain useful independently: for parsing captured configs, validating backup files, auditing change management records, or diffing exported rulesets. The inspector auto-detects the active framework — iptables-legacy, iptables-nft, or nftables-native — from the VM’s own version strings and routes to the correct parser automatically.
The probe is a static bash script delivered via SSH or az vm run-command. It runs iptables-save and nft --json list ruleset only — no state is modified on the target VM.
Baseline Integrity
Every baseline is written with a SHA-256 companion file: {session_id}_snapshot.json and {session_id}_snapshot.json.sha256. The hash is computed over the exact bytes written to disk (2-space-indented JSON).
When --compare-baseline is used, the companion file is verified before the diff runs. A missing companion file raises IntegrityError — its absence is treated as tamper evidence, not a recoverable warning. A hash mismatch raises the same error with the first 16 characters of both the stored and computed hashes.
This is not security theater. When the Ghost Agent reads a baseline as part of an investigation, you want confidence that the baseline reflects the actual state that was captured, not a file that was edited in the interim. The integrity check is the mechanism for that confidence — and it gives any snapshot the properties of an audit record.
Two Access Patterns
The inspector supports two providers:
CASE 1 — DIRECT ACCESS (public IP) CASE 2 — VIA BASTION (private IP only)
Operator Operator
│ │
│ az run-command (probe delivery) │ az run-command (probe delivery)
│ SCP (output retrieval) │ SCP via ProxyCommand
│ │
▼ ▼
Target VM Bastion VM
(public IP) │ ProxyCommand tunnel
▼
Target VM
(private IP)
AzureProvider delivers the probe via az vm run-command invoke. This works even when the target VM has no public IP — Azure routes the command through its management plane, and output is returned inline. SCP (using the target’s IP and SSH key) retrieves the output file.
SSHProvider delivers the probe over SSH: ssh user@host "sudo bash -s -- SESSION_ID SSH_USER" < probe.sh. This works for any SSH-accessible Linux host — Multipass VMs, bare metal, VMs in other clouds. Both providers share the same SCP-based retrieval and cleanup path through a common base class.
For VMs accessible only via a bastion, both providers support two-hop access via SSH ProxyCommand. Set BASTION_PUBLIC_IP in the config file, and the tool tunnels through the bastion transparently.
Ghost Agent Integration
In the Ghost Agent’s investigation pipeline, the Netfilter Inspector surfaces as a detect_config_drift tool. The Ghost Agent runs the inspector as a subprocess through SafeExecShell. Probe commands (iptables-save, nft --json list ruleset) are classified SAFE — no human-in-the-loop gate is required. Remediation commands, if recommended, are MUTATIVE and require operator approval before execution.
GHOST AGENT EVIDENCE HIERARCHY
Cloud layer OS layer
───────────────────── ─────────────────────
az network nsg rule list firewall_inspector.py
az network route-table list --is-baseline
az network watcher ... --compare-baseline
│ │
│ │
▼ ▼
NSG rules, route tables, {session_id}_drift.json
service endpoints drift_detected
has_critical_changes
rules_added / removed
policy_changes
│ │
└────────────────┬────────────────────┘
│
▼
Ghost Agent RCA chain
(structured evidence, both layers)
The Ghost Agent does not parse the inspector’s stdout. It reads the structured {session_id}_drift.json artifact directly. The drift report contains drift_detected, has_critical_changes, per-family summaries with rule counts, and the full classified change list.
The design boundary is explicit: the inspector captures and diffs state; the Ghost Agent reasons over it. The inspector never calls the Ghost Agent; the Ghost Agent invokes the inspector as a tool and reads its output. Neither side needs to understand the other’s internals.
Why This Matters for Engineering Leaders
If you are leading an engineering team that operates cloud workloads, three aspects of this tool are worth understanding directly:
Zero dependencies, full auditability. The inspector runs on Python 3.9+ stdlib only — no pip install, no package management, no supply chain surface. A forensic tool that runs on production infrastructure and reads firewall state should be fully auditable. Every line of code is visible; no transitive dependency can introduce unexpected behaviour.
Read-only by design. The probe cannot modify firewall state because it does not contain any commands that can. This is not a runtime check or a permission guard — it is the absence of mutating commands from the probe script entirely. Safe to run during an active incident without risk of making the situation worse.
Integrity-protected baselines as audit records. SHA-256 companion files mean every baseline has the properties of an audit record. You can demonstrate that the firewall state at time T matched the record at time T — or that it did not. This is directly useful for change management, compliance, and post-incident review. The Ghost Agent’s investigation report cites the baseline session ID and the drift report, giving you a traceable evidence chain from the investigation back to a cryptographically verifiable state capture.
Framework-agnostic across mixed fleets. Production Azure fleets often contain a mix of iptables-legacy, iptables-nft, and nftables-native VMs — the result of different image vintages, distribution upgrades, and Docker installations that each manage the kernel stack differently. The inspector detects the active framework from the VM itself and routes to the correct parser. A single config file handles all three.
Using It
Baseline capture:
python3 firewall-inspector/firewall_inspector.py \
--config config.env \
--is-baseline
Drift detection against a prior baseline:
python3 firewall-inspector/firewall_inspector.py \
--config config.env \
--compare-baseline fw_20260315_090000
The parsers run standalone too — useful for parsing captured configs, auditing backup files, or diffing two exported rulesets:
# Parse any iptables-save file
python3 iptables-parser/iptables_parser.py /path/to/iptables-save.txt
# Diff two nftables snapshots
python3 nftables-parser/nftables_diff.py baseline.json current.json
The config file is a simple KEY=VALUE file. The session ID is auto-generated as fw_{YYYYMMDD}_{HHMMSS} if not set, making artifacts sortable by time and preventing accidental baseline overwrite.
What This Enables
With Netfilter Inspector in the pipeline, the Ghost Agent’s evidence hierarchy for a cloud connectivity failure covers both layers:
- Cloud layer: NSG rules, route tables, service endpoint configuration — queried via Azure CLI
- OS layer: iptables/nftables state — captured by the Netfilter Inspector and diffed against a known-good baseline
A change-window bracket — baseline before the change, diff after — gives you structured, integrity-protected before/after visibility at the OS firewall layer. Combined with the Ghost Agent’s cloud-layer inspection, OS-layer firewall state that would otherwise require manual SSH investigation on each VM can now be captured, diffed, and fed into an automated investigation chain.
Get Started
The Netfilter Inspector source is in the agentic-network-tools repository under netfilter-inspector/. The parsers and diff engines work standalone from the CLI — no Ghost Agent setup required. If you run Linux workloads on Azure and want to understand what your OS-layer firewalls are actually doing, start with the parsers against an existing iptables-save backup, then bring in the inspector for live capture.
If you run into framework detection edge cases, or have a VM configuration the probe does not handle cleanly, open an issue — the tool is actively developed and real-world ruleset variety is the primary driver of parser improvements.