The Pipe and Filter Pattern: How ApiShield Processes API Specs

ApiShield is built around a pipe-and-filter architecture. Understanding why that choice was made — and what it costs — is more useful than just describing what it does.

The Pattern

Pipe and filter structures a system as a sequence of processing stages (filters) connected by data channels (pipes). Each filter:

• Receives input from the pipe
• Transforms or analyzes it
• Passes output to the next pipe

Filters are independent. They don't know about each other. They only know the shape of the data they receive and the shape they emit.

Input → [Parser] → [Normalizer] → [Scanner] → [Reporter] → Output

This is the architecture of Unix pipelines (cat file | grep pattern | sort | uniq). It's old and boring and works extremely well for sequential data transformation.

ApiShield's Pipeline

Stage 1: Detection and Parsing

The entry point receives an input path (file, URL, or collection file) and detects its type. Each supported format has a dedicated parser:

• openapi.js — parses OpenAPI 3.x JSON/YAML specs
• postman.js — extracts endpoints and schemas from Postman Collections
• har.js — infers API structure from HTTP Archive (browser network logs)
• live.js — probes live URLs, performs dynamic schema inference from responses

The detection logic (detectInputType()) examines file extension, content-type, and document structure — it doesn't require the user to specify the format.

Stage 2: Normalization

Every parser emits a normalized internal representation based on OpenAPI 3.0. This is the critical design decision. The scanner doesn't know what format the input was — it only works with the normalized format.

interface NormalizedSpec {
  paths: Record<string, PathItem>;
  components: { schemas: Record<string, Schema> };
  info: { title: string; version: string };
}

Adding a new input format means writing a parser that outputs NormalizedSpec. The scanner doesn't change.

Stage 3: Scanning

The scanner iterates through the normalized spec. For each endpoint, it checks:

• Are there authentication requirements?
• Do response schemas expose sensitive fields (passwords, tokens, SSNs, PII patterns)?
• Are there rate limiting indicators in the spec?
• Do error responses follow a consistent schema (or reveal implementation details)?
• Are there unrestricted upload endpoints?

findSensitiveFields() runs regex and keyword matching against field names and descriptions in schemas. It's not perfect — a field named password is obviously sensitive; a field named p is ambiguous — but it covers the majority of common cases.

Stage 4: Reporting

The final stage maps findings to the STRIDE threat model and OWASP API Top 10. The same finding can map to multiple categories. Output format is configurable: JSON, Markdown, or console table.

generateThreatModel() in lib/reporters/threatModel.js assigns each finding a severity (Info, Low, Medium, High, Critical) and maps it to the appropriate category.

Why This Pattern

Testability. Each filter is a pure function (in the practical sense): input goes in, transformed output comes out. You can unit test each stage independently with a fixed input and assert the output, without running the full pipeline.

Extensibility. Adding HAR support didn't require touching the scanner or reporter — just writing a new parser that emits NormalizedSpec. Adding a new check to the scanner doesn't require touching any parser.

Parallelism potential. If the scanning stage is slow, you can fan out: split the normalized spec into chunks and run multiple scanner instances in parallel, then aggregate findings.

Where It Breaks Down

The pattern struggles when stages need to communicate in both directions. If the reporter needs context that isn't in the finding (e.g., the original raw input line number), that context has to be threaded through the pipeline explicitly — it's not free.

It also struggles with stateful analysis. If you want to correlate a finding in endpoint A with a schema definition in endpoint B ("this endpoint exposes a user ID that could be used for BOLA against that endpoint"), the simple pipeline needs to either pass accumulated context forward or do a second pass.

ApiShield handles the correlation problem by accumulating all findings in a shared context object that's passed through the pipeline — it's not a pure pipe anymore, but it's close enough to get the benefits of independent stages while supporting cross-reference analysis.