Production Features

ApiTap includes built-in safeguards for production environments: Schema Validation, Audit Trails, and Dead Letter Queues (DLQ).

Goal: These features ensure data quality and lineage without requiring external tools.

Schema Validation

Enforce data structure at source

Audit Metadata

Automatic lineage tracking

Dead Letter Queue

Never lose failed records

1. Schema Validation

Enforce Data Structure

Fail or warn when API structure changes unexpectedly

How to Enable

Add schema_validation to your source config:

yaml

sources:
  critical_api:
    url: https://api.example.com
    schema_validation:
      fail_on_mismatch: true   # Stop pipeline on error
      required_fields:
        - "id"
        - "email"
      field_types:
        id: "Number"
        email: "String"
        active: "Boolean"

2. Audit Metadata

Automatic Lineage Tracking

ApiTap appends metadata columns to every inserted row

How to Enable

yaml

targets:
  - name: warehouse
    type: postgres
    audit:
      enabled: true
      prefix: "_meta"  # Optional, default is _apitap

Added Columns

Column	Type	Description
`_meta_run_id`	UUID	Unique ID for the pipeline run
`_meta_ingested_at`	Timestamp	UTC time of insertion
`_meta_source`	String	Source name (e.g., "critical_api")

3. Dead Letter Queue (DLQ)

Never Lose Data on Failures

Failed records are saved to a separate table for review

How to Enable

yaml

targets:
  - name: warehouse
    type: postgres
    dlq:
      enabled: true
      table_suffix: "_errors"  # Saves to {table_name}_errors

DLQ Table Schema

•
original_record: The raw JSON that failed
•
error_message: Why it failed
•
failed_at: Timestamp