Configuration
YAML-Based Configuration: ApiTap is configured via pipelines.yaml - a declarative approach that's easy to version control and maintain.
Global Settings
Define Defaults for All Sources
Set pagination strategies, concurrent limits, and more
defaults:
pagination:
kind: page_number
page_param: _page
per_page: 100
concurrency: 5
rate_limit:
requests_per_second: 10
Pagination Options
- •
page_number - •
limit_offset - •
cursor_based - •
link_header
Performance Tuning
- • Concurrency level
- • Rate limiting
- • Retry strategies
- • Timeout settings
Environment Variables
Secure Secrets Management
Use environment variables for sensitive data
Supports envsubst-style variable expansion:
targets:
- name: postgres_db
type: postgres
host: "${DB_HOST}"
database: "${DB_NAME}"
auth:
username: "${DB_USER}"
password: "${DB_PASSWORD}"
Best Practice: Never commit secrets to git! Use environment variables and a .env file instead.
Complete Example
Full Configuration Example
defaults:
pagination:
kind: page_number
page_param: page
per_page: 100
concurrency: 5
rate_limit:
requests_per_second: 10
sources:
github_repos:
url: https://api.github.com/orgs/${ORG_NAME}/repos
headers:
- key: Authorization
value: "Bearer ${GITHUB_TOKEN}"
- key: Accept
value: application/vnd.github.v3+json
targets:
- name: postgres_sink
type: postgres
host: localhost
port: 5432
database: mydb
auth:
username: "${PG_USER}"
password: "${PG_PASS}"
Configuration Sections
Sources
Define your API endpoints and connection details
- • URL and headers
- • Authentication
- • Pagination config
- • Rate limiting
Targets
Configure where data should be written
- • PostgreSQL
- • MySQL
- • Files (JSON, CSV)
- • Custom sinks