# Frankfurter Frankfurter is a free and open-source currency data API built with Ruby that tracks reference exchange rates from 50+ institutional sources (central banks, the IMF, the Federal Reserve, etc.). ## Architecture - Roda - SQLite with Sequel - Puma - Rufus scheduler - Foreman - Cloudflare CDN ## Project Structure ``` lib/ ├── app.rb # Main Roda app — mounts v1 and v2 ├── base_conversion.rb # Rebases rates from any base to a common base ├── blender.rb # Blends multi-provider rates: rebase → consensus → weighted average ├── bucket.rb # Shared SQL bucket expressions for weekly/monthly aggregation ├── cache.rb # Cloudflare cache purge ├── carry_forward.rb # Carries forward most recent provider rate within a lookback window ├── consensus.rb # Cross-provider outlier detection (MAD-based) ├── currency.rb # Currency model (materialized from rates) ├── currency_coverage.rb # CurrencyCoverage model (provider-currency join) ├── db.rb # Database configuration ├── currency_patches.rb # Patches Money::Currency: registers historical codes, fixes mangled names ├── log.rb # Shared logger ├── monthly_rate.rb # MonthlyRate model on monthly_rates rollup table ├── peg.rb # Currency peg definitions (from db/seeds/pegs/*.json) ├── provider.rb # Provider model: identity, backfill ├── provider/ │ ├── adapters/ │ │ ├── adapter.rb # Abstract adapter: fetch interface, chunked iteration │ │ └── .rb # One adapter per provider (auto-discovered) │ └── adapters.rb # Auto-requires all adapters ├── rate.rb # Rate model on rates table ├── rate_scopes.rb # Shared dataset scopes for rate tables (rates, weekly, monthly) ├── roundable.rb # Currency-aware decimal rounding ├── weekly_rate.rb # WeeklyRate model on weekly_rates rollup table ├── weighted_average.rb # Recency-weighted averaging with exponential decay ├── versions/ │ ├── v1.rb # Legacy API (ECB-only, frozen) │ ├── v1/ # V1 internals (quotes, query, rounding, currency names) │ ├── v2.rb # Multi-provider API │ └── v2/ │ └── rate_query.rb # V2 rate query builder (blending, filtering) ├── public/ │ ├── root.json # Root index document │ ├── v1/openapi.json # V1 OpenAPI spec │ └── v2/openapi.json # V2 OpenAPI spec └── tasks/ ├── consensus.rake # Consensus scan across providers ├── db.rake # Database migrations and setup ├── default.rake # Default task (lint + test) ├── providers.rake # Dynamic backfill task for all providers ├── rollups.rake # Rebuild weekly/monthly rollup tables ├── rubocop.rake # Linter task └── test.rake # Test suite task spec/ # Minitest test suite db/migrate/ # Sequel migrations db/seeds/ ├── pegs/ # One JSON file per peg (e.g. aed.json, bam.json) └── providers/ # One JSON file per provider (e.g. ecb.json, boi.json) ``` ## Key Components ### Adapters (lib/provider/adapters/) - `Provider::Adapters::Adapter`: Abstract base class — `fetch` interface, `fetch_each` for chunked iteration, sleep no-op in test env - Adapters are pure data extraction: they know how to talk to an external API and parse its response - No identity — adapters have no `key` or `name`. Provider model owns identity. - Optional class methods: `def backfill_range = N`, `def api_key = ENV[...] || raise(ApiKeyMissing)` - Auto-discovered from `lib/provider/adapters/` via loader ### Models - `Rate`: Sequel model on `rates` table. Scopes via `RateScopes`: `latest(date)`, `between(interval)`, `only(*quotes)`, `downsample(precision)` - `WeeklyRate`, `MonthlyRate`: Rollup models on `weekly_rates` / `monthly_rates`, share scopes via `RateScopes` - `Currency`: Sequel model on `currencies` table. Materialized from rates during backfill. Tracks global date ranges per currency. - `CurrencyCoverage`: Join model on `currency_coverages` table. One row per (provider, currency) with per-provider date ranges. Belongs to Provider and Currency. - `Provider`: Sequel model on `providers` table. Static config-as-data: seeded from `db/seeds/providers/*.json` on every container start so provider metadata always tracks the image. - `#adapter`: finds adapter by convention (`Provider::Adapters.const_get(key)`) - `#backfill`: incremental backfill — starts from `last_synced` or `coverage_start`, delegates to `adapter.fetch_each`, filters excluded quotes, stamps provider key, upserts to DB, refreshes currency summaries - `#start_date`, `#end_date`: derived from currency coverages - `many_to_many :currencies` through `currency_coverages` - `Peg`: Value object for currency pegs (from `db/seeds/pegs/*.json`) ### Blending Pipeline - `Blender`: orchestrates rebase → consensus → weighted average - `BaseConversion`: rebases rates from each provider's native base to a common base via inversion or cross rates - `Consensus`: MAD-based outlier detection — flags rates that deviate significantly from the cross-provider median - `WeightedAverage`: recency-weighted averaging with exponential decay past a grace period ### API (lib/app.rb) - V1 at `/v1/*` — frozen legacy, ECB-only - V2 at `/v2/*` — multi-provider with blended rates - CORS enabled for all origins - OpenAPI specs served as static files at `/v1/openapi.json` and `/v2/openapi.json` ### Scheduler (bin/schedule) - Runs as its own process, started by foreman alongside the web server (see `Procfile`) - Calls `provider.backfill` directly on Provider model instances - Staggers startup backfill for all providers (2s apart) - Cron schedule read from `publish_schedule` in the providers table (5-field cron; `null` for historical-only providers) - Convention: poll every 30 min across a 3-hour window starting at the publish hour (encoded directly in the cron expression, e.g. `*/30 14-16 * * 1-5` for ECB) - Backfill is incremental: fetches only from the last stored date forward ## Database SQLite database with `rates`, `weekly_rates`, `monthly_rates`, `providers`, `currencies`, and `currency_coverages` tables. ### rates - `date`, `base`, `quote`, `rate`, `provider` - Unique index on `(provider, date, base, quote)` ### weekly_rates, monthly_rates - Pre-aggregated rollups keyed by `bucket_date` (Monday for weekly, first-of-month for monthly) - Rebuilt by `rake rollups:rebuild` and refreshed during backfill ### providers - `key`, `name`, `rate_type`, `country_code`, `data_url`, `terms_url`, `publish_schedule`, `publish_cadence`, `coverage_start`, `pivot_currency` - Seeded from `db/seeds/providers/*.json` - `publish_schedule`: 5-field cron expression (minute hour day-of-month month day-of-week) in UTC, or `null` for historical-only providers. Convention: `*/30 H-H+2 * * D` where H is the publish hour and D is the day-of-week range, giving a 3-hour polling window. - `publish_cadence`: one of `daily`, `weekly`, `monthly`, or `null` for historical-only providers. Dispatches `publishes_missed` to the right algorithm (per-fire-day count for daily; ISO-week bucket for weekly; year-month bucket for monthly). - `coverage_start`: earliest date for historical data (used as backfill starting point) ### currencies - `iso_code` (PK), `start_date`, `end_date` - Global date range per currency, materialized during backfill ### currency_coverages - `provider_key`, `iso_code`, `start_date`, `end_date` - PK `(provider_key, iso_code)` - Per-provider date range per currency, materialized during backfill ## Testing ```bash APP_ENV=test bundle exec rake # Run linter and test suite APP_ENV=test bundle exec rake rubocop # Run linter only APP_ENV=test bundle exec rake spec # Run test suite only ``` Separate SQLite databases per environment (`APP_ENV`): test, development, production. ### Test stack - Minitest - Rack::Test for HTTP testing - VCR + WebMock for HTTP recording/mocking - Minitest-focus for targeted test runs - Global transaction rollback via `Minitest::Spec#around` - Test fixtures seed on suite load via `spec/helper.rb` ## Running Locally ```bash bundle install # Install dependencies bundle exec rake db:setup # Run migrations and seed providers bundle exec rake backfill # Backfill all providers (takes a while) bundle exec unicorn -c config/unicorn.rb # Start web server on port 8080 bundle exec foreman start # Start web + scheduler together (mirrors prod) ``` Or with Docker: ```bash docker run -d -p 80:8080 lineofflight/frankfurter ``` ### Legacy TLS BCN's endpoint only supports TLS 1.0, which OpenSSL 3.5+ disables by default. Set `OPENSSL_CONF=config/openssl_legacy.cnf` to enable it. Without this, BCN skips backfill with "legacy TLS required, skipping". ## Rake Tasks ```bash rake db:setup # Run migrations and seed providers rake db:migrate # Run database migrations rake db:seed # Seed provider metadata rake backfill # Backfill all providers (threaded, incremental) rake backfill[ecb] # Backfill a single provider rake rollups:rebuild # Rebuild weekly and monthly rollups rake rollups:rebuild[ecb] # Rebuild rollups for a single provider ``` ## Adding a New Provider See [.agents/skills/implementing-providers/SKILL.md](.agents/skills/implementing-providers/SKILL.md) for the full checklist and workflow. ## Currency Patches `db/seeds/currency_patches.json` patches `Money::Currency` at boot via `lib/currency_patches.rb`. Two purposes: - Register historical ISO 4217 codes (pre-euro, pre-redenomination) the gem doesn't include — full entry with `name`, `symbol`, `subunit_to_unit`, `iso_numeric`. - Override mangled names on existing entries (e.g. `Cfa` → `CFA`) — partial entry with just `iso_code` and `name`; existing fields are preserved via merge. When adding a new provider, check whether it serves historical currencies and note them in coverage research. To pick up previously-dropped records, re-backfill the provider from its `coverage_start`: ```ruby Provider["key"].backfill(after: Date.new(YYYY, 1, 1)) ``` ## Development Notes - Ruby (see `Gemfile`) - Linting: RuboCop with Shopify style guide (120-char line length) - Migrations in `db/migrate/` - Update `CHANGELOG.md` for changes that directly impact user experience ## Handling Data Relay what providers publish. Don't editorialize. ## API Endpoints ### V2 (lib/versions/v2.rb) Multi-provider API with blended rates. Full spec at `/v2/openapi.json`. ``` GET /v2/rates # latest blended rates GET /v2/rates?base=USD # rebased GET /v2/rates?quotes=USD,GBP # filtered GET /v2/rates?date=2024-01-15 # specific date GET /v2/rates?from=2024-01-01&to=2024-01-31 # date range GET /v2/rates?providers=ecb,tcmb # filter by providers GET /v2/currencies # currencies with names and providers GET /v2/providers # available data providers ``` Response: normalized array of `{ date, base, quote, rate }` records. ### V1 (lib/versions/v1.rb) Frozen legacy API, ECB-only. Full spec at `/v1/openapi.json`.