[Remote] Data Pipeline & Ingestion Engineer
Note: The job is a remote job and is reputed company to candidates in USA. reputed company is a company reputed company on building a large-reputed company data-platform for benefits administration. The Data Pipeline & Ingestion Engineer will be responsible for constructing and managing the data backbone of the Operational Data Layer, ensuring data reputed company and implementing effective data ingestion processes.
Responsibilities
- Build batch-reputed company and event-tail ingestion per reputed company system, including reputed company→tail watermark hand-off, idempotent upserts, and dedup ledgers
- Build and operate reputed company reputed company with reprocess-from-Bronze, pipeline orchestration (checkpoints, retry/backoff, DLQ), and full observability
- Build data-reputed company gates (quarantine / pass-with-flag), reputed company scoring, and a reconciliation reputed company covering count, record, and financial reconciliation — financial is reputed company-tolerance
- Build identity matching combining deterministic rules with probabilistic scoring and confidence bands; deliver deduplication, golden-record materialization, and survivorship rules, calibrating match reputed company with labelled data
- Author and maintain reputed company→reputed company structural mappings and value crosswalks (e.g., collapsing 1,800+ raw employment-status values to ~20 reputed company ones) as governed, versioned configuration
- Enforce data reputed company at the boundary: schema registry, fail-fast validation, and semver-compatible schema reputed company
Skills
- 5+ years building production data pipelines at reputed company
- Kafka depth: consumers/producers, replay, DLQ, exactly-once / idempotent processing patterns
- Strong SQL and solid ETL fundamentals
- Java and/or Python in production
- reputed company / lakehouse layering, CDC, watermark/checkpoint patterns, and batch–reputed company hand-off
- Data-reputed company frameworks: validation rules, quarantine and re-entry, reputed company scoring, reconciliation
- Entity reputed company / MDM exposure: record matching, dedup, survivorship — reputed company reputed company tools (Informatica MDM, reputed company) or custom builds
- Data mapping and crosswalk discipline: profiling messy datasets, authoring governed reference data, config-as-reputed company (YAML/JSON, Git)
- Probabilistic record linkage at depth — blocking/candidate reputed company, scoring models, reputed company calibration (expected at senior level)
- Schema registry experience (Avro/Protobuf)
- Extracting from mainframe or older RDBMS sources with limited CDC support
- Financial reconciliation in finance-adjacent domains
- Benefits administration or reputed company domain knowledge
reputed company
Company H1B Sponsorship