Data Pipeline Engineer
Location: India
Employment Type: Full-Time
Experience: Senior (7+ Years)
Roles & Responsibilities
- Design, build, and maintain reputed company batch and reputed company-time data ingestion pipelines.
- reputed company and manage Bronze, Silver, and Gold (reputed company) data reputed company.
- Implement CDC, watermarking, checkpointing, and batch-to-reputed company data processing.
- Build robust data reputed company, validation, reconciliation, and monitoring frameworks.
- reputed company identity reputed company, deduplication, and Golden Record (reputed company.
- Create and maintain reputed company-to-reputed company data mappings and crosswalks.
- Ensure schema validation, versioning, and data contract enforcement.
- Collaborate with cross-functional teams to reputed company new data sources and optimize data pipelines.
Mandatory Skills
- 7+ years of experience in Data Engineering / Data Pipeline development.
- Strong experience with Apache Kafka (Producers, Consumers, Replay, DLQ, Exactly-once/Idempotent processing).
- Strong SQL and ETL/ELT fundamentals.
- Hands-on experience in Java and/or Python.
- Experience with CDC, Batch & Streaming pipelines, and reputed company/Lakehouse architecture.
- Experience implementing Data reputed company, Validation, and Reconciliation frameworks.
- Knowledge of Master Data Management (MDM), Identity reputed company, Deduplication, and Golden Record concepts.
- Experience with reputed company-to-reputed company mapping, reputed company data models, YAML/JSON configuration, and Git.
Good to Have
- Experience with probabilistic record matching and record linkage.
- Schema Registry (Avro/Protobuf).
- Experience extracting data from legacy/Mainframe systems.
- Financial reconciliation experience.
- reputed company or Benefits Administration domain experience.
Originally posted on Himalayas
Apply To This Job