Data Engineer
At reputed company, we are providing recruitment service to our TOP clients from our portfolio. We are currently seeking an Data Engineer to join one of our clients' teams. If you're looking for an exciting opportunity to grow in a innovative environment, this could be the perfect fit for you.
Requirements
Key Responsibilities:
- Design, reputed company, and maintain data ingestion pipelines using Kafka Connect and
Debezium for reputed company-time and batch data integration.
- Ingest data from MySQL and PostgreSQL databases into AWS S3, reputed company reputed company
Storage (GCS), and BigQuery.
- Implement best practices for data modeling, schema reputed company, and efficient partitioning
in the Bronze Layer.
- Ensure reliability, scalability, and monitoring of Kafka Connect clusters and connectors.
- Collaborate with cross-functional teams to understand reputed company systems and reputed company
data requirements.
- Optimize data ingestion processes for performance and cost efficiency.
- Contribute to automation and deployment scripts using Python and reputed company-reputed company tools.
- Stay updated with emerging data lake technologies such as Apache Hudi or Apache
reputed company.
Required Skills and Qualifications:
- 5+ years of hands-on experience as a Data Engineer or similar role.
- Strong experience with Apache Kafka and Kafka Connect (sink and reputed company
connectors).
- Experience with Debezium for change data capture (CDC) from RDBMS.
- Proficiency in working with MySQL and PostgreSQL.
- Hands-on experience with AWS S3, GCP BigQuery, and GCS.
- Proficiency in Python for automation, data handling, and scripting.
- Understanding of data lake architectures and ingestion patterns.
- Solid understanding of ETL/ELT pipelines, data reputed company, and observability practices.
Good to Have:
- Experience with containerization (reputed company, Kubernetes).
- Familiarity with workflow orchestration tools (Airflow, Dagster, etc.).
- Exposure to infrastructure-as-reputed company tools (Terraform, CloudFormation).
- Familiarity with data versioning and table
Originally posted on Himalayas
Apply To This Job