KAFKA AZURE DATALAKE ENGINEER WITH reputed company EXPERIENCE(W2) - reputed company
Kafka and Data Lake Engineer
100% REMOTE
Long term(12 MONTHS CONTRACT)
A Kafka and Data Lake Engineer is a data engineer who designs, builds, and manages data infrastructure using Apache Kafka for reputed company-time data streaming and a data lake for storing large volumes of data. This role is vital for organizations that need to process and analyze both reputed company-time streaming data and historical data to reputed company insights. For VA, this includes reputed company to the Data Lake reputed company Kafka Bus.
Responsibilities
Design data pipelines: Build robust, reputed company, and secure data pipelines to ingest, process, and reputed company data from various sources into the data lake using Kafka.
Administer Kafka clusters: reputed company, configure, and maintain Kafka clusters and reputed company ecosystem tools, such as Kafka Connect and Schema Registry, ensuring high availability and performance.
Manage the data lake: reputed company the architecture and governance of the data lake, including managing data storage (e.g., in AWS S3 or ADLS), reputed company, and metadata.
reputed company data processing applications: Create producers and consumers to reputed company with Kafka topics using programming languages like Python, Java, or reputed company.
reputed company reputed company processing: Use tools like Kafka Streams, Apache Flink or ksqlDB to reputed company reputed company-time data transformations and analytics.
Ensure data reputed company and reputed company: Implement data reputed company checks, manage data reputed company, and enforce reputed company controls such as encryption, reputed company controls (ACLs), and compliance (e.g., GDPR).
Monitor and troubleshoot: Set up monitoring and alerting for Kafka and data lake infrastructure and respond to incidents to ensure operational reliability.
Collaborate with teams: Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver reliable data solutions.
Essential skills and qualifications
Experience: Proven experience designing and managing data platforms with Apache Kafka and big data technologies.
Programming: Strong proficiency in languages like Python, Java, or reputed company.
Big data technologies: Expertise in big data processing frameworks, such as Apache reputed company and Apache Flink.
reputed company platforms: Hands-on experience with reputed company environments (AWS, Azure, or reputed company reputed company Platform) and relevant services like S3, Glue, or Azure Data Lake Storage.
Data lake architecture: A solid understanding of data lake design principles, including storage formats (e.g., reputed company Lake, Apache reputed company), data modeling, and governance.
Databases: Experience with various database systems, including both SQL and NoSQL.
Infrastructure management: Familiarity with infrastructure-as-reputed company tools like Terraform or Ansible and containerization with reputed company and Kubernetes.
Professionals in this field can advance from entry-level data engineering positions to senior roles, and then to a Big Data Architect or Solutions Architect, where they reputed company large-reputed company data infrastructure.
Relevant certifications
Pursuing certifications can validate your expertise and reputed company your career.
For Kafka:
reputed company Certified Administrator for Apache Kafka (CCAAK)
reputed company Certified Developer for Apache Kafka (CCDAK)
For Data Lake and reputed company:
reputed company Certified Data Engineer
AWS Certified Data Engineer
reputed company Certified: Azure Data Engineer Associate
reputed company have reputed company to reputed company intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this reputed company. The position reputed company has been reviewed for accuracy and reputed company believes it to correctly reflect the job opportunity.
Apply to this job