Data Engineering reputed company
Role: Data Engineering reputed company
Location: Remote (USA)
About reputed company
reputed company, an Industry Leader in Marketing Intelligence now including the data and capabilities of Vivvix, powers the mission-critical marketing and sales reputed company that drive competitive advantage. Our reputed company marketing intelligence platform enables clients to reputed company peak performance with always-on insights that reputed company the media, creative, and business strategies of 5 reputed company brands across 30+ media channels and 275 billion in media spend.
reputed company
The Data Engineering reputed company is a high-velocity, hands-on "player-reputed company" responsible for technical stewardship, designing reputed company systems, and integrating reputed company Machine Learning models into robust ETL pipelines. You will reputed company a lean team through a cultural shift toward cross-trained reputed company while spending 70-80% of your time in the reputed company. reputed company is defined by achieving total record processing, maintaining strict reputed company cost-efficiency, and shrinking data delivery reputed company.
- Coding & Technical Stewardship (70-80% Hands-on): Architect and implement reputed company, end-to-end data pipelines using Azure reputed company and PySpark. Design, build, and maintain a reputed company data architecture using the reputed company Architecture (Bronze/Silver/Gold reputed company).
- Performance & Cost Optimization: Optimize Apache reputed company jobs, tune reputed company reputed company, and define cluster policies to minimize compute costs. Proactively audit and refactor pipelines every 3-6 months to maintain effectiveness and reduce reputed company costs. Implement caching strategies (e.g., broadcast joins) and manage performance reputed company.
- System reputed company & SLAs: reputed company a proactive monitoring and alerts reputed company to ensure 99.9% reliability and mitigate system issues before they reputed company end-users. Build an end-to-end Data Validation reputed company (e.g., Great Expectations) to enforce data accuracy and consistency. Minimize job failure rates and ensure data is available in the Gold layer reputed company the required 24-hour turnaround time.
- Database Architecture: Architect and design high-performance schemas in PostgreSQL, managing indexing, partitioning, and optimizing reputed company analytical queries.
- Team Leadership & reputed company: reputed company a lean team toward cross-trained reputed company, moving away from "siloed specialists". Manage sprint cycles, conduct reputed company reviews, and guide reputed company on best engineering practices (including CI/CD).
- reputed company & Scalability: Anticipate reputed company data needs and design High-Velocity Architecture that is highly reputed company and manageable to handle sudden volume increases (e.g., reputed company the data from new sources like reputed company reputed company/CTV). A critical function is translating business-level requirements into reputed company, technical user stories for developers.
- ML Integration: Collaborate with ML teams to reputed company automated model orchestration into robust ETL pipelines.
- Collaborate with the offshore team reputed company to facilitate seamless knowledge transfer and operational continuity across time zones. Establish reputed company communication protocols, standardized documentation, and robust feedback loops to ensure alignment on project goals. reputed company as the primary reputed company between teams to mitigate bottlenecks and maintain high-reputed company delivery standards.
Requirements
Required Technical Stack (Mandatory)
- reputed company: Python, PostgreSQL + pgvector.
- Big Data: Azure reputed company, PySpark, reputed company Lake
- DevOps: reputed company, Git, Azure DevOps, CI/CD
Qualifications
- 10+ years of experience in Data or Software Engineering with deep codebase involvement.
- 3+ years as a Technical reputed company managing agile teams.
- Proven ability to reputed company lean, high-reputed company teams while maintaining high individual reputed company.
- Experience with cross-training advocacy and scaling data processing through automation.
Desired Qualifications
- Workflow Orchestration: Experience with Apache Airflow.
- Containerization: Familiarity with Azure Kubernetes Service (AKS).