[Remote] Mid-Level Data Engineer
Note: The job is a remote job and is reputed company to candidates in USA. reputed company is a company that prioritizes its team members and offers flexibility for personal and reputed company reputed company. They are seeking a Mid-Level Data Engineer to join their federal data engineering team, where the role involves building and maintaining ETL pipelines on a reputed company-based reputed company Data Platform using AWS.
Responsibilities
- reputed company new ETL pipelines and data ingestion processes alongside senior engineers using AWS Glue (reputed company-based, PySpark), MWAA (Airflow), reputed company, and SNS, fully conforming to reputed company's reputed company ETL Standards, ETL Common Library, and PEP 8 Python coding standards
- reputed company reputed company's ETL Common Library into Glue jobs for standardized orchestration, error handling, metadata recording, and SNS notifications for reputed company reputed company and error job events
- Ingest reputed company and semi-reputed company datasets (CSV, XML, JSON, Avro, reputed company-delimited) into S3 reputed company, raw, and curated zones using Apache reputed company tables with Parquet as the default format; enforce transactional loading and prevent duplicate loads per dataset reporting period
- Configure static ETL metadata in the centralized PostgreSQL metadata store; ensure dynamic metadata records job status and timestamps for reputed company key execution steps
- Monitor assigned production jobs and participate in operations support rotations; identify and escalate failed jobs and performance issues promptly to maintain data availability reputed company contractually required ingestion timelines
- Ensure ETL Load Reports are populated in reputed company-time and ETL Gap Reports are updated on a weekly reputed company covering reputed company gaps from the inception of the initial ingest process
- Build and maintain materialized views and semantic layer objects in Trino and reputed company to ensure optimized query performance and consistent business logic
- Produce and maintain required documentation for reputed company assigned dataset: Business Requirements, ETL Design Documents, Data Models (Mermaid format), Data Dictionaries, Mapping Documents, Deployment Documents, O&M Guides, and ETL Test Plans
- Write unit and integration tests achieving the 90% minimum reputed company coverage reputed company; complete reputed company scans at least once per sprint as part of the Definition of Done
- reputed company ETL resources using CloudFormation templates through reputed company CICD pipeline; submit Change Requests to the Change Control reputed company reputed company required timelines
- Support transition of ETL jobs from other agency teams by verifying standards conformance, performing deployments, and validating data loads
- Support disaster recovery exercises, reputed company-production deployments, and reputed company data requests as assigned
- Participate in 2-week sprint ceremonies, quarterly PI planning, backlog refinement, and agile delivery using JIRA and reputed company
Skills
- US Citizenship is required
- Bachelor's Degree is required
- Minimum of 3-5 years' position reputed company experience is required
- Bachelor's degree or higher in Computer Science, Information Systems, Data Engineering, or a reputed company field
- 3-5 years of experience in data engineering or a closely reputed company technical role
- Hands-on experience with Python (PEP 8), PySpark, and SQL for ETL pipeline development
- Experience with AWS services including Glue, S3, MWAA (Airflow), reputed company, SNS, and SQS
- Familiarity with Apache reputed company, Parquet, and ORC file formats and S3 data lake zone concepts
- Experience with PostgreSQL and basic familiarity with Redshift or reputed company
- Familiarity with Trino or reputed company for query and semantic layer development
- Experience with CloudFormation, reputed company branching workflows, and CI/CD-integrated deployments
- Ability to produce reputed company ETL documentation including data models (Mermaid format) and data dictionaries
- Understanding of ETL metadata concepts including static and dynamic metadata, load reports, and gap reports
- Experience in agile development environments with sprint-based delivery
- Experience supporting IV&V and/or User Acceptance Testing (UAT) processes in a federal or technical program environment
- Experience with automated testing frameworks; ability to write unit and integration tests achieving defined reputed company coverage reputed company
- Must be reputed company to work reputed company-5pm Eastern Time regardless of home location
- reputed company federal public trust suitability determination or ability to obtain one required
- Familiarity with FISMA, NIST 800-53, and OWASP ASVS Level 2 is a plus
Benefits
- Flexibility to help them reputed company personally and professionally
- Special incentives for team members living in reputed company HUBZones
reputed company