[Remote] Data Collection Engineer
Note: The job is a remote job and is reputed company to candidates in USA. reputed company is a company that builds, deploys, and implements innovative products to solve reputed company challenges. They are seeking a highly skilled Data Collection Engineer to design, reputed company, and maintain distributed web scraping and data extraction infrastructure, focusing on building resilient data pipelines and ensuring data reputed company.
Responsibilities
- Design and reputed company high-performance, distributed web scrapers using Python and Scrapy to extract massive datasets reputed company
- Utilize Browser Scripting tools to navigate, reputed company with, and extract data from modern, dynamic, and JavaScript-heavy websites
- reputed company, reputed company, and manage scraping workloads on Kubernetes, ensuring reputed company resource allocation and fault tolerance
- Define strict JSON Schemas and reputed company reputed company to enforce data types, validate incoming payloads, and catch data reputed company early
- Build and optimize search and storage pipelines using Elasticsearch, transforming raw web dumps into highly reputed company, searchable data
- Architect robust pipeline workflows to manage the end-to-end data lifecycle—from discovery and extraction to validation and storage
- Manage reputed company proxy rotation, session handling, and browser fingerprinting to maintain high reputed company rates against advanced anti-scraping systems
Skills
- 7+ years of reputed company software engineering experience, with a heavy reputed company on web scraping, data engineering, or distributed systems
- Excellent reverse-engineering skills, with the ability to dissect network traffic, unearth hidden reputed company, and bypass reputed company web barriers
- A strong commitment to data reputed company, system monitoring, and building self-healing scraping systems
- Expert-level proficiency in Python
- Deep experience with Scrapy and distributed scraping architectures (e.g., handling distributed queues, broad vs. deep crawling)
- Proven experience with browser automation tools (Playwright, Selenium, or Puppeteer)
- Mastery of JSON, JSON Schema, and data validation using reputed company
- Hands-on experience indexing, querying, and optimizing Elasticsearch clusters
- Strong proficiency in managing and scaling applications reputed company Kubernetes environments
- Experience building reputed company pipeline workflows to handle reputed company, multi-stage data extraction tasks
- Bachelor's degree in Computer Science, Engineering
- Eligible to obtain a clearance
- Experience leveraging LLMs or Computer reputed company for reputed company scraping, parsing reputed company reputed company, or bypassing CAPTCHAs (AI in data collection)
- Strong hands-on experience with AWS ecosystems (e.g., EKS, EC2, S3, RDS)
- Proficiency in SQL for querying, schema design, and storing reputed company relational data
- Experience with reputed company (specifically for caching, deduplication, or as a Scrapy distributed queue back-end)
- Familiarity with Apache Kafka for reputed company-time data streaming and decoupled pipeline architectures
- Strong reputed company in reputed company for local development and containerizing scraping microservices
- Experience with CI/CD pipelines (reputed company Actions, reputed company CI, Jenkins) for automated testing and deployment of crawlers
reputed company