Senior Back-End Engineer - Scanning + Headless Browser

Position Overview

As part of our browser scanning team, we are seeking a senior Back-End Software Engineer focused on building and scaling high-performance web scanning and data ingestion systems. You will design, implement, and operate robust distributed services that crawl, scrape, process, and store large volumes of web data while maintaining reliability, compliance, and efficiency at scale.

Key Responsibilities

• Design, implement, and maintain scalable microservices and distributed systems for web crawling, scraping, and data ingestion using Go or Rust or Python.
• Build and operate headless browser automation and scraping pipelines using Puppeteer, Playwright, Selenium, or lightweight HTTP clients with robust HTML parsing strategies.
• Develop and optimize REST and GraphQL APIs to serve downstream consumers.
• Implement concurrency and parallelism patterns to maximize throughput while controlling resource usage.
• Architect and tune message-driven pipelines with brokers like Kafka, RabbitMQ, and task scheduling for ETL workflows.
• Design storage and indexing solutions using relational databases, NoSQL stores, and search systems.
• Build robust proxy management and anti-bot evasion techniques as needed.
• Containerize services with Docker and deploy them on Kubernetes or serverless platforms.
• Instrument systems for observability and reliability using logging and monitoring tools.
• Collaborate with product, data, and security teams to define SLAs and own CI/CD pipelines.
• Mentor engineers, perform code reviews, and contribute to architectural decisions.

Qualifications

Bachelors or Masters degree in Computer Science, Engineering, or equivalent practical experience. 5+ years building infrastructure or web scanning tools.