Member of Technical Staff (Backend Software Engineer, Search Crawler)

About the Role

The Member of Technical Staff will be responsible for designing, developing, and operating systems that manage web-scale data to support advanced search technologies. This role requires expertise in both backend and frontend components, focusing on large-scale web crawlers and data processing systems.

Responsibilities

• Take ownership of and lead projects focused on developing large-scale web crawlers, ingestion pipelines, and data processing systems.
• Build, maintain, and optimize core backend and frontend components for crawler services, including storage, retrieval, and UI dashboards for data management.
• Collaborate closely with Search and Infrastructure teams to ensure the reliable, high-quality ingestion and processing of billions of web pages.
• Architect and implement fullstack features and scalable distributed systems that handle high-load and real-time data operations.
• Rapidly iterate, experiment with novel approaches, and continuously enhance system performance, usability, and reliability.

Qualifications

• Minimum of 5 years of software development experience, with strong knowledge of data structures and algorithms in at least one of the following languages: Python, C++, Rust, or Go.
• Experience with large-scale web crawlers is highly desirable.
• Proven experience building, deploying, and optimizing high-load, distributed, and hardware-adjacent services.
• Deep understanding of cloud infrastructure, with hands-on experience in Kubernetes (K8s) and AWS.
• Demonstrated passion for writing clean, efficient, and scalable systems.