Member of Technical Staff (Index, Search Crawler)
Perplexity · London, Belgrade, Dublin · Posted Nov 17, 2026
Transparency
FreshOnline for 3 days and never reposted. Early applicants get the most attention, so now is the time to apply.
- First seen
- Nov 18, 2026
- Offered for
- 3 days
Overview
- Category
- Software engineering
- Employment type
- Permanent
- Workplace type
- Remote
- Country
- United Kingdom, Serbia, Ireland
- Seniority
- Staff
- Schedule
- Full time
- Salary
- Not listed
Key skills
- Large‑scale indexing systems
- Design, implement and operate indexing pipelines that ingest and organize massive crawled data.
- High‑throughput data processing
- Build and optimise pipelines that handle continuous, high‑volume data streams efficiently.
- Storage format design
- Develop compaction, update mechanisms and storage layouts to keep indexed data fresh and accessible.
- Observability and metrics
- Implement structured logs, metrics and data‑quality signals to monitor pipeline health and performance.
- Go / C++ / Rust programming
- Write production‑grade backend code in Go, C++ or Rust for indexing and data infrastructure.
- Distributed systems engineering
- Apply fault tolerance, consistency and scalability principles to large‑scale data services.
Key phrases
- large-scale indexing systems
- data pipelines
- offline and nearline pipelines
- high-throughput conditions
- resource efficiency
- storage formats
- compaction
- update mechanisms
- observability
- structured logs
- metrics
- data quality signals
- retrieval, ranking, and ML teams
- controlled rollouts
- production backend
From the posting
AI answers are only as trustworthy as the information behind them. Perplexity grounds its answers in fresh, real-world sources, and the index is what makes that possible. We're looking for an engineer to work on the indexing and data processing layer of Perplexity's Search Crawler. You'll build systems that ingest, process, and organize large volumes of crawled content into efficient, queryable structures, working primarily on offline and nearline pipelines that keep data fresh, complete, and accessible to downstream retrieval. Throughput, scalability, and correctness are critical across datasets that change continuously.
Responsibilities
Design, implement, and operate large-scale indexing systems and data pipelines at the core of Perplexity's search infrastructure.
Develop and optimize indexing strategies that balance performance, freshness, and resource efficiency.
Work on storage formats, compaction, and update mechanisms that keep data accessible and current.
Keep pipelines reliable and predictable under high-throughput conditions.
Build well-tested components with clear responsibilities and interface contracts that stay flexible as the system evolves.
Define and implement observability across offline and near line pipelines, including structured logs, metrics, and data quality signals.
Monitor throughput, resource usage, and cost, and drive optimisations as business needs change.
Partner with retrieval, ranking, and ML teams to make sure indexing outputs meet their requirements.
Enable safe experimentation on indexing strategies and processing logic through controlled rollouts and clearly defined quality signals.
Qualifications
5+ years of experience building production backend or data infrastructure systems.
Strong programming experience in Go, C++, or Rust.
Experience with high-throughput data processing systems and very large datasets.
Experience building or operating databases, storage systems, data planes, or indexing pipelines.
Strong understanding of distributed systems, fault tolerance, consistency, and scalability.
Experience running production systems and handling operational incidents.
A systems-thinking mindset and the ability to reason about end-to-end data flows.
