Pipeline run

655a31d7-f6e8-4465-a2e8-6c70793acda8

Pipeline LLM cost (USD)

API 1: $0.0037 API 2: $0.0002 API 3: $0.0000 Total: $0.0040

Client output enrichment

v2 Skill cluster · Nature of work · AI index · Tech stack maturity · Evidence · KRA description

role baseline loaded sources · ai_index: jd · nature_of_work: jd · tech_stack_maturity: jd

Nature of work · Data pipeline development

Build and maintain high-volume ETL pipelines and real-time/batch data services, improve performance and automate manual work, while evaluating new data technologies and mentoring teammates.

"“design, implement, and maintain data pipelines for extraction, transformation, and loading of data from a wide variety of data sources”"

Tech stack maturity

Mainstream Modern

A data engineer with machine-learning as a primary skill typically works in modern data and ML platforms, but the role alone does not imply cutting-edge AI-native or legacy-only stack characteristics.

AI index (0 = no AI use, 5 = totally AI-dependent · v2.1)

1.70 / 5

· Title match

✓ Has AI skill

✓ AI skill (primary)

· AI skill (secondary)

· On AI team

· Builds AI products

vocab breakdown (legacy)

Assistants (×1): —

Frameworks (×2): —

Models / concepts (×3): Machine Learning

Evidence — skills matched in JD (6)

Big Data Data Pipelines ETL Real-time Streaming Batch Processing Machine Learning

Skill cluster (2 dimension groups, role-scoped)

AI Governance and Model Security

Machine Learning

Cross-cutting / unaligned

Big Data Data Pipelines ETL Real-time Streaming Batch Processing

Show KRA description ↓

We’re looking for a Senior Data Engineer to join our team and help build the next generation of big data solutions at Index Exchange with real-time streaming and batch analytical capabilities. Data is a big deal at Index Exchange (Index). Index’s advertising exchange handles billions of auctions and generates terabytes of auction-related information every day. Our team builds tools and infrastructure to manage this vast amount of data and make it available to both internal and external customers and partners for their reporting and analytics as well as machine model training needs. • Evaluate new technologies, design, implement, and maintain data pipelines for extraction, transformation, and loading of data from a wide variety of data sources to various data services • Identify, design, and implement system performance improvements • Identify, design, and implement internal process improvements • Automate manual processes and optimize data delivery • Lead and mentor team members • Identify and assess potential solutions for technical and business suitability • Experience and Leadership: A senior engineer with exposure leading projects and mentoring junior developers. A leader who continuously challenges the status quo and brings forth innovative ideas and improvements for the team. • Problem Solvers: You don’t stop until the problem gets solved and you find more than one way to solve it. You love working with other people, presenting your viewpoint but ultimately working towards the best solution, regardless of where it comes from • Knowledge Hungry: Learning new frameworks and languages is exciting to you – you’re not satisfied with the status quo. We use a variety of languages and tools to solve problems and we're interested in what you're looking to learn. • Passionate: You have a passion for Big Data and an interest in the latest trends and developments constantly researching new tools and data technologies

Signals

Skill ml-engineer

0.17

Alias data-engineer

1.00

KRA data-engineer

0.55

Post-classification

Centroidupdated · n=495

Alias collision log—

New-role queue—

New skills captured5

New KRA captured—

Captured for admin review

Big Data primary ↔ Data Engineer pending

Data Pipelines primary ↔ Data Engineer pending

ETL primary ↔ Data Engineer pending

Real-time Streaming primary ↔ Data Engineer pending

Batch Processing primary ↔ Data Engineer pending

Status: completed Created: 2026-05-27T17:17:39.300391Z Updated: 2026-05-27T17:18:27.671372Z API 3 duration: 1453 ms

Flow Current 3-step pipeline

1 POST /skills/extract-from-jd

2 POST /skills/extract-details

3 POST /skills/final-role-output

Role Chosen role & resolution

Data Engineer

CASE A

slug: data-engineer · id: 2 · source: db

Exact alias hit on data-engineer (1.0) — no other alias at this confidence; skill_top ml-engineer 0.17 does not contradict

Resolution: in_db — role exists in library; skill↔dim and role↔dim links saved when applicable.

New skills

Skill↔dim saved

Role↔dim saved

Skipped

Job description

We shaped the earliest forms of ad tech, and we’re looking for the technical expertise to help shape its future. Our customers have unique problems that can only be solved at internet scale, and that’s where the technical skills of our team make a real difference.

Our exchange handles over 500 billion requests every day (for comparison Google serves an estimated 9 billion searches a day), all running in our own global data centers. Every member of our technology team has an enormous amount of autonomy in building and managing our systems to support and enable our growing level of scale. Through the transparency of our technology, dedication to innovation and integrity, and long-standing customer relationships, we lead through change.

What’s it like to work at Index?

We have more than 550 Indexers around the globe dedicated to building a safe and transparent marketplace that provides a trusted experience for consumers.

Index is an exciting and fast-paced place to work. We’re built on our values of change, support, learning and teaching, trust, and intention. We pride ourselves on our independence and openness, not only in our technology, but in our teams, too. Our diverse and inclusive culture celebrates how we can leverage our unique differences to help drive Index forward.

Our culture of success is truly supportive and collaborative. In working together across our teams, we’re continually investing in the people and technology to solve the industry’s most complex problems. As we extend the promise of ad tech to every channel, we’re looking for talented engineers to help advance Index, and the industry, forward.

Are you ready to join the programmatic evolution?

Index Exchange funds the open web. Content and journalism across the internet are funded through advertising, and we are the engine that helps to make that happen transparently, safely and efficiently. Handling hundreds of billions of auctions per day within milliseconds requires an intense understanding of the exchange and the ecosystem that we live in.

Our business is growing significantly every year and is poised to grow even faster. Our people and our platforms are the foundation and enabler of that growth. We are significantly expanding our technology teams, and are looking for technologists with a passion for high performance software development, and a drive to deliver software products and platforms that enable and empower industries at a global scale.

About The Role

Working with exciting technologies, your team will experiment with new tools and engineer innovative approaches to solve interesting challenges. Things shift very quickly in our industry, and we rely on our Engineering teams to keep Index and our clients ahead of the curve and moving in the right direction. We’re looking for Engineers who have experience in an Agile environment, who can drive innovation, and be a technical leader on our team.

Index’s scale spans the globe, our transactions happen 24x7 in our global data centers, and every second that passes millions of requests are evaluated across our exchange. In order to achieve our mission, global efficiency and reliability are absolutely key, as every millisecond quite literally counts in our business.

What We’re Looking For

• Experience and Leadership: A senior engineer with exposure leading projects and mentoring junior developers. A leader who continuously challenges the status quo and brings forth innovative ideas and improvements for the team.
• Problem Solvers: You don’t stop until the problem gets solved and you find more than one way to solve it. You love working with other people, presenting your viewpoint but ultimately working towards the best solution, regardless of where it comes from
• Knowledge Hungry: Learning new frameworks and languages is exciting to you – you’re not satisfied with the status quo. We use a variety of languages and tools to solve problems and we're interested in what you're looking to learn.
• Passionate: You have a passion for Big Data and an interest in the latest trends and developments constantly researching new tools and data technologies

Here’s What You’ll Be Doing

• Evaluate new technologies, design, implement, and maintain data pipelines for extraction, transformation, and loading of data from a wide variety of data sources to various data services
• Identify, design, and implement system performance improvements
• Identify, design, and implement internal process improvements
• Automate manual processes and optimize data delivery
• Lead and mentor team members
• Identify and assess potential solutions for technical and business suitability

Here's What You Need

• Bachelor/ Master’s Degree in Computer Science or Engineering related fields
• 8+ years of experience as a Software Engineer in enterprise grade, large scale distributed software product development
• 5+ years of work experience designing and building high performance data pipelines and applications using Hadoop/Ceph, Spark/Flink, Hive, Presto/Trino, Kafka, StarRocks/Vertica, Airflow, or other similar technologies
• Proficiency in some of the following languages: SQL, Scala, Java, Python, Bash
• Deep understanding of design principles of large scale distributed systems and familiar with mainstream big data related technologies and distributed frameworks
• Strong leadership, mentorship, and communication skills, with experience collaborating in a globally distributed, culturally diverse team
• Knowledge of data modeling, data warehousing, streaming data processing, and business intelligence reporting tools
• Experience working in Agile methodologies with continuous integration and delivery as CI/CD
• Experience working with containerization, and virtualization tools such as Kubernetes and Docker

Why You’ll Love Working Here

• Company paid comprehensive health and life insurance plans
• Paid Time off and flexible work schedules
• Company contribution to Provident Fund
• Participation in our company Stock options plan
• Company paid Parental Leave
• Monthly internet stipend
• Quarterly Wellness allowance
• Community engagement opportunities and donation-matching program
• Volunteer paid day off
• Annual virtual company retreats and regular community-led team events
• A workplace that supports a diverse, equitable, and inclusive environment – learn more here

Notification

Index Exchange is aware that there have been recent scams directed toward candidates regarding job interviews and offers.

Please be vigilant and do not accept interview requests, job offers, or other hiring-related documents from anyone other than our dedicated recruitment team, from the domain of @indexexchange.com. Our interview process consists of several steps, including phone screens and video interviews. We do not conduct interviews via an email questionnaire or request money at any point in the process.

We remain dedicated to resolving this matter and we appreciate your support.

Equal employment opportunity

At Index Exchange, we believe that successful products are built by teams just as diverse as the audience who uses them. As such, we are committed to equal employment opportunities. We celebrate diversity of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or expression, or veteran status. Additionally, we realize that diversity is deeper than any status or classification—diversity is the human experience. For those who show grit, passion, and humility—Index will welcome you.

Accessibility For Applicants With Disabilities

Index Exchange welcomes and encourages individuals with disabilities to apply to work with us.

If you require an accommodation, please share the details of your request and any information how we can assist you with the hiring recruiter when they contact you. Index Exchange will make reasonable efforts to ensure accommodation requests are met throughout the recruitment process.

Index Everywhere, Index Anywhere

Our corporate headquarters are in Toronto, with major offices in New York, Montreal, Kitchener, London, San Francisco, and many other global cities. As a major global advertising exchange, we are committed to operating as a tightly knit global team and embracing and empowering talent wherever our colleagues may be.

Skills from this JD

Each row merges API 1 extraction, API 2 library match / v3 orchestration (dimensions + locked dims), and API 3 persistence tags.

Big Data Primary New / orchestrated API 3: new canonical path (new) New / unmatched skill (orchestrated in API 2)

Skill enrichment (orchestrator / LLM)