Senior Software Engineer Spark over YT

The company

Nebius AI is an AIcentric public cloud platform specifically crafted to serve AI models for training and inference.

Our mission is to help ML practitioners concentrate on their core jobs while DevOps MLOps and infrastructurerelated tasks are handled by us. The idea is to build an MLspecific cloud platform covering the entire ML lifecycle from A to Z: from data preparation and labeling to ML training and inference.

We recognize the potential of ML and AI technologies and aim to provide our future users with the perfect environment to train and finetune their models. We are committed to delivering the best user experience and excellent customer support.

Four development hubs:
Nebius is headquartered in the Netherlands with hubs in Finland Serbia and Israel.

Data center in Europe:
Our own data center in Finland features server racks designed inhouse for MLspecific high load with powerefficient solutions including a freecooling system.

500 professionals:
Our mature team of engineers has a proven track record in developing sophisticated cloud and ML solutions and designing cuttingedge hardware.

The role

At Nebius were on a mission to harness the power of massive data and were looking for an innovative and passionate engineer experienced in Apache Spark internals to join our team. Our platform YTsaurus operates with exabytes of data and weve recently made this powerful tool opensource. Youll be at the forefront of integrating Apache Spark with YTsaurus creating an efficient data handling ecosystem.

The unique feature of Spark over YTsaurus or SPYT as we call it is its deep lowlevel integration between Apache Spark compute and YTsaurus storage. This integration allows for efficient processing by utilizing metadata. SPYT supports YTsaurus transactions and uses knowledge of table sorting to eliminate the shuffle phase during JOIN operations. Additionally SPYT employs YTsaurus as an execution environment for launching Spark clusters in a cloud manner enabling dozens of SPYT clusters to operate simultaneously within YTsaurus.

For more detailed overview of SPYT have a look at couple of talks by active and former members of YTsaurus SPYT special interest group:

• (2023) YTsaurus SPYT Webinar: (Apache Spark YTsaurus)
• (2023) YTsaurus SPYT: Helping Apache Spark scheduler be even more efficient
• (2021) How one connects Apache Spark to a proprietary data source: theory practice dirty hacks and duct tape

SPYT is actively used by both internal and commercial users of YTsaurus in Nebius and outside our company. Also we must define the place of Spark in our nextgen AI platform based on top of YTsaurus. To accomplish these tasks we need a passionate Spark expert ready for challenges and not afraid of taking responsibility.

Youre welcome to work in our offices in Amsterdam andBelgrade hybrid or remotely.

In this position your responsibility will be to:

• Enable and optimize modern features of Apache Spark within the SPYT framework;
• Develop new features for seamless integration between Spark and YTsaurus;
• Provide support to both internal and external users of SPYT and YTsaurus ensuring a smooth operation;
• Engage with the opensource community collaborating with big data experts to enhance YTsauruss capabilities;
• Advocate for SPYT as the key tool for AI applications including training and dataset preparation.

We expect you to have:

• 5 years of experience as a software engineer;
• Practical knowledge of Scala (preferably) or other JVMbased language (Java Kotlin);
• Familiarity with Apache Spark or Apache Hadoop user side;
• Knowledge of classical algorithms and data structures;
• Ability to use SSH experience deploying your code and debugging software using logs.

It would be an added bonus if you had:

• Experience in Apache Spark internals or even PRs to opensource upstream;
• Background in developing distributed DBMS or computation engines;
• Understanding of data streaming primitives and streaming computation engines: Apache Kafka Spark Streaming Apache Flink;
• Experience using DataBricks SnowFlake or Google DataProc;
• Knowledge of YTsaurus from the users side.

Does all that sound like your kind of challenge Then join us!