Senior Data Lake Developer
Senior Data Lake Developer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
Our client has started a greenfield project to build an inhouse data lake solution named Origin. This will serve various BI services and projects as well as other Data Science tools and initiatives within CIB. Origin will ingest near realtime market data from multiple sources and centrally store them on the cloud (AWS). The scope and type of the data consumed and target consumers will continue to expand.
Currently the existing functionality includes the following:
- Ingestion of multiple data sources in batch / near realtime in a shared storage.
- Decoupled storage from compute and data processing to provide increased scalability and performance while reducing storage costs.
- The data is compressed and partitioned using parquet data formats which improves performance while reducing the cost of data retrieval.
- Centralized metadata catalogue that maintains a single view of the data model for all its consumers.
- Integration with other AWS services (Athena EMR QuickSight) and thirdparty solutions (Apache Spark Presto Tableau).
The project is sponsored at high levels within the business and is a new initiative. Working in a small development team the focus will be on delivering highquality solutions for electronic trading and pricing workflows. The aim of the project is to build an agile team that works closely with the business sponsors to ensure a highquality platform that delivers on the needs of a global trading business.
Your Role
As a Senior developer you will be experienced in working with big data ingestion and large data sets in general. You will be responsible for creating and maintaining highquality ETLs. This is an exciting opportunity to build a new greenfield solution working as part of a small team to help the business improve their marketmaking capabilities within the fixed income space. The platform itself is housed within the AWS ecosystem.
Job Responsibilities:
- Take responsibility for the software delivery by ensuring quality and scope expectations are met.
- Contribute and take ownership of the technical design and ensure all aspects of the system architecture are well documented.
- Work closely with partner technology teams and collaborate effectively.
- Candidates must have the technical skills listed below and in addition have worked within a data team in the last 510 years.
Qualifications :
Education:
- Bachelor degree in Computer Science / Information Technology or a related field or substantial practical experience of software delivery at an advanced level
Technical Skills Required:
- Very deep understanding of Tableau
- Experience in SQL Hive Hadoop.
- Experience Python Pyspark Pandas JulyterLab (working with notebooks).
- Experience using AWS platform.
- Experience with continuous integration and continuous delivery tools like Git Jenkins etc.
- Agile development/Software life cycle.
- Excellent interpersonal and communication skills in English (b2)
Nice to have Skills:
- Experience with Kafka
- Specifically experience using EMR (Elastic Map Reduce) in AWS to run Spark clusters.
- Knowledge of Terraform
- Experience with Ansible Bash scripting boto3
- Experience configuring continuous integration and continuous delivery tools.
Soft Skills:
- Energetic motivated and determined
- Pragmatic and resultsoriented
- Adaptable to diverse set of technical responsibilities
- Excellent analytical and problemsolving skills.
- Productive and able to manage time effectively.
- Strong written and verbal communication skills
Additional Information :
What do we offer you
- Fulltime contract
- Training and career development opportunities.
- Be part of a multicultural team working on international projects.
If you have read this far and you are looking forward to joining this challenge do not hesitate to apply... we would be delighted to meet you!
Remote Work :
Yes
Employment Type :
Fulltime
Employment Type
Remote
