Lead Data Engineer
Lead Data Engineer
Posted on :
11-01-2025
Employer Active
1 Vacancy
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Description
Roles and Responsibilities:
- Working with clients to understand their data.
- Based on the understanding you will be building the data structures and pipelines.
- You will be working on the application from end to end collaborating with UI and other development teams.
- You will be working with various cloud providers such as Azure & AWS.
- You will be engineering data using the Hadoop/Spark ecosystem.
- You will be responsible for designing building optimizing and supporting new and existing data pipelines.
- Orchestrating jobs using various tools such as Oozie Airflow etc.
- Developing programs for cleaning and processing data.
- You will be responsible for building the data pipelines to migrate and load the data into the HDFS either onprem or in the cloud.
- Developing data ingestion/process/integration pipelines effectively.
- Creating Hive data structures metadata and loading the data into data lakes/big data warehouse environments.
- Optimized (performance tuning) many data pipelines effectively to minimize cost.
- Code versioning control and the Git repository are up to date.
- You should be able to explain the data pipeline to internal and external stakeholders.
- You will be responsible for building and maintaining the CI/CD of the data pipelines.
- Bachelors degree in computer science or a related field.
- Minimum of 5 years of working experience with Spark and Hadoop ecosystems.
- Minimum of 4 years of working experience on designing data streaming pipelines
- Minimum experience of 3 years on NoSQL and Spark Streaming.
- Proven experience with big data ecosystem tools such as Sqoop Spark SQL API Hive Oozie Airflow etc.
- Solid experience in all phases of SDLC with 10 years of experience (plan design develop test release maintain and support)
- Handson experience using Azures data engineering stack.
- Should have implemented projects using programming languages such as Scala or Python.
- Working experience on SQL complex data merging techniques such as windowing functions etc.
- Handson experience with onprem distribution tools such as Cloudera/Hortonworks/MapR.
Employment Type
Full Time
Key Skills
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
