Senior Lead Data Engineer Spark SQL
Senior Lead Data Engineer Spark SQL
Posted on :
20-12-2024
Employer Active
1 Vacancy
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Description
Company Size
MidSized
Experience Required
5 9 years
Working Days
5 days/week
Office Location
HSR Layout Bengaluru
Map
Role & Responsibilities
Lifesight is growing rapidly and seeking a strong Data Engineer to be a key member of the Data and Business Intelligence organization with a focus on deep data engineering projects. You will be joining as one of the few initial data engineers as part of the data platform team in our Bengaluru office. You will have an opportunity to help define our technical strategy and data engineering team culture in India.
You will design and build data platforms and services while managing our data infrastructure in cloud environments that fuel strategic business decisions across Lifesight products.
A successful candidate will be a selfstarter who drives excellence is ready to jump into a variety of big data technologies & frameworks and is able to coordinate and collaborate with other engineers as well as mentor other engineers in the team.
What You ll Be Doing
- Build highly scalable available faulttolerant distributed data processing systems (batch and streaming systems) processing over 100s of terabytes of data ingested every day and petabytesized data warehouse and elasticsearch cluster.
- Build quality data solutions and refine existing diverse datasets to simplified models encouraging selfservice.
- Build data pipelines that optimize on data quality and are resilient to poorquality data sources.
- Own the data mapping business logic transformations and data quality.
- Lowlevel systems debugging performance measurement & optimization on large production clusters.
- Participate in architecture discussions influence product roadmap and take ownership and responsibility over new projects.
- Maintain and support existing platforms and evolve to newer technology stacks and architectures.
Ideal Candidate
- Proficiency in Python and PySpark.
- Deep understanding of Apache Spark Spark tuning creating RDDs and building data frames.
- Experience in big data technologies like HDFS YARN MapReduce Hive Kafka Spark Airflow Presto etc.
- Experience in building distributed environments using any of Kafka Spark Hive Hadoop etc.
- Good understanding of the architecture and functioning of distributed database systems.
- Experience working with various file formats like Parquet Avro etc. for large volumes of data.
- Experience with one or more NoSQL databases.
- Experience with AWS GCP.
- 5 years of professional experience as a data or software engineer.
data engineering,pyspark,aws,spark,data quality,gcp,kafka,apache,airflow,apache spark,presto,hdfs,nosql databases,yarn,engineers,python,map-reduce,data,hive
Employment Type
Full Time
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
