Senior Data Engineer
Senior Data Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Title: Senior Data Engineer
Location: Remote
Job Description
We are looking for a highly skilled Senior Data Engineer with a strong background in big data engineering cloud computing and software development. The ideal candidate will have a proven track record of designing and implementing scalable data solutions using AWS Spark and Python. The candidate should have handson experience with Databricks optimizing Spark applications and building ETL pipelines. Experience with CI/CD unit testing and big data problemsolving is a plus.
Key Responsibilities:
- Design build and maintain largescale data pipelines using AWS EMR Spark and Python
- Develop and optimize Spark applications and ETL pipelines for performance and scalability
- Collaborate with product managers and analysts to design and implement data models and data warehousing solutions
- Work with crossfunctional teams to integrate data systems with other applications and services
- Ensure data quality integrity and security across all data systems
- Develop and maintain unit test cases for data pipelines and applications
- Implement CI/CD pipelines for automated testing and deployment
- Collaborate with the DevOps team to ensure seamless deployment of data applications
- Stay up to date with industry trends and emerging technologies in big data and cloud computing
Requirements:
- At least 8 years of experience in data engineering big data or a related field
- Proficiency in Spark including Spark Core Spark SQL and Spark Streaming
- Experience with AWS EMR including cluster management and job optimization
- Strong skills in Python including data structures algorithms and software design patterns
- Data bricks GraphDB/Vector DB.
- Handson experience with Databricks including Databricks Lakehouse (advantageous)
- Experience with optimizing Spark applications and ETL pipelines for performance and scalability
- Good understanding of data modeling data warehousing and data governance
- Experience with CI/CD tools such as Jenkins GitLab or CircleCI (advantageous)
- Strong understanding of software development principles including unit testing and testdriven development
- Ability to design and implement scalable data solutions that meet business requirements
- Strong problemsolving skills with the ability to debug complex data issues
- Excellent communication and collaboration skills with the ability to work with crossfunctional teams
Employment Type
Remote
Company Industry
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
