DATA ENGINEER
DATA ENGINEER
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Overall Experience level:
7 years of recent GCP experience
Apache Hudi for big data processing and storage
7 years of handson experience Hadoop Hive or Spark Airflow or a workflow orchestration solution
Experience with programming languages: Python Java Scala etc.
Experience with scripting languages: Perl Shell etc.
Responsibilities:
Responsibilities
Design develop and maintain robust and scalable ETL workflows and data pipelines using tools like Hive Spark and Airflow.
Implement and manage data storage and processing solutions using Apache Hudi and BigQuery.
Develop and optimize data pipelines for structured and unstructured data in GCP environments leveraging GCS for data storage.
Write clean maintainable and efficient code in Scala and Python to process and transform data.
Ensure data quality integrity and consistency by implementing appropriate data validation and monitoring techniques.
Work with crossfunctional teams to understand business requirements and deliver data solutions that drive insights and decisionmaking.
Troubleshoot and resolve performance and scalability issues in data processing and pipelines.
Stay updated with the latest developments in big data technologies and tools and incorporate them into the workflow as appropriate.
Required Skills and Qualifications
Proven experience as a Data Engineer preferably in a big data environment.
Expertise in Hive Spark and Apache Hudi for big data processing and storage.
Handson experience with BigQuery and Google Cloud Platform (GCP) services such as GCS Dataflow and Pub/Sub.
Strong programming skills in Scala and Python with experience in building data pipelines and ETL processes.
Proficiency with workflow orchestration tools like Apache Airflow.
Solid understanding of data warehousing concepts data modelling and schema design.
Knowledge of distributed systems and parallel processing.
Strong problemsolving skills and ability to work with large datasets in a fastpaced environment.
Act creatively to develop applications and select appropriate technical options optimizing application development maintenance and performance by employing design patterns and reusing proven solutions account for others developmental activities
1. Interpret the application/feature/component design to develop the same in accordance with specifications.
2. Code debug test document and communicate product/component/feature development stages.
3. Validate results with user representatives; integrates and commissions the overall solution
4. Select appropriate technical options for development such as reusing improving or reconfiguration of existing components or creating own solutions
5. Optimises efficiency cost and quality.
6. Influence and improve customer satisfaction
7. Set FAST goals for self/teamTypical performance measures:
1. Adherence to engineering process and standards (coding standards)
2. Adherence to project schedule / timelines
3. Number of technical issues uncovered during the execution of the project
4. Number of defects in the code
5. Number of defects post delivery
6. Number of non compliance issues
7. On time completion of mandatory compliance trainingsPerformance Areas:
Code as per design
Follow coding standards templates and checklists
Review code for team and peers
Employment Type
Full Time
