Principal Engineer Data Analytics Engineering
Principal Engineer Data Analytics Engineering
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
We are seeking a passionate candidate dedicated to building robust data pipelines and handling largescale data processing. The ideal candidate will thrive in a dynamic environment and demonstrate a commitment to optimizing and maintaining efficient data workflows. The ideal candidate will have handson experience with Python MariaDB SQL Linux Docker Airflow administration and CI/CD pipeline creation and maintenance. The application is built using Python Dash and the role will involve application deployment server administration and ensuring the smooth operation and upgrading of the application.
Key Responsibilities:
- Minimum of 9 years of experience in developing data pipelines using Spark.
- Ability to design develop and optimize Apache Spark applications for largescale data processing.
- Ability to implement efficient data transformation and manipulation logic using Spark RDDs and Data Frames.
- Manage server administration tasks including monitoring troubleshooting and optimizing performance. Administer and manage databases (MariaDB) to ensure data integrity and availability.
- Ability to design implement and maintain Apache Kafka pipelines for realtime data streaming and eventdriven architectures.
- Development and deep technical skill in Python PySpark Scala and SQL/Procedure.
- Working knowledge and understanding on Unix/Linux operating system like awk ssh crontab etc.
- Ability to write transact SQL develop and debug stored procedures and user defined functions in python.
- Working experience on Postgres and/or Redshift/Snowflake database is required.
- Exposure to CI/CD tools like bit bucket Jenkins ansible docker Kubernetes etc. is preferred.
- Ability to understand relational database systems and its concepts.
- Ability to handle large table/dataset of 2TB in a columnar database environment.
- Ability to integrate data pipelines with Splunk/Grafana for realtime monitoring analysis and Power BI visualization.
- Ability to create and schedule the Airflow Jobs.
Qualifications :
- Minimum of a bachelors degree in computer science or engineering. Masters degree preferred.
- AWS developer certification will be preferred.
- Any certification on SDLC (Software Development Life Cycle) methodology integrated source control system continuous development and continuous integration will be preferred.
Additional Information :
Because Western Digital thrives on the power of diversity and is committed to an inclusive environment where every individual can thrive through a sense of belonging respect and contribution we are committed to giving every qualified applicant and employee an equal opportunity. Western Digital does not discriminate against any applicant or employee based on their protected class status and complies with all federal and state laws against discrimination harassment and retaliation as well as the laws and regulations set forth in the Equal Employment Opportunity is the Law poster.
Remote Work :
No
Employment Type :
Fulltime
Employment Type
Full-time
