Spark

Request ID:

Work Location: Plano TX 75024 (ONSITE)

Job Details:

Must Have Skills:

Spark

Databricks

AWS Cloud

Proficiency of Python PySpark for data manipulation and visualization of large datasets.

Experience querying databases and reporting with SQL.

Experience in automating Data Engineering pipelines

Nice to have skills

Datadog monitoring & alerting for Data Reporting and infra pipelines.

Detailed Job Description:

Key Responsibilities

Collaborate with data scientists and analysts to understand data requirements for supply chain projects.

Design develop and maintain scalable data pipelines using Spark Databricks.

Work with Business Product Owners to refine the requirement and acceptance criteria.

Lead other junior data engineers within the team by mentoring reviewing the code and providing feedback.

Implement and optimize ETL processes to extract transform and load data from various sources into the data lake or warehouse.

Work closely with AWS services ensuring efficient data storage processing and retrieval.

Apply data quality and validation techniques to ensure accuracy and reliability of datasets.

Troubleshoot and resolve datarelated issues ensuring smooth project .

Collaborate with crossfunctional teams to integrate data engineering solutions into the overall project framework.

Required Qualifications

7 years of experience developing data pipelines handling large volume of data.

Proficiency of Spark for data manipulation and visualization of large datasets.

Experience querying databases and reporting with SQL.

Experience in automating Data Engineering pipelines Datadog monitoring & alerting for Data Reporting and infra pipelines.

Ability to clearly communicate technical information.

Preferred Qualifications

Experience with Databricks Airflow and AWS tools like Lambda Glue EC2 S3 Applied experience in supply chain logistics or related areas.

Top 3 responsibilities you would expect the Subcon to shoulder and execute:

Datadog monitoring alerting for Data Reporting and infra pipelines.

Ability to clearly communicate technical information.

Oniste and offshore coordination.

Regards