Python Pyspark and AWS TL Position
Python Pyspark and AWS TL Position
Posted on :
17-10-2024
Employer Active
1 Vacancy
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Description
Do you love a career where you Experience Grow & Contribute at the same time while earning at least 10% above the market If so we are excited to have bumped onto you.
Learn how we are redefining the meaning of work and be a part of the team raved by Clients Jobseekers and Employees.
If you are a Python Pyspark and AWS TL Position looking for excitement challenge and stability in your work then you would be glad to come across this page.
We are an IT Solutions Integrator/Consulting Firm helping our clients hire the right professional for an exciting long term project. Here are a few details.
Check if you are up for maximizing your earning/growth potential leveraging our Disruptive Talent Solution.
Role:Python Pyspark and AWS TL Position
Location: Hyderabad/Bangalore/Gurgaon/Mumbai/Pune
Hybrid Mode Position
Exp: 5 6 years
Requirements
We are seeking a Python Pyspark and AWS TL Position highly skilled with expertise in Python PySpark and AWS cloud services. The ideal candidate will have a strong background in Business Intelligence (BI) and data warehousing technologies with a focus on ETL processes and RDBMS. You will work closely with the application development teams to design build and optimize data pipelines and workflows throughout the development lifecycle. Additionally you will lead a small team of engineers to ensure the successful delivery of datadriven solutions.
Key Responsibilities:
Design and Develop Data Pipelines:
- Create and maintain robust scalable data pipelines using Python PySpark and Spark SQL.
- Build Airflow DAG pipelines for seamless data ingestion and workflow automation.
- Create and maintain robust scalable data pipelines using Python PySpark and Spark SQL.
Data Management and Processing:
- Work with large datasets processing them into usable formats using PySpark and DataFrames.
- Leverage your expertise in SQL for data extraction transformation and loading (ETL) ensuring optimal performance of queries and data models.
- Work with large datasets processing them into usable formats using PySpark and DataFrames.
Cloud Services and Architecture:
- Implement and maintain data solutions on AWS using services like S3 Glue Redshift Lambda IAM EMR Athena and Spectrum.
- Ensure data security scalability and costeffectiveness within the AWS environment.
- Implement and maintain data solutions on AWS using services like S3 Glue Redshift Lambda IAM EMR Athena and Spectrum.
Collaboration and Support:
- Work closely with application development teams to align data engineering solutions with business requirements and development lifecycles.
- Gather requirements from key business stakeholders and translate them into actionable data solutions.
- Mentor and lead a team of 4 or more engineers ensuring best practices in coding testing and deployment.
- Work closely with application development teams to align data engineering solutions with business requirements and development lifecycles.
Performance Optimization and Monitoring:
- Conduct performance tuning and optimization of SQL queries to ensure efficient data access and transformation processes.
- Monitor and troubleshoot issues within data pipelines ensuring data integrity and operational efficiency.
- Conduct performance tuning and optimization of SQL queries to ensure efficient data access and transformation processes.
Leadership and Ownership:
- Take full ownership of tasks and drive them to successful completion showing initiative in problemsolving and improving existing processes.
- Contribute to continuous improvement efforts by identifying and implementing automation opportunities and enhancing existing workflows.
- Take full ownership of tasks and drive them to successful completion showing initiative in problemsolving and improving existing processes.
Required Skills and Qualifications:
Technical Skills:
- Proficiency in Python PySpark Spark SQL and working with DataFrames.
- Strong understanding of ETL processes data warehousing concepts and relational databases (RDBMS).
- Handson experience with Airflow for managing and automating data pipelines.
- Extensive knowledge of AWS cloud services such as S3 Glue Redshift Lambda IAM EMR Athena and Spectrum.
- Solid experience in SQL query optimization and performance tuning.
- Proficiency in Python PySpark Spark SQL and working with DataFrames.
Experience:
- Minimum 56 years of experience in data engineering or related fields.
- Proven track record of leading teams of at least 4 members with handson involvement in project delivery.
- Experience working within agile development environments collaborating with crossfunctional teams.
- Minimum 56 years of experience in data engineering or related fields.
Soft Skills:
- Strong written and verbal communication skills capable of translating technical concepts for nontechnical stakeholders.
- Excellent problemsolving skills with a proactive attitude and ability to take initiative.
- Skilled at gathering and understanding business requirements and aligning them with technical solutions.
- Strong written and verbal communication skills capable of translating technical concepts for nontechnical stakeholders.
Benefits
CEO Message: Click Here
Clients Testimonial: Click Here
Employment Type
Full Time
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
