Data Engineer 2024TA1106
Data Engineer 2024TA1106
Posted on :
13-12-2024
Employer Active
1 Vacancy
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Description
This is a remote position.
Our Customer is looking for Data Engineers.
- Data Pipeline Maintenance: Continuously monitor and maintain data pipelines for ingesting and transforming data using Scala and SQL on Spark. Diagnose and resolve errors and performance bottlenecks addressing data discrepancies ambiguities and inconsistencies as needed.
- Technical Support and Version Control: Provide technical support for data analysis while managing source code and configuration artifacts via GitHub. Deploy code artifacts through GitHub Workflows/Actions.
- Technical Leadership: Offer handson technical guidance and leadership in developing Sparkbased data processing applications using Scala with a focus on Microsoft Azure Synapse Spark Runtime.
- Pipeline Optimization: Design and enhance data pipelines to streamline processing across various stages of the Medallion architecture using Azure Synapse Pipelines.
- Data Management: Oversee data ingestion processes enforce data quality checks using tools like DQ and manage validation and errorhandling workflows.
- Configuration Management: Develop and manage configuration settings using JSONbased configurations (e.g. ApplicationConfig TableConfig) for multiple data zones.
- Collaboration: Work closely with data scientists analysts and crossfunctional teams to ensure smooth integration of data engineering efforts with marketing and business strategies.
- Logging and Auditing: Implement and manage logging auditing and errorhandling practices to maintain data processing integrity leveraging tools like Azure Log Analytics and KQL queries where applicable.
- Testing and Quality Assurance: Conduct unit testing with tools like ScalaTest and maintain rigorous data quality checks to ensure dependable processing outcomes.
- 7 years of professional experience in Data Engineering
- Fluent English
- Strong proficiency in programming languages such as Python and Scala
- Extensive experience with Microsoft Azure
- Handson experience with data pipeline tools
- Expertise in relational and nonrelational databases including SQL
- Solid knowledge of cloud platforms and their data services.
- Familiarity with containerization and orchestration (e.g. Docker Kubernetes) is a plus.
- Strong problemsolving skills and the ability to work in a fastpaced environment.
- Excellent communication and collaboration abilities.
Employment Type
Full Time
Key Skills
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
