Job Description: Data Engineer (Azure Databricks & PySpark)
Position: Data Engineer
Experience: 9 to 12 years
Primary Skills: Azure Databricks PySpark SQL (M)
Secondary Skills: ADF (Azure Data Factory) (M)
Project Exposure: Cloud migration (M)
Location: Bengaluru/Hyderabad
Mode of Work: Hybrid
Salary : 13 lac to 17 lac
Notice Period: 15 to 30 Days (M)
Databricks Engineer Job Description
Responsibilities:
- Work as part of a global distribution team to design and implement Hadoop big data solutions in alignment with business needs and project schedules.
- 5 years of data warehousing/engineering software solutions design and development experience.
- Code test and document new or modified data systems to create robust and scalable applications for data analytics.
- Work with other Big Data developers to make sure that all data solutions are consistent.
- Partner with the business community to understand requirements determine training needs and deliver user training sessions.
- Perform technology and product research to better define requirements resolve important issues and improve the overall capability of the analytics technology stack.
- Evaluate and provide feedback on future technologies and new releases/upgrades.
- Support Big Data and batch/realtime analytical solutions leveraging transformational technologies.
- Work on multiple projects as a technical team member or drive:
- User requirement analysis and elaboration
- Design and development of software applications
- Testing and build automation tools
- Research and incubate new technologies and frameworks.
- Experience with agile or other rapid application development methodologies and tools like Bitbucket Jira and Confluence.
- Have built solutions with public cloud providers such as AWS Azure or GCP.
Expertise Required:
- Handson experience in Databricks stack
- Data Engineering technologies (Ex: Spark Hadoop Kafka etc.)
- Proficiency in Streaming technologies
- Handson experience in Python and SQL
- Expertise in implementing Data Warehousing solutions
- Expertise in any ETL tool (e.g. SSIS Redwood)
- Good understanding of submitting jobs using Workflows API & CLI