Databricks Engineer

Databricks Engineer (8years exp)

Cincinnati Ohio

Responsibilities:

Data Pipeline Development: Design develop and maintain robust data pipelines using Databricks to process and transform large volumes of data.

ETL Process Management: Implement ETL (Extract Transform Load) processes to integrate data from various sources into Databricks ensuring data quality and integrity.

Data Integration: Integrate Databricks with other data storage solutions and data lakes ensuring seamless data flow and accessibility.

Performance Optimization: Optimize data processing and query performance within Databricks to ensure efficient data retrieval and processing.

Data Analysis and Visualization: Utilize Databricks to perform complex data analysis and create visualizations to support datadriven decisionmaking.

Collaborate with Data Scientists and Analysts: Work closely with data scientists and analysts to understand their requirements and provide the necessary infrastructure and tools within Databricks.

Security and Compliance: Ensure that data processing within Databricks complies with organizational security policies and industry regulations implementing necessary security measures. This includes setting up encryption managing network security configurations and performing regular security audits.

Monitoring and Troubleshooting: Monitor data pipelines and workflows for performance issues or errors and troubleshoot any problems that arise to maintain smooth operations.

Cluster Management: Manage the creation configuration and scaling of Databricks clusters to ensure optimal performance and costefficiency. This includes monitoring cluster usage resource allocation and ensuring high availability.

User and Access Management: Implement and manage user access controls ensuring that only authorized personnel have access to Databricks resources. This involves setting up rolebased access controls (RBAC) managing permissions and integrating with identity management systems.

Backup and Disaster Recovery: Develop and implement backup and disaster recovery plans for Databricks environments. Ensure that data and configurations are regularly backed up and that there are clear procedures in place for restoring services in the event of a failure.


Required Qualifications:

Technical Skills

• Experience with Databricks: Handson experience with Databricks including familiarity with its architecture features and services.
• Proficiency in Spark: Strong knowledge of Apache Spark including Spark SQL Spark Streaming and Spark MLlib as Databricks is built on Spark.
• Programming Languages: Proficiency in programming languages commonly used in data engineering such as Python Scala SQL and Java.
• Data Warehousing and ETL: Experience with data warehousing concepts ETL processes and tools like Apache Airflow Talend or Informatica.
• Database Management: Knowledge of relational and NoSQL databases data modeling and query optimization.
• Big Data Technologies: Familiarity with big data technologies and ecosystems including Hadoop Hive and Kafka.

• Data Analysis: Ability to perform complex data analysis and create data visualizations to support business decisions.
• ProblemSolving: Strong analytical and problemsolving skills to troubleshoot and resolve issues in data pipelines and workflows.

• Communication Skills: Excellent verbal and written communication skills to collaborate with data scientists analysts and other stakeholders.
• Team Collaboration: Ability to work effectively in a team environment and contribute to crossfunctional projects.

• Databricks Certifications: Certifications such as Databricks Certified Associate Developer for Apache Spark or Databricks Certified Professional Data Scientist can demonstrate expertise and enhance job prospects.
• Cloud Certifications: Certifications from cloud providers (e.g. Azure Certified Solutions Architect Azure Data Engineer) can be advantageous.