Data Engineer

At Minutes to Seconds we match people having great skills with tailorfitted jobs to achieve welldeserved success. We know how to match people to the right job roles to create that perfect fit. This changes the dynamics of business success and catalyzes the growth of individuals. Our aim is to provide both our candidates and clients with great opportunities and the ideal fit every time. We have partnered with the best people and the best businesses in Australia to achieve success on all fronts. We re passionate about doing an incredible job for our clients and job seekers. Our success is determined by the success of individuals at workplace

We would love the opportunity to work with YOU!!

Our client is looking for a Data Engineer to join a dynamic team in Sydney on a permanent basis.

Requirements

• Ensure reliable performance of the data warehouse through monitoring performance tuning and intelligent scheduling of workloads.
  
• Manage the existing data platform infrastructure and perform necessary upgrades
  
• Manage and configure data engineering workflows
  
• Ensure integrity and accuracy of data models by developing data quality checks reconciliation processes and thorough testing practices.
  
• Contribute to the ongoing support of releases in production.
  
• Monitor and ensure adequate logging and alerting are set up for data solutions.
  
• Maintain enterprise data warehouse/data lake and data engineering solutions on the cloud at an enterprise scale.
  
• Supports the development of agreed standards and tools
  
• Work with data consumers to design flexible data marts that are reliable easy to use and efficient.
  

• Minimum of 3 years experience in a similar role with a background in data and support
  
• Time management and organisational skills to manage competing deadlines and conflicting priorities.
  
• Understanding of architecture patterns and technical Implementation details relating to security compliance and governance for BI applications and Infra deployment on cloud
  
• Strong skills in Jira and Confluence.
  
• Problem solving
  
• Proficient with SQL with the ability to write very complex SQL queries and to query multiple data sources.
  
• Good understanding and experience with ETL data structures and data modelling (e.g. data warehousing and data marts) and how to best optimise them
  
• Design and document data models including the underlying principles their relationship to the information architecture and managing integrity and changes to those models.
  
• Experience in building/managing the cloud infrastructure required for a data platform in AWS.
  
• Proven expertise working with very large and complex datasets.
  
• Good understanding of Git and CI/CD pipeline technologies. Ideally buildkite terraform AWS CDK and Docker containers.
  
• Experience with schema design and dimensional data modelling.
  

Desirable/Bonus

• Experience in AWS Technologies like Redshift EMR EC2 CloudFormation Security groups VPCs DMS S3 EKS Athena Redshift Glue RDS EC2 Snowflake Kubernetes Docker Airflow and Lambda but not essential
  
• Experience using Apache Spark for data processing
  
• Experience with data streaming applications (e.g. kinesis Rabbitmq).
  
• Experience using Terraform for Infrastructure as code
  
• Experience maintaining Tableau Server or other Linux based applications.