Data Engineer

• Should be able to create and maintain optimal data pipeline architecture.
  
• Bee able to work on large datasets and to meet functional and business requirements.
  
• Design infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, Data Bricks, MS Azure Data Factory.
  
• Should be able to understand existing implementation and should enhance wherever needed.
  
• Create and maintain document and communicate standard methods and tools used.
  
• Team player and should work with other data engineers, data ingestion specialists, and experts across the team.
  
• Experience in performing root cause analysis on internal and external data and processes.
  
• Experienced using the following software/tools:
  
• Big data tools: Spark (Must), Hadoop (Not mandatory)
  
• Relational SQL and NoSQL databases, including COSMOS.
  
• Data pipeline and workflow management tools: DataBricks (Spark + Python), ADF, Dataflow
  
• Understanding and implementation knowledge on various Microsoft Azure Tools.
  
• Stream-processing systems: Streaming-Analytics, IoT Hub, Event Hub (Added advantage)
  
• Object-oriented/object function scripting languages: Python, SQL
  
• Knowledge and understanding of MS Azure DevOps.
  
• Should have experience in Agile methodology.
  

Should be able to create and maintain optimal data pipeline architecture. Bee able to work on large datasets and to meet functional and business requirements. Design infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, Data Bricks, MS Azure Data Factory. Should be able to understand existing implementation and should enhance wherever needed. Create and maintain document and communicate standard methods and tools used. Team player and should work with other data engineers, data ingestion specialists, and experts across the team. Experience in performing root cause analysis on internal and external data and processes. Experienced using the following software/tools: Big data tools: Spark (Must), Hadoop (Not mandatory) Relational SQL and NoSQL databases, including COSMOS. Data pipeline and workflow management tools: DataBricks (Spark + Python), ADF, Dataflow Understanding and implementation knowledge on various Microsoft Azure Tools. Stream-processing systems: Streaming-Analytics, IoT Hub, Event Hub (Added advantage) Object-oriented/object function scripting languages: Python, SQL Knowledge and understanding of MS Azure DevOps. Should have experience in Agile methodology.