Data Engineer
Job Summary
Develop API-driven systems to govern manage and monitor large-scale batch Big Data applications.
Build scalable backend services and data engineering solutions that support data processing and operational workflows.
Develop and maintain data transformation processes using Spark SQL Hive Python Scala and related technologies.
Work with AWS cloud services such as S3 EC2 EMR Lambda DynamoDB and API Gateway to build cloud-native data and backend solutions.
Build and enhance workflow orchestration using Apache Airflow including advanced DAG design dependency management scheduling monitoring and failure handling.
Define technical scope objectives and implementation approaches by participating in requirements gathering technical research and process definition.
Participate in architecture reviews code reviews performance tuning and operational readiness activities.
Contribute to test planning and validation for application integrations functional areas and project deliverables.
Partner with product owners data engineers backend engineers QA DevOps and other cross-functional teams to deliver high-quality solutions in an Agile/Scrum environment.
Troubleshoot production issues improve system reliability and drive automation across data and backend workflows.
Responsibilities
5 years of hands-on experience developing enterprise-scale applications data platforms or distributed systems.
5 years of experience developing and operating Big Data platforms and cloud-based infrastructure services preferably using AWS EMR and the Hadoop ecosystem.
Hands-on experience with AWS Spark Python and/or Scala Airflow SQL Hive and related data processing technologies.
Proficiency in either Python or Scala with the ability to build production-grade data pipelines and backend services.
Experience with AWS cloud services including S3 EC2 EMR Lambda DynamoDB and API Gateway.
Strong experience with Apache Airflow orchestration including complex workflows scheduling dependency handling monitoring and operational support.
Strong understanding of data engineering concepts including data pipelines data transformation batch processing real-time processing data quality and performance optimization.
Excellent analytical and problem-solving skills.
Strong communication and presentation skills with the ability to explain technical concepts clearly to both technical and non-technical audiences.
Experience with version control and CI/CD tools including Git and Jenkins for source code management build automation testing and deployment workflows.
Ability to work effectively in cross-functional Agile/Scrum teams.
Qualifications
Experience with LLMs Generative AI Agentic AI or AI-assisted engineering workflows.
Experience with API design microservices event-driven architecture or serverless backend development.
Experience with CI/CD pipelines infrastructure as code automated testing and production deployment practices.
Experience with data governance metadata management lineage data observability or operational controls for enterprise data platforms
Required Experience:
IC
About Company
Bristlecone is the leading provider of connected logistics solutions, including AI applications, SAP integration, and more.