Data Engineer
Job Summary
Data Engineer Cloud & Big Data
Role Summary:
Responsible for designing developing and maintaining scalable cloud-based data pipelines and data platforms across AWS GCP Kafka and Databricks. The role involves building reliable ETL/ELT pipelines real-time data ingestion data transformation monitoring and performance/cost optimization.
In this role you will...
- Design and develop ETL/ELT data pipelines using AWS Glue Databricks and Spark.
- Build and maintain batch and real-time data pipelines using Apache Kafka / Confluent Kafka.
- Develop CDC-based data ingestion and event-driven data processing.
- Work with AWS services including Glue S3 Lambda IAM KMS Secrets Manager and related services.
- Develop and optimize data solutions on GCP including BigQuery Cloud Storage Cloud Functions and related services.
- Implement data processing using Databricks PySpark Delta Lake Bronze/Silver/Gold architecture and DLT.
- Manage and optimize Elasticsearch/OpenSearch data ingestion indexing queries and cluster performance.
- Implement data quality validation monitoring logging and alerting across data pipelines.
- Troubleshoot production issues involving Kafka schema compatibility CDC Spark Glue Databricks and cloud infrastructure.
- Optimize pipeline performance and cloud infrastructure costs.
- Implement secure data solutions using IAM encryption network controls secrets management and least-privilege access.
- Collaborate with application DevOps QA analytics and product teams to deliver reliable data solutions.
Youvegot what it takes if you have
- 5 years of experience in Data Engineering / Big Data Engineering.
- Strong hands-on experience with AWS Databricks Kafka.
- Experience working in multi-cloud environments (AWS/GCP).
- Strong understanding of distributed systems streaming CDC schema evolution and data governance.
- Experience troubleshooting high-volume production data pipelines.
- Experience with cloud performance and cost optimization.
#LI-Onsite
Our Culture:
Spark Greatness. Shatter Boundaries. Share Success. Are you ready Because here right now is where the future of work is happening. Where curious disruptors and change innovators like you are helping communities and customers enable everyone anywhere to learn grow and advance. To be better tomorrow than they are today.
Who We Are:
At Cornerstone we believe in AI that works in the service of people amplifying their judgment to drive high-performing future-ready organizations forward. Cornerstone Workforce AI the intelligence platform for workforce readiness brings together workforce and labor market data into a proprietary Cornerstone People Graph translating signals into intelligence targeting learning where it matters developing critical skills and surfacing hidden talent. Delivered as an open enterprise platform across whatever application your people work in every day Cornerstone Workforce AI is built for scale security and trust with certified AI guardrails. As an industry leader Cornerstone is helping approximately 7000 organizations 140M users across 186 countries build continuous workforce readiness.
Check us out on LinkedIn Comparably Glassdoor and Facebook!
Required Experience:
IC
About Company
Cornerstone and our intelligence platform for workforce readiness deliver the insights leaders want, the skills and learning people need, and AI agents that make action easy.