Data Engineer

Location: Indore/Raipur

Experience: 2 to 4 years

Key Responsibilities

• Design, develop, and manage scalable data pipelines and ETL workflows using Databricks, PySpark, and SQL for large-scale data processing.
• Build and maintain data ingestion frameworks to extract data from enterprise systems such as SAP APIs, REST services, and relational databases .
• Develop and optimize Delta Lake based data architecture to ensure reliable, high-performance data storage and processing.
• Design and implement data transformation pipelines to convert raw data into curated datasets for analytics and reporting.
• Optimize Spark jobs and SQL queries to improve performance and reduce compute costs.
• Implement data quality validation, monitoring, and error handling frameworks for reliable pipeline execution.
• Build automated workflow orchestration and scheduling mechanisms for end-to-end data processing pipelines.
• Collaborate with data analysts, business stakeholders, and platform teams to design efficient data solutions.
• Develop and maintain data models and schema design for data lake and downstream analytical systems.
• Support data platform engineering activities , including cluster configuration, performance tuning, and reusable utility development.
• Troubleshoot production pipeline failures, data inconsistencies, and performance issues .
• Develop Python utilities and frameworks to support data ingestion, transformation, and automation tasks.
• Implement data governance, security, and access control standards across enterprise data pipelines.
• Participate in code reviews, documentation, and best practices to improve overall data engineering standards.
• Support large-scale data integrations and migrations from legacy systems to modern cloud data platforms.
• Ownership of the entire data pipeline lifecycle, from development to deployment

Required Skills

• 2+ years of experience in Data Engineering, Data Pipeline Development, and Data Processing .
• Strong experience with Python, PySpark, and SQL for large-scale data transformations.
• Hands-on experience with Databricks, Delta Lake, and distributed data processing frameworks .
• Experience integrating data from REST APIs, SAP systems, and enterprise data sources .
• Strong knowledge of data modeling, schema design, and ETL best practices .
• Experience working with cloud data platforms (GCP / AWS / Azure) and cloud storage systems.
• Experience with workflow orchestration, job scheduling, and automated data pipelines .
• Ability to optimize Spark workloads and troubleshoot performance issues in large datasets.
• Strong problem-solving skills and ability to work in fast-paced data platform environments .