Senior Software Engineer II [Lead Data Engineer / Mumbai OR Bangalore]

As the Lead Data and AI Engineer, you are the technical architect for Nielsen’s GenAI and data evolution in the region. You will bridge the gap between high-level strategy and functional execution by leading the setup, definition, and industrialization of our Databricks ecosystem. This role is not just about writing code; it is about building a sustainable, governed engineering foundation that transitions experimental AI into enterprise-grade solutions. You will act as the "Technical Commander" for the Databricks environment, ensuring every solution is outcome-driven and architecturally sound.

Core Goals & Responsibilities

• Databricks Ecosystem Leadership: Lead the technical definition and setup of the Databricks environment. You are responsible for ensuring scalability and maintaining the integrity of the Unity Catalog for global data governance.
• End-to-End Engineering: Architect and build robust pipelines using common open-source data formats, ensuring they are optimized for both traditional analytics and emerging AI workloads.
• Operational Excellence: Partner closely with CloudOps and DevOps to ensure 99.99% reliability and seamless CI/CD integration.
• Functional Collaboration: Engage directly with Finance, HR, and Product owners to understand business requirements and translate them into high-impact technical solutions.
• Community & Mentorship: Build and lead a community of developers within the Databricks environment. You will provide technical guidance, unblock delivery friction, and ensure the team builds with "Strategic Foresight" regarding cloud costs (FinOps).

 

Expertise & Technology Stack

• Core Engineering: 8–10+ years of deep experience in data engineering and architectural design.
• Frameworks & Languages: Expert-level proficiency in Python, Heavy SQL, and dbt for transformation layers.
• AI & Orchestration: Hands-on experience with LangChain and vector database integration for GenAI applications.
• Databricks Mastery: Deep knowledge of Delta Lake, Unity Catalog, and Databricks Workflows.

Analytical Thinking: Ability to challenge status-quo assumptions and provide a pragmatic pivot when technical risks arise.

• Experience: 8–10+ years in a Senior or Lead Engineering capacity.
• Education/Certification: * Required: Proven track record of delivering complex, large-scale data solutions.
• Preferred: Databricks Certified Data Engineer Professional or AWS Certified AI/Machine Learning Specialty.
• Communication: A "Bottom Line Up Front" (BLUF) communication style. You must be able to engage with non-technical stakeholders (HR/Finance) and explain the "how" and "why" of technical decisions clearly.

Please be aware that job-seekers may be at risk of targeting by scammers seeking personal data or money. Nielsen recruiters will only contact you through official job boards, LinkedIn, or email with a nielsen.com domain. Be cautious of any outreach claiming to be from Nielsen via other messaging platforms or personal email addresses. Always verify that email communications come from an @nielsen.com address. If you're unsure about the authenticity of a job offer or communication, please contact Nielsen directly through our official website or verified social media channels.