Senior Software Engineer II (Databricks Engineer)

Strategic Mandate

As a Senior Data and AI Engineer II, your mandate is to bridge the gap between architectural design and hands-on execution. Reporting into the Lead Data and AI Engineer, you will be a key contributor to the AI Braintrust, focusing on the technical industrialization of our Databricks environment. This role is built for a "Strategic Executor"—someone who understands the functional "how" of scaling AI pipelines while maintaining a sharp focus on the "why" behind business requirements from Finance, HR, and Product stakeholders.

Core Goals & Responsibilities

• Databricks Engineering: Execute the end-to-end build of data solutions within the Databricks ecosystem, ensuring all pipelines are optimized for performance and cost-efficiency (FinOps).

• Unity Catalog Implementation: Assist in the definition and rigorous maintenance of the Unity Catalog to ensure data governance, residency, and security protocols are met across all workflows.

• Modern Tooling & Frameworks: Develop and maintain transformation layers using dbt, and leverage Python and Heavy SQL to build resilient data structures.

• AI Orchestration: Implement GenAI patterns using LangChain and open-source data formats to transition experimental models into production-ready features.

• Collaborative Delivery: Work in lockstep with CloudOps and DevOps to ensure seamless CI/CD integration and 99.99% system reliability.

• Stakeholder Engagement: Partner with owners in Finance and HR to ensure data outputs are accurate, actionable, and aligned with departmental goals.

Expertise & Technology Stack

• Experience: 5+ years of progressive experience in data engineering, with a focus on cloud-native architectures.

• Technical Proficiency: * Advanced Python and SQL skills.

• Hands-on experience with dbt for data modeling.

• Exposure to AI frameworks such as LangChain.

• Cloud Infrastructure: Familiarity with AWS environments and automated deployment patterns.

• Problem Solving: A pragmatic approach to engineering; you focus on building sustainable solutions rather than "over-engineering" for the sake of complexity.

• Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field.
• Certifications: * Preferred: Databricks Certified Data Engineer Associate/Professional.
• Bonus: AWS Certified AI/ML Specialty.
• Communication Style: Direct and scannable. You prioritize BLUF (Bottom Line Up Front) communication when reporting status or risks to technical leads and business partners.

Please be aware that job-seekers may be at risk of targeting by scammers seeking personal data or money. Nielsen recruiters will only contact you through official job boards, LinkedIn, or email with a nielsen.com domain. Be cautious of any outreach claiming to be from Nielsen via other messaging platforms or personal email addresses. Always verify that email communications come from an @nielsen.com address. If you're unsure about the authenticity of a job offer or communication, please contact Nielsen directly through our official website or verified social media channels.