Azure Databricks Platform Lead Engineer Job in
- Consultancy
- 5 to 9 Yrs
- Noida, Gurugram
- 1 month ago
- Email to a friend
- Report this job
Job Description
- Birlasoft, a powerhouse where domain expertise, enterprise solutions, and digital technologies converge to redefine business processes. We take pride in our consultative and design thinking approach, driving societal progress by enabling our customers to run businesses with unmatched efficiency and innovation. As part of the CK Birla Group, a multibillion-dollar enterprise, we boast a 12,500+ professional team committed to upholding the Group's 162-year legacy. Our core values prioritize Diversity, Equity, and Inclusion (DEI) initiatives, along with Corporate Sustainable Responsibility (CSR) activities, demonstrating our dedication to building inclusive and sustainable communities. Join us in shaping a future where technology seamlessly aligns with purpose About the Role - Azure Databricks Platform Lead Engineer Location - Noida /Pune Responsibilities - Design and Maintain Data Pipelines: Develop and optimize data pipelines using Azure Data Factory and Databricks. Ensure smooth and efficient data flow through the pipelines. Troubleshooting and Optimization: Identify and resolve issues related to cluster creation, performance, and resource allocation. Optimize existing data pipelines for performance and reliability. Data Quality and Integrity: Implement measures to ensure data quality and integrity across various data sources. Develop custom error handling in notebooks to ensure proper failure reporting. Collaboration and Communication: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions. Provide processed and usable data for business decision-making. Security and Compliance: Implement data security best practices, including encryption and role-based access control (RBAC). Ensure compliance with data governance and regulatory requirements. Monitoring and Performance Management: Use tools like Azure Monitor to track data pipeline performance and troubleshoot issues. Manage and monitor cluster performance to avoid overload and service throttling. Managing Databricks Notebooks Across Environments: Use version control systems like Git to manage notebooks across different environments (dev, homologation, prod). Modularize code by breaking down notebooks into reusable modules for easier testing and deployment. Implement a CI/CD pipeline to automate the deployment of notebooks from development to production using tools like Azure DevOps or GitHub Actions. Token Management: Avoid hardcoding tokens; instead, store them in Azure Key Vault and retrieve them programmatically within notebooks. Use service principals for authentication, which is more secure and scalable than using personal access tokens. Compute Engine Selection: Use job clusters instead of all-purpose clusters for running production jobs to reduce costs. Consider using serverless compute options if supported, as they automatically scale according to workload and can be more cost-effective. Implement compute policies to preconfigure clusters for specific purposes, such as development, testing, and production. Streaming Pipelines: Develop and manage streaming pipelines using Azure IoT Hub and Stream Analytics to analyze data for efficiency and handle data coming in from IoT devices. Required Skills - Azure Data Factory Azure Databricks Azure Monitor, APIM Azure Key Vault Azure IoT Hub Azure Stream Analytics Git (Version Control) CI/CD Tools (e.g., Azure DevOps, GitHub Actions) Cluster Management (Job clusters, all-purpose clusters) Serverless Compute Options Data Security (Encryption, Role-Based Access Control) Data Governance and Compliance Databricks Notebooks Service Principals for Authentication,
Employement Category:
Employement Type: Full timeIndustry: IT Services & Consulting
Role Category: Not Specified
Functional Area: Not Specified
Role/Responsibilies: Azure Databricks Platform Lead Engineer Job in
Contact Details:
Company: BirlasoftLocation(s): Noida, Gurugram
+ View Contact
Keyskills: Git Cluster Management Data Security Data Governance Azure Data Factory
Fraud Alert to job seekers!
₹ Not Specified
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in