Databricks Architect @ Deloitte Consulting

Job Description

Summary Position Summary Databricks Architect Lead Architect About the Role: We are searching for a passionate and strategic Databricks Architect to join our team as a Lead Architect. You will be a technical leader who plays a crucial role in shaping the organization's data strategy and its implementation on the Databricks platform. You will be responsible for designing, developing, and deploying highly scalable, secure, and performant data pipelines and architectures. You will also be responsible for building and fostering a high-performing Databricks engineering team. Responsibilities: Lead the design and implementation of complex data pipelines and architectures on Databricks. Experience with Dataiku along with Databricks Collaborate with data engineers, scientists, and stakeholders to understand data needs and translate them into technical solutions. Optimize performance, scalability, and cost-effectiveness of Databricks workloads. Develop and implement security. best practices for data access and governance on Databricks (ACLs, IAM roles, etc.). Automate data pipelines and infrastructure using tools like Terraform, CloudFormation, or Databricks Workflows. Lead and mentor other Databricks engineers, fostering a knowledge-sharing environment. Stay up to date on the latest advancements and best practices within the Databricks ecosystem. Collaborate with stakeholders across the organization to ensure data solutions meet business needs. Working on data architecture, data modeling, and data management Analyze, interpret, and derive insights from complex data sets. Technical Details: Databricks Expertise: Deep understanding of Apache Spark architecture and operations In-depth knowledge of Databricks architecture, including clusters, notebooks, libraries (e.g., Koalas, MLflow), and security best practices. Experience with Delta Lake tables, including Delta Lake optimizations, data governance strategies, and ACID transactions. Ability to configure Spark configurations for optimal performance on Databricks clusters (cluster types, resource allocation, etc.). Experience with Databricks workspace management and resource optimization (cluster autoscaling, workload scheduling). Strong experience in building solutions on Databricks platform using Unity Catalog Proficiency in languages like Python, Scala, or SQL, which are commonly used for data analysis and manipulation in Databricks. Data Engineering & Warehousing: Familiarity with data warehousing concepts like star schema design, data modeling techniques, and data lineage. Understanding of data quality frameworks and data cleansing techniques. Experience with data ingestion tools (e.g., Kafka, Flume) for streaming data pipelines and data lake architectures. Expertise of big data technologies like Hadoop, Hive etc. Expertise in Cloud Data Integration platforms like Azure Data Factory, AWS Glue, Data Fusion Expertise in using S3 as a data lake for storing and managing large datasets in various formats (parquet, CSV, etc.). CI/CD & Automation: Knowledge of CI/CD pipelines and DevOps methodologies for automating data pipelines and infrastructure deployments. Experience with version control systems (Git) for code management and collaboration. Cloud Platforms: Expertise in one or more cloud platforms (AWS, Azure, GCP) for deploying and managing Databricks clusters and related infrastructure. Understanding of cloud storage options (S3, Azure Blob Storage, GCP Cloud Storage) for data storage and integration with Databricks. Qualifications: Minimum 7+ years of experience in designing and implementing big data solutions. Proven experience with Databricks, including Delta Lake, notebooks, clusters, and Spark configurations. Strong understanding of data warehousing, data modeling, and data governance principles. Experience with cloud platforms like AWS and Azure. Excellent problem-solving and analytical skills. Strong communication and collaboration skills. Proven ability to lead and mentor technical teams. Ability to clearly explain technical concepts to non-technical stakeholders. Preferred Qualifications: Experience with machine learning and data science workflows on Databricks. Certifications in Databricks (e.g., Databricks Certified Spark Developer) or AWS Certified Solutions Architect a plus. Experience with containerization technologies like Docker and Kubernetes for scalable deployments. Knowledge of MLOps is a plus. Recruiting tips From developing a stand out resume to putting your best foot forward in the interview, we want you to feel prepared and confident as you explore opportunities at Deloitte. Check out recruiting tips from Deloitte recruiters. Benefits At Deloitte, we know that great people make a great organization. We value our people and offer employees a broad range of benefits. Learn more about what working at Deloitte can mean for you. Our people and culture Our diverse, equitable, and inclusive culture empowers our people to be who they are, contribute their unique perspectives, and make a difference individually and collectively. It enables us to leverage different ideas and perspectives, and bring more creativity and innovation to help solve our client most complex challenges. This makes Deloitte one of the most rewarding places to work. Learn more about our inclusive culture. Our purpose Deloittes purpose is to make an impact that matters for our clients, our people, and in our communities. We are creating trust and confidence in a more equitable society. At Deloitte, purpose is synonymous with how we work every day. It defines who we are. We are focusing our collective efforts to advance sustainability, equity, and trust that come to life through our core commitments. Learn more about Deloitte's purpose, commitments, and impact. Professional development From entry-level employees to senior leaders, we believe theres always room to learn. We offer opportunities to build new skills, take on leadership opportunities and connect and grow through mentorship. From on-the-job learning experiences to formal development programs, our professionals have a variety of opportunities to continue to grow throughout their career. Requisition code: 177451

Employement Category:

Employement Type: Full time
Industry: Management Consulting / Strategy
Role Category: Management Consulting / Strategy
Functional Area: Not Applicable
Role/Responsibilies: Databricks Architect

+ View Contact

Keyskills:   Apache Spark Python Scala SQL Data Warehousing Hadoop Hive DevOps Git AWS Azure GCP Data Science Machine Learning Docker Kubernetes Databricks Dataiku Delta Lake Terraform CloudFormation Azure Data Factory AWS Glue S3 CICD MLOps