Azure Data Engineer (Databricks Expert) Job at Eviden, Canada

N0k5OVNhK1hVYVVCeHYwUjNjbzJpaW1MZEE9PQ==
  • Eviden
  • Canada

Job Description

Job Description

Role : Azure Data Engineer (Databricks Expert)

Location : Toronto, ON

Hire Type : Full-Time with Eviden

Job Description:

About the Role:

We are seeking a highly skilled and experienced Senior Azure Data Engineer to join team and tackle critical performance challenges within Azure data platform. You will play a pivotal role in optimizing and re-architecting key components of data infrastructure, directly impacting the performance and scalability of core data marts. This is not a maintenance role; this is a chance to make a significant impact by redesigning and implementing solutions for a high-visibility project.

Responsibilities:

  • Performance Optimization: Lead the performance tuning and optimization of Azure-based data pipelines, specifically addressing the current bottleneck where processing 1000 records takes 40 minutes. This requires a deep understanding of Azure Data Factory (ADF), Databricks (Spark), and data processing best practices.
  • Architecture Review and Redesign: Evaluate the existing Azure data platform architecture, identify bottlenecks and design flaws, and propose and implement solutions for a more efficient and scalable system. This will involve working with Delta tables and optimizing data storage and retrieval.
  • Databricks Expertise: Utilize your advanced PySpark skills, particularly with structured streaming, to re-engineer data transformation and processing logic within Databricks. The ideal candidate will have a proven track record of optimizing Spark jobs for performance.
  • Data Integration: Work with Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Database (DB), and file exchange processes (batch mode) to ensure seamless data flow and integration.
  • MDM Integration (Desirable): Contribute to the ongoing development and optimization of Master Data Management (MDM) system, which follows a hub-and-spoke architecture. Experience with large-scale MDM implementations is highly desirable.
  • Collaboration: Work closely with other engineers, business stakeholders, and vendors to understand requirements, communicate progress, and ensure successful project delivery. You will be expected to mentor and guide other team members.

Qualifications:

  • Minimum experience of implementing 2 projects on Azure data platform
  • Minimum experience of around 8-12 years
  • "Ninja" Level Expertise: A proven track record of success in optimizing and scaling Azure data platforms, particularly with Databricks and Spark. We are looking for an expert who can quickly diagnose and resolve complex performance issues.
  • Deep Azure Knowledge: Extensive experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse, Event hub, and other relevant Azure services. Experience with Azure SaaS offerings is a plus.
  • Databricks Mastery: Advanced proficiency in PySpark and structured streaming. Demonstrated ability to write and optimize complex Spark queries for performance.
  • Data Warehousing and MDM: Solid understanding of data warehousing principles and experience with Master Data Management (MDM) implementations, preferably with a hub-and-spoke architecture.
  • Problem-Solving Skills: Exceptional analytical and problem-solving skills, with the ability to identify root causes of performance bottlenecks and design effective solutions.
  • Communication Skills: Excellent communication and collaboration skills, with the ability to work effectively 1 with both technical and non-technical 2 stakeholders.

Desirable:

  • Experience with AI Search and Cosmos DB.
  • Experience with Informatica MDM in an Azure environment.

Job Tags

Full time,

Similar Jobs

Wellspring Nurse Source

Travel Acute Inpatient Rehabilitation Physical Therapist Job at Wellspring Nurse Source

 ...Job Description Wellspring Nurse Source is seeking a travel Physical Therapist for a travel job in Hilo, Hawaii. Job Description...  ...Requirements ~ Specialty: Physical Therapist ~ Discipline: Therapy ~ Start Date: 02/23/2026~ Duration: 26 weeks ~40 hours... 

Maiden Home

Interior Design Advisor Job at Maiden Home

 ...their colleagues, and an eagerness to reimagine the world of luxury furnishings. We are looking for an ambitious and talented Interior Design Advisor to provide world-class consultation and support to our clientele. Our ideal candidate will champion the Maiden Home... 

PrimeFlight Aviation Services

Ramp Agent - MSP Airport Job at PrimeFlight Aviation Services

 ...LIFTING BAGS, GUIDING PLANES, LAUNCHING JOURNEYS - NOW HIRING RAMP AGENTS! The Ramp Agent is responsible for safely loading and unloading...  ...supporting ground operations to ensure efficient and secure airport service flow in compliance with safety and airline standards.... 

Keller Williams Flagship of Maryland

Social Media and Marketing Specialist Job at Keller Williams Flagship of Maryland

 ...for Full-TIme and Part Time Event and Media Specialist and Paid Interns. Our on-location...  ...* Press Releases * Digital and Print Marketing The ideal candidate demonstrates...  ...hourly) and bonus. to commensurate with experience. ON LOCATIONS . Company Description... 

Jobsultant Solutions

Remote Data Entry Clerk - Work at Home Job at Jobsultant Solutions

 ...About the job Remote Data Entry Clerk - Work at Home Data Entry Clerk - Administrative Assistant - Survey Panelist - Online Operate At Home Position (Part-time/ Full-time) - Become Part Of Our USA Market Research Panel Today! Are you data entry clerk, administrative...