Azure Data Engineer (Databricks Expert) Job at Eviden, Canada

N0k5OVNhK1hVYVVCeHYwUjNjbzJpaW1MZEE9PQ==
  • Eviden
  • Canada

Job Description

Job Description

Role : Azure Data Engineer (Databricks Expert)

Location : Toronto, ON

Hire Type : Full-Time with Eviden

Job Description:

About the Role:

We are seeking a highly skilled and experienced Senior Azure Data Engineer to join team and tackle critical performance challenges within Azure data platform. You will play a pivotal role in optimizing and re-architecting key components of data infrastructure, directly impacting the performance and scalability of core data marts. This is not a maintenance role; this is a chance to make a significant impact by redesigning and implementing solutions for a high-visibility project.

Responsibilities:

  • Performance Optimization: Lead the performance tuning and optimization of Azure-based data pipelines, specifically addressing the current bottleneck where processing 1000 records takes 40 minutes. This requires a deep understanding of Azure Data Factory (ADF), Databricks (Spark), and data processing best practices.
  • Architecture Review and Redesign: Evaluate the existing Azure data platform architecture, identify bottlenecks and design flaws, and propose and implement solutions for a more efficient and scalable system. This will involve working with Delta tables and optimizing data storage and retrieval.
  • Databricks Expertise: Utilize your advanced PySpark skills, particularly with structured streaming, to re-engineer data transformation and processing logic within Databricks. The ideal candidate will have a proven track record of optimizing Spark jobs for performance.
  • Data Integration: Work with Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Database (DB), and file exchange processes (batch mode) to ensure seamless data flow and integration.
  • MDM Integration (Desirable): Contribute to the ongoing development and optimization of Master Data Management (MDM) system, which follows a hub-and-spoke architecture. Experience with large-scale MDM implementations is highly desirable.
  • Collaboration: Work closely with other engineers, business stakeholders, and vendors to understand requirements, communicate progress, and ensure successful project delivery. You will be expected to mentor and guide other team members.

Qualifications:

  • Minimum experience of implementing 2 projects on Azure data platform
  • Minimum experience of around 8-12 years
  • "Ninja" Level Expertise: A proven track record of success in optimizing and scaling Azure data platforms, particularly with Databricks and Spark. We are looking for an expert who can quickly diagnose and resolve complex performance issues.
  • Deep Azure Knowledge: Extensive experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse, Event hub, and other relevant Azure services. Experience with Azure SaaS offerings is a plus.
  • Databricks Mastery: Advanced proficiency in PySpark and structured streaming. Demonstrated ability to write and optimize complex Spark queries for performance.
  • Data Warehousing and MDM: Solid understanding of data warehousing principles and experience with Master Data Management (MDM) implementations, preferably with a hub-and-spoke architecture.
  • Problem-Solving Skills: Exceptional analytical and problem-solving skills, with the ability to identify root causes of performance bottlenecks and design effective solutions.
  • Communication Skills: Excellent communication and collaboration skills, with the ability to work effectively 1 with both technical and non-technical 2 stakeholders.

Desirable:

  • Experience with AI Search and Cosmos DB.
  • Experience with Informatica MDM in an Azure environment.

Job Tags

Full time,

Similar Jobs

Penn Tank Lines

CDL-A Local Company Tanker Truck Driver Job at Penn Tank Lines

 ...Description At Penn Tank Lines , we recognize that our Company Drivers and Independent Contractors play a crucial role in our success....  ...Experience Preferred What We Offer: Home Every Day: As a Local Driver, you'll be home every day. No more long-haul trips just... 

City of New York

Digital Forensic Lab Analyst (DFL Analyst) Job at City of New York

 ...Justice with Integrity, the BXDA places a focus on assistance for crime victims while simultaneously ensuring fairness to defendants....  ...the BXDA seeks experienced and enthusiastic Digital Forensic Lab Analyst (DFL Analyst). The Digital Forensics Laboratory, part of the... 

BeaconFire Inc.

Technical Recruiter Job at BeaconFire Inc.

 ...We are seeking a dynamic and results-driven Technical Recruiter to join our growing team. The ideal candidate will work closely with hiring managers to understand and fulfill the talent needs of our organization and be responsible for connecting with top technical... 

Diversified Protection Corporation

Asset Protection & Security Officer Job at Diversified Protection Corporation

 ...monitoring security procedures. Candidates must be U.S. citizens or lawful residents, at least 21 years old, and possess a high school...  ...two years of relevant experience in armed security or law enforcement. This position pays $34.32 per hour and requires background checks... 

Southwest Airlines

Ramp Agent - LIH Job at Southwest Airlines

 ...phone, virtually, and/or inperson and not selected for the Ramp Agent position at LIH within the last 12 months.*** Pay & Benefits...  ...Must be able to obtain a SIDA badge and meet all local airport requirements May be required by Station Leadership to obtain...