ETL DEVELOPER – DATABRICKS

Location: Toronto, Downtown core, hybrid

Salary: Market Rate+

RESPONSIBILITIES:

  • Understand business requirements for data and information especially in the financial services industry.
  • Design data integration or ETL processing using Databricks, Spark, Hadoop, and Hive for client engagements based on an approach that is holistic, scalable, pragmatic, and effective design of data processing.
  • Ingest data and transform data in Databricks for batch or near real-time processing using a variety of tools such as Pyspark, SparkSQL, Hive, Azure Data Factory, and Microsoft Azure.
  • Design and integrate pipelines with Continuous Integration/Delivery framework as needed.
  • Develop automated unit-testing.
  • Develop design and operational documentation for job schedules and data pipelines.
  • Produce estimates for data analysis and mapping, design, development, and testing.
  • Ensure compliance with business, data, and technical requirements.
  • Ensure that client’s enterprise architecture standards, policies, and procedures are followed.​

 

REQUIREMENTS:

  • Minimum 5 years experience in Information Technology, participating in complex projects.
  • Minimum Bachelors degree in Computer Science.
  • Strong communication skills.
  • Strong knowledge in data analysis, database development, big data, data lakehouse, data warehousing life cycle and data integration methodologies.
  • Strong knowledge of Databricks features and working experience in developing data integration solutions using Databricks.
  • Strong programming skills in Pyspark and SparkSQL.
  • Working knowledge in Hadoop/Hive and one of Oracle or Microsoft SQL databases.
  • Knowledgeable in both relational and dimensional data modeling (with both Kimball and Inmon approaches).
  • Experience in .NET development and UNIX shell scripting.
  • Knowledge in data management and Continuous Integration and Delivery (CI/CD) principles.
  • Good overall business knowledge in financial industries (i.e. one or more in retail banking, commercial banking, capital markets, wealth management, insurance, pension fund, and fintech).
  • Knowledgeable of Azure computing concepts and services.
  • Ability to work independently and excel in a team environment.

Apply this job