Location: Toronto, Downtown core, hybrid
Salary: Market Rate+
RESPONSIBILITIES:
- Understand business requirements for data and information especially in the financial services industry.
- Design data integration or ETL processing using Databricks, Spark, Hadoop, and Hive for client engagements based on an approach that is holistic, scalable, pragmatic, and effective design of data processing.
- Ingest data and transform data in Databricks for batch or near real-time processing using a variety of tools such as Pyspark, SparkSQL, Hive, Azure Data Factory, and Microsoft Azure.
- Design and integrate pipelines with Continuous Integration/Delivery framework as needed.
- Develop automated unit-testing.
- Develop design and operational documentation for job schedules and data pipelines.
- Produce estimates for data analysis and mapping, design, development, and testing.
- Ensure compliance with business, data, and technical requirements.
- Ensure that client’s enterprise architecture standards, policies, and procedures are followed.
REQUIREMENTS:
- Minimum 5 years experience in Information Technology, participating in complex projects.
- Minimum Bachelors degree in Computer Science.
- Strong communication skills.
- Strong knowledge in data analysis, database development, big data, data lakehouse, data warehousing life cycle and data integration methodologies.
- Strong knowledge of Databricks features and working experience in developing data integration solutions using Databricks.
- Strong programming skills in Pyspark and SparkSQL.
- Working knowledge in Hadoop/Hive and one of Oracle or Microsoft SQL databases.
- Knowledgeable in both relational and dimensional data modeling (with both Kimball and Inmon approaches).
- Experience in .NET development and UNIX shell scripting.
- Knowledge in data management and Continuous Integration and Delivery (CI/CD) principles.
- Good overall business knowledge in financial industries (i.e. one or more in retail banking, commercial banking, capital markets, wealth management, insurance, pension fund, and fintech).
- Knowledgeable of Azure computing concepts and services.
- Ability to work independently and excel in a team environment.