Data Engineer (Remote)
Contract Remote
Azure, Apache Spark, Scala, Yaml.
About the Role
We are looking for an extremely talented Data Engineer to design, build, and optimize large scale data solutions. In this role we require experience with Apache Spark, Scala, Azure big data products also a strong background in analysis is a must to support enterprise level reporting and analytics projects.
Primary Responsibilities
Design out, build out, and manage large scale big data pipelines with Apache Spark and Scala. Write complex SQL queries for transform, analysis and reporting. Create and improve Power BI dashboards for business insights. Work with cross functional teams to integrate data pipelines, analytics, and reporting solutions. See to it that data quality, security, governance and compliance is met in all envs. Work with large distributed data sets and improve data work flows for performance and cost.
Required Skills Experience.
Strong in hands on experience with Apache Spark and Scala. In terms of SQL have great skill set for data manipulation and analysis. Had experience in developing Power BI dashboards and reports. Also have a good handle on distributed computing (Hadoop, Hive, etc. which is a plus. Also we see value in experience with storage formats like Parquet and Delta Lake. Also very much into Object Oriented Programming concepts and best practices in Scala development. Highly Desirable.
Spark performance tuning we have the ability to diagnose and resolve issues. Also have in depth experience with Azure big data technologies:.
Azure Data Lake (ADLS Gen2)
Azure Databricks
Azure Synapse Analytics
Azure VM (IaaS)
Azure SQL Server
Experience in ADF, ADL, YAML and cloud based data pipelines. In the financial domain background or familiar with ERP systems (SAP, Oracle ERP) is required. We are looking for an understanding of regulatory and compliance issues related to financial data. We are looking for experience in developing and troubleshooting Azure Automation Runbooks (PowerShell). Also we are looking for knowledge of SQL Server backup/restore in storage accounts (Azure and On Prem). We have a preference for candidates who have developed Microsoft Fabric reports. Prior experience in Microsoft environments also preferred. Also we will consider exposure to AWS or GCP cloud platforms.
