Azure Data Engineer || Purchase NY, Florham Park NJ, Charlotte NC, NYC, Conshohocken PA (ONSITE)

Hi Associates,

Job Title: Azure Data Engineer

Location: Purchase NY, Florham Park NJ, Charlotte NC, NYC, Conshohocken  PA (ONSITE)

Duration: 6-12 months

Mandatory skills: Azure Data Lake Storage (ADLS), PySpark, SparkSQL, Databricks

ROLE SUMMARY:

The Data Engineer will be a hands-on developer that will work within the team to build data pipelines, implement best practices, and provide thought leadership along with contributing to the overall system architecture and work closely with other teams within Sompo International to collaborate and share best practices.

ROLE RESPONSIBILITIES:

Integrating and transforming database applications and web service application programing interfaces (APIs) through innovative strategies to support operations and decision making.

Construct and maintain high performing data pipelines, streaming and batch, for internal and external datasets.

Work with domain experts to translate data requirements to processing specifications.

Implement procedures to improve system efficiency, security, and compliance.

Integrating and transforming data utilizing dbt to build and maintain the Sompo International analytics environment sitting on top of the Sompo International Data Lake.

Construct and maintain high performing analytic pipelines.

Work with domain experts to translate data requirements into functioning models.

Develop and maintain documentation relating to all assigned systems and projects

TECHNICAL QUALIFICATIONS:

Experience working with disparate input files and sources like Json, XML, CSV, Parquet, Delta and SQL DB

Hands on experience in the Azure data stack - Azure ADLS Gen2, Azure Databricks, Azure Dev Ops (NO ADF or Synapse)

Working knowledge of with Pyspark & SQL developing data pipelines in Azure Databricks

Working knowledge of Scala

Databrick Application tuning

Spark SQL

Experience with dbt

Exposure to Jinja / Python

GENERAL QUALIFICATIONS:

Self-motivated with superior analytical and problem-solving skills, and a strong sense of accountability

Team player who believes in open and continuous communication

Experience in the Insurance industry is a plus

Willingness to learn

Thanks and Regards

Nick Walker

Senior IT Recruiter

Ph. No : 732-576-3829,

Email : nick.walker@cssitconsulting.com