Java software engineer with GCP experience and DataFlow SME.
Location : Canada
Job Type : Temp/Contract
Hours : Full Time
Required Years of Experience : 4+
Required Education : Bachelor degree (BS in Computer Science) or similar
Travel : No
Relocation : No
Job Description :
GAL AeroStaff, which provides specialized staffing and recruiting services in the Business & Commercial aviation industry, is presently seeking a Java software engineer with GCP experience and DataFlow SME. based out in Canada. GAL AeroStaff is an internationally approved OEM service provider/supplier. We offer competitive salaries, as well as a professional and ethical work environment
- Design and develop data processing pipelines using Java/Apache Beam and Cdap and deploying into Google cloud in Google GCP Dataflow.
- Developed various custom Cdap plugins in Java for transformations. Developed data pipelines monitoring system using Cdap Rest APi & Stackdriver API, Alerts from Java application.
- Design and develop a solution to keep data in sync from on-premises Hadoop cluster to GCP cloud. A data change in Mapr clusters copies the data in near real-time to GCP and triggers a job in GCP.
- Migrating Hive based data transformation pipelines from on-premises Mapr Hadoop clusters to Google GCP Dataproc clusters and converting it to Spark pipelines using Pyspark
- Developed POC using various technologies Apache Beam/Spark/Cdap/DataFusion based data pipelines on GCP
- Explore cloud-based new technologies/products for Marketing Analytics and Data Science team and provide data engineering consultancy
Required Qualifications :