Robosoft Technologies
This listing was posted on hirist.
Robosoft Technologies - Data Pipeline Engineer - ETL (7-12 yrs) Anywhere in
Location:
Anywhere in India/Multiple Locations
Description:
Data Pipeline Engineer6 MONTHS CONTRACTRoles and Responsibilities:- Develop, design and automate large scale, high-performance data processing systems (batch and/or streaming) to drive business growth and improve the product experience.- Evangelize high quality software engineering practices towards building data infrastructure and pipelines at scale. - Lead data engineering projects to ensure pipelines are reliable, efficient, testable, & maintainable- Design our data models for optimal storage and retrieval and to meet critical product and business requirements.- Understand and influence logging to support our data flow, architecting logging best practices where needed- Contribute to shared Data Engineering tooling & standards to improve the productivity and quality of output for Data Engineers across the company- Partner with leadership, engineers, program managers and data scientists to understand data needs.- Educate your partners: Use your data engineering experience to 'see what's missing', identifying and addressing gaps in their existing logging and processes.- Work with stakeholders to build data lineage, data governance and data cataloging.- Leading projects using agile methodologies.- Communicate effectively with people of all levels in the organization.- Recruit, retain and develop people skills to take bigger responsibilities and challenges. Experience & Skills:- 7+ years of relevant industry experience.- Experienced in custom ETL design, implementation and maintenance.- Working knowledge of relational databases and query authoring (SQL).- Experience with Java / Scala / Spark / Pyspark / Python/Databricks is preferred- Any prior cloud experience (AWS preferred)- Working experience with data at the large scale.- Experience designing, building and operating robust data pipelines.- Experience designing and deploying high performance pipelines with reliable monitoring and logging practices.- Effectively work across team boundaries to establish overarching data architecture and provide guidance to individual teams.- Optimize your ETL pipeline to minimize data transfer and storage costs. - Use Data Bricks SQL's auto scaling feature to adjust the number of workers based on the workload,Tech Stack: - AWS Services - Databricks SQL Serverless - Databricks SQL, - Databricks workspaces, - Databricks notebooks, - Scheduling Databricks Job, - Delta Lake Architecture, - Lakehouse Concepts and understanding, - Spark Structured/Streaming - File Formats (CSV, Avro, Parquet) - CI/CD for ETL - Databricks SQL unified governance model (using Unity catalog) across clouds, open formats, APIs (ref:hirist.tech)
Visit Our Partner Website
This listing was posted on another website. Click here to open: Go to hirist
Important Safety Tips
- Always meet the employer in person.
- Avoid sharing sensitive personal and financial information.
- Avoid employment offers that require a deposit or investment.
To learn more, visit the Safety Center or click here to report this listing.
More About this Listing: Robosoft Technologies - Data Pipeline Engineer - ETL (7-12 yrs) Anywhere in
Robosoft Technologies - Data Pipeline Engineer - ETL (7-12 yrs) Anywhere in is a Engineering Engineer Job at Robosoft Technologies located in India. Find other listings like Robosoft Technologies - Data Pipeline Engineer - ETL (7-12 yrs) Anywhere in by searching Oodle for Engineering Engineer Jobs.
Robosoft Technologies - Data Pipeline Engineer - ETL (7-12 yrs) Anywhere in is a Engineering Engineer Job at Robosoft Technologies located in India. Find other listings like Robosoft Technologies - Data Pipeline Engineer - ETL (7-12 yrs) Anywhere in by searching Oodle for Engineering Engineer Jobs.