Senior Data Engineer-ADF, Databricks, Snowflake, 10+years

This listing was posted on The Resumator.

Senior Data Engineer-ADF, Databricks, Snowflake, 10+years

Location:

Hyderabad

Description:

Data Engineer Who We Are Streamline is your trusted partner for achieving digital transformation. With deep expertise in Enterprise Mobility, Product Engineering, and IT Transformation, we equip organizations to navigate the ever-changing IT landscape. We work closely with our clients, fostering lasting partnerships and embracing open communication to deliver transformative solutions. Our team of world-class strategists, architects, product and program managers, engineers, developers, and QA professionals possesses a deep understanding of the challenges and opportunities that organizations face in today's digital world. We work collaboratively with your team, working together to achieve short-term goals and enable long-term success. Streamline empowers organizations to achieve digital success. Role Summary This is a full-time remote position. The Data Engineer will work with a team of other software developers to define, design, develop, integrate, and re-engineer the Enterprise Data warehouse. Data Marts, Data Virtualization and Data Visualization components in different environments which meet customers’ analytical and business intelligence requirements, scales easily and supports deployment in highly available environments. The Senior engineer acts as the development and technical lead and individual contributor on complex projects, contributing to strategic vision and technical decisions, participating in vendor analysis and selection projects, introducing process improvements, completing a proof-of-concept projects for the introduction of changes to our architecture, and providing oversight on the work of other technical staff. Role Responsibilities Build data-intensive solutions that are highly available, scalable, reliable, secure, and cost-effective Create and maintain highly scalable data pipelines using Databricks , Azure, AWS, Kafka . Design and build ETL/ELT data pipelines to extract and process data from a variety of external/internal data sources. Identifies, designs, and implements internal process improvements: automating manual processes, optimizing data delivery Build data insights, data analytics, ML models, fraud and anomaly detection using Snowflake Build and deploy modern data solutions. Define, implement, and build jobs to populate data models. Relational and NoSQL Database management DevOps building CI/CD pipelines Technical System Expertise : Understands data warehouse development practices and processes including ETL design, Dimensional models, slowly changing dimensions, Data Security components, Data quality methods, how MPP databases operate and data flows. Aware of current technology benefits. Expected to independently develop full stack ETL solution from source to staging to presentation layers. Understands the building blocks, interactions, dependencies, and tools required to complete BI Data warehousing software and automation work. An Independent study of current technology is expected. Interact with system engineers to define continuous delivery and/or DevOps solutions, data security and/or necessary requirements for automation. Technical Engineering Services: Supports BI Data Warehousing and Enterprise Data Solutions projects by analyzing, designing, and developing ETL solutions; conducting tests and inspections; creating BI reports and virtualization components. Create interface jobs/APIs for data sharing with internal and external stakeholders Ensure we use accurate and secure methods to extract data. Analyze Big Data and MPP Databases to discover trends and patterns. Participates in reviews (walkthroughs) of technical specifications and program code with other members of the DevOps team. Expected to supervise associate engineers on occasion. Innovation: Presents new ideas which improve existing BI Data Warehousing systems/processes/services. Presents new ideas which utilize new frameworks and reusable components to improve existing ETL jobs/processes/services. Express new perspectives based on an independent study of the industry. Review current company processes to highlight questions that may drive process refinement and optimization. Technical Writing : Maintains knowledge of existing technology documents. Writes basic documentation on how technology works. Contributes clear documentation for new code and systems used. Documenting systems designs, presentations, and business requirements for consumption at the engineer level. Develop application support documentation as required by the application support teams for acceptance of systems changes into production. Technical Leadership : Collaborates with other engineering development teams and utilizes data engineering/analyst expertise to deliver technical solutions. Continuously learn and mentor new technologies. Qualifications & Skills Bachelor’s degree in CS, Statistics, Information systems or equivalent experience. Strong expertise working with relational databases. Exceptional ability to author and optimize complex SQL queries/workflows Strong analytical skills to work with unstructured data. Experience building and optimizing highly scalable data pipelines, architectures, and data sets. A successful history of manipulating, processing, and extracting value from large, disconnected datasets and provide valuable insights. Proficiency in workflow orchestration (Databricks, Azure data factory). Experience working with streaming data solutions such as Kafka, IoT hub and Spark Streaming. Working experience building insights , ML models, fraud and anomaly detection using Snowflake/Databricks. Experience working with NoSQL databases like MongoDB, Cassandra, Cosmos DB Proficiency in Java/Python/Scala, pandas, pySpark, NumPy 8+ years of ETL Development experience using Azure Databricks, Azure data factory, SSIS, Talend. 8+ years Professional experience in designing and building cost-effective large scale data marts, data warehousing, distributed big data processing MPP Systems (Databricks, Spark, Snowflake) 5+ years of Expertise in Logical and Physical Data Model design using various modelling Tools like Erwin 7.3 and Power Designer. 5+ years of experience with Azure and AWS cloud stack 3+ years BI Visualization experience developing reports using Snowflake , PowerBI , Grafana , Qlik and analytic cubes utilizing SQL Server Analysis Services (SSAS). Preferred Skills Certifications DevOps experience Bash, Golang scripting experience Docker/Kubernetes experience CI/CD pipelines What We Offer A challenging and rewarding role in a dynamic and international environment. Opportunity to be part of a growing company with a strong commitment to innovation and excellence. A supportive and collaborative team culture that values personal growth and development. Competitive compensation and benefits package. Skills Assessment We want to ensure that all applicants have a fair and equal chance, so we’re using a TestGorilla assessment ( https://www.testgorilla.com ). This minimizes unconscious bias in our hiring process. Successful candidates will be invited to a job interview.Ready to join our team? Start by clicking the link below. https://app.testgorilla.com/s/zuha0j8e Powered by JazzHR

Company:

Streamline Digital

Posted:

May 10 on The Resumator

Visit Our Partner Website

This listing was posted on another website. Click here to open: Go to The Resumator

Important Safety Tips

Always meet the employer in person.
Avoid sharing sensitive personal and financial information.
Avoid employment offers that require a deposit or investment.

To learn more, visit the Safety Center or click here to report this listing.

Senior Data Engineer-ADF, Databricks, Snowflake, 10+years | Engineer in Engineering Job at Streaml1

Senior Data Engineer-ADF, Databricks, Snowflake, 10+years

Visit Our Partner Website