Senior Cloud Developer/Data Engineer

We are looking for an experienced and innovative Senior Cloud Developer/Data Engineer to join our team in Hyderabad. The ideal candidate will have a strong background in designing and developing data pipelines using Scala or Python, with significant experience in AWS technologies. You will be responsible for enhancing large-scale applications, creating robust data processing pipelines, and solving complex data integrations across multiple systems. Your ability to provide technical guidance, collaborate with business units, and drive real-time data analysis strategies will be crucial in delivering high-quality solutions. If you are passionate about leveraging cutting-edge technologies and enjoy working in a dynamic and fast-paced environment, we want to hear from you.

• Opportunity to work with cutting-edge technologies and innovative solutions.
• Collaborate with a dynamic team in a fast-paced environment.
• Engage with top-tier clients and contribute to impactful projects.
• Competitive compensation and benefits package.

• Collaborate as part of a development team to design and enhance large-scale applications using Python and Spark.
• Evaluate and plan software designs, test results, and technical manuals using AWS.
• Confer with business units and development staff to understand both business and technical requirements for producing technical solutions.
• Create and review technical and user-focused documentation for data solutions (data models, data dictionaries, business glossaries, process and data flows, architecture diagrams, etc.).
• Extend and enhance the business Data Lake. Create or implement solutions for metadata management.
• Solve complex data integrations across multiple systems.
• Design and execute strategies for real-time data analysis and decision-making.
• Build robust data processing pipelines using AWS Services and integrate with multiple data sources.
• Translate client user requirements into data flows and data mappings.
• Analyze and determine data integration needs and follow Agile practices.

• Collaborate as part of a development team to design and enhance large-scale applications using Python and Spark.
• Evaluate and plan software designs, test results, and technical manuals using AWS.
• Confer with business units and development staff to understand both business and technical requirements for producing technical solutions.
• Create and review technical and user-focused documentation for data solutions (data models, data dictionaries, business glossaries, process and data flows, architecture diagrams, etc.).
• Extend and enhance the business Data Lake. Create or implement solutions for metadata management.
• Solve complex data integrations across multiple systems.
• Design and execute strategies for real-time data analysis and decision-making.
• Build robust data processing pipelines using AWS Services and integrate with multiple data sources.
• Translate client user requirements into data flows and data mappings.
• Analyze and determine data integration needs and follow Agile practices.

• At least 5+ years of experience in designing and developing Data Pipelines for Data Ingestion or Transformation using Scala or Python.
• At least 4 years of experience with Python and Spark.
• At least 3 years of experience working with AWS technologies.
• Experience in designing, building, and deploying production-level data pipelines using tools from AWS Glue, Lambda, Kinesis, and databases like Aurora and Redshift.
• Experience with Spark programming (Pyspark or Scala).
• Hands-on experience with AWS components like EMR, S3, Redshift, Lambda, API Gateway, and Kinesis in production environments.
• Strong analytical skills and advanced SQL knowledge, including indexing and query optimization techniques.
• Experience using ETL tools for data ingestion.
• Experience with Change Data Capture (CDC) technologies and relational databases such as MS SQL, Oracle, and DB.
• Ability to translate data needs into detailed functional and technical designs for development, testing, and implementation.
• 5+ years of Development / Data Engineering Experience.
• 4+ years developing AWS Cloud solutions.
• 4+ years developing with Spark/Pyspark.
• 4+ years developing AWS Glue ETL.
• 4+ years AWS Storage models (S3 and DynamoDB).
• Some hands-on experience with on-prem ETL tooling (Ab Initio preferred, Informatica).

• Strong experience in AWS Glue, EMR, and Hudi, including extracting data from multiple sources and loading data into Data Lakes AWS Redshift.
• Experience working with AWS Elasticsearch, RDS, and PostgreSQL preferred.
• Experience with AWS services such as Lambda, EMR, SNS/SQS, EventBridge, Lake Formation, and Athena.
• Experience integrating applications/systems (data producers) with Enterprise Kafka topics (Confluent Kafka integration with AWS S3, Redshift).
• Experience in Requirements Analysis, Data Analysis, Application Design, Application Development, and Integration Testing.
• Working knowledge of on-prem Extraction, Transformation, Cleansing, and Loading methodology and principles.
• Experience implementing and adding to DevOps principles (GitLab, maintaining CI/CD pipelines).
• Experience with Service Oriented Architecture (SOA).

• Any Bachelor’s Degree