شرح موقعیت
We are looking for a Data Engineer with a strong software engineering mindset who can combine data engineering expertise with software development practices to solve complex data and software challenges. You will work closely with Data and Software Engineering teams and technical squads, contributing to data platform initiatives and product development while building reliable, scalable, and data-driven solutions.
مسئولیتها
- Design, develop, and maintain scalable, reliable, and maintainable data pipelines across batch, incremental, CDC, streaming, and event-driven workloads, ensuring reliable data ingestion, processing, and delivery.
- Build data-intensive applications and services and contribute to software engineering initiatives within the data platform ecosystem, following sound software architecture and API design practices.
- Design, maintain, and optimize operational and analytical data stores and data models that support analytical, reporting, and product-facing use cases, with a focus on query performance, indexing, and scalability.
- Establish and improve data quality, monitoring, and observability practices across data pipelines and platforms; proactively identify and troubleshoot data consistency, availability, reliability, and performance issues.
- Contribute to the design and evolution of analytical data architectures, including OLAP systems, Data Warehouses, Data Lakes, and Lakehouse architectures.
- Enable self-service analytics for analysts and product teams, and contribute to the development, customization, and integration of Apache Superset and related analytical tooling.
- Collaborate closely with Data Engineering, Software Engineering, Analytics, and Product teams and contribute to the continuous evolution of a scalable, reliable, secure, and maintainable Data Platform.
نیازمندیها
- At least 3 years of hands-on professional experience in Data Engineering, designing and operating production data pipelines and platforms.
- Strong proficiency in SQL, with hands-on experience working with relational and analytical databases, particularly PostgreSQL and ClickHouse, including query optimization, indexing, and performance tuning.
- Proficiency in Python, with experience writing clean, maintainable, testable, and production-ready code.
- Hands-on experience designing, developing, and maintaining scalable data pipelines using data integration and flow management tools such as Apache NiFi, with a solid understanding of batch, incremental, and data ingestion patterns.
- Solid understanding of data modeling for operational and analytical workloads, including dimensional modeling and data structures designed for analytical use cases.
- Solid understanding of software engineering fundamentals, including software architecture, API design and development, testing, version control, and maintainable coding practices.
- Hands-on experience with Apache Kafka and a solid understanding of event-driven architectures, messaging concepts, and distributed data workflows.
- Solid understanding of Data Warehousing and OLAP concepts, with familiarity with Data Lake and Lakehouse architectures.
- Experience working with Linux environments and Git, along with common development, debugging, and troubleshooting workflows.
- Practical experience using AI-assisted development tools in day-to-day engineering workflows for coding, debugging, testing, and documentation, while critically evaluating and validating AI-generated outputs.
- Strong problem-solving and troubleshooting skills, with the ability to investigate and resolve issues across data pipelines, databases, applications, and data platforms.
- Ability to collaborate effectively with Software Engineering, Analytics, and Product teams.
- Experience working with NoSQL databases, particularly MongoDB.
- Experience with distributed storage systems such as HDFS and familiarity with distributed storage concepts.
- Experience with Apache Spark for large-scale batch or stream data processing using PySpark or Scala.
- Experience with Apache Superset, including self-service analytics, dataset management, customization, integration, or development.
- Familiarity with containerization and orchestration technologies such as Docker and Kubernetes.
- Experience designing and building AI agents, AI-powered automation, or LLM-based tools for engineering, data, or internal workflow use cases.