شرح موقعیت
We are looking for a DataOps/Data Engineer to design, build, and maintain data pipelines and the infrastructure that powers them. You will work hands-on with distributed data systems such as Qdrant, OpenSearch, Elasticsearch, Apache Flink, Apache Spark, and Apache Kafka, both in Kubernetes-based and Docker Compose–based environments. This role blends Data Engineering with Platform/Infrastructure Operations (DataOps), so you should be comfortable both writing pipeline code and operating, tuning, and troubleshooting the systems it runs on. Key Responsibilities Design, build, and maintain scalable batch and streaming data pipelines. Develop and optimize stream processing jobs using Apache Flink and Apache Spark. Build and manage data ingestion/integration flows using Apache Kafka (topics, partitions, consumer groups, schema management). Ensure data quality, consistency, and reliability across pipelines. Operate, maintain, and fine-tune core data platforms: Qdrant, OpenSearch, Elasticsearch, Flink, Spark, and Kafka. Own performance tuning, indexing strategy, and configuration optimization for search and vector databases (OpenSearch, Elasticsearch, Qdrant). Manage and maintain a Kafka cluster running on both Kubernetes and Docker Compose environments. Monitor system health, set up alerting, and proactively troubleshoot performance or reliability issues. Fine-tune and optimize data-related workloads running on Kubernetes (resource limits, storage configuration, Helm chart values, etc.), while the DevOps team owns cluster provisioning and lifecycle management. Work closely with DevOps, backend engineers, and other data teams to ensure workloads are deployed, scaled, and configured correctly on shared infrastructure. Troubleshoot data platform–related issues and provide timely resolutions.
نیازمندیها
- Hands-on experience with : Kafka, Flink/Spark, Elasticsearch/OpenSearch, Qdrant (or a similar vector database).
- Solid understanding of distributed systems concepts (partitioning, replication, consistency, scaling).
- Experience with Docker / Docker Compose for service deployment.
- Strong scripting/programming skills (Python and/or Bash); working knowledge of Java/Scala is a plus for Flink/Spark jobs.
- Comfortable with Linux system administration basics and reading logs/metrics to debug production issues.
- Familiarity with monitoring/observability stacks (Prometheus, Grafana, or similar).
- Strong problem-solving skills and the ability to work in a fast-paced environment.
- Experience running workloads on Kubernetes (Deployments, StatefulSets, ConfigMaps, Helm) — you don't need to manage the cluster itself, but you should be comfortable configuring and tuning what runs on it.
- Experience with vector search / embeddings and how they're used in Qdrant or similar systems.
- Familiarity with CI/CD pipelines and infrastructure/config change management.
- Knowledge of Infrastructure as Code (IaC) tools like Terraform or Ansible.
- Experience with open table formats such as Apache Iceberg (or Delta Lake / Hudi).