Senior GCP engineer

Company Description

DataBees is a dynamic company specializing in the data domain, offering services such as cloud and data migration, business intelligence, data engineering and analysis, and data lifecycle management. Focused on data-driven digital transformation, DataBees helps organizations accelerate business outcomes and implement effective data integration strategies. By leveraging advanced methodologies, including generative AI solutions, and emphasizing data quality and governance, DataBees empowers organizations to thrive in a digital-first landscape. Known for innovation and industry expertise, the company partners with clients to drive impactful technology-driven transformations and informed decision-making. DataBees is based in Stockholm and works closely with businesses to achieve their goals.

Role Description

This full-time hybrid role as a Senior GCP Engineer is based in Stockholm, with some flexibility for remote work. The Senior GCP Engineer will design, develop, and manage cloud-based solutions on Google Cloud Platform (GCP) to support clients' data management and processing needs. Responsibilities include implementing best practices in data integration, ensuring high-performance cloud architectures, and contributing to the development of scalable and secure systems. The role also requires collaborating with cross-functional teams to optimize infrastructure, conducting system performance monitoring, and troubleshooting issues to ensure seamless operations.

Qualifications

• Proficiency in GCP services, including Compute Engine, BigQuery, Cloud Storage, and Cloud Functions
• Hands-on experience with Data Engineering, SQL, Kubernetes, and CI/CD pipelines
• Strong understanding of system architecture, cloud security principles, and automation tools
• Experience in data migration, data quality management, and governance
• Demonstrated ability to collaborate with cross-functional teams and stakeholders
• Problem-solving mindset with experience in troubleshooting and optimizing performance
• Bachelor's degree in Computer Science, Engineering, or a related field; certifications in GCP are a plus
• Experience in enterprise-level data transformation projects and working knowledge of Business Intelligence tools is a bonus

Engineering & Dataflow Responsibilities:

• Advanced Dataflow Pipeline Engineering:
  Architect and develop production-grade
  Apache Beam
  pipelines (using Scala for high-performance batch and Python for agile streaming) on
  GCP Dataflow
  .
• Performance Tuning & Optimization:
  Deep-dive into Dataflow execution metrics to optimize
  Shuffle Service
  usage and
  Streaming Engine
  performance. Conduct hands-on tuning of worker machine types (vCPU/memory), disk throughput, and autoscaling thresholds to manage INGKA's global data peaks cost-effectively.
• Complex Windowing & Watermarking:
  Implement advanced streaming logic in Dataflow (Sliding/Session windows, Triggers, and late-data handling) to ensure real-time event signals from
  GCP Pub/Sub
  are accurately sequenced before hitting AEP.
• XDM Transformation Logic:
  Build robust, type-safe Scala transformation layers within Dataflow to convert raw BigQuery/Avro/JSON schemas into complex
  Adobe XDM
  structures, ensuring data integrity at the source.
• Airflow-Driven Orchestration:
  Design and maintain
  Apache Airflow (Cloud Composer)
  DAGs to manage the full lifecycle of Dataflow jobs. Develop custom Airflow operators to trigger, monitor, and auto-restart Dataflow pipelines based on data availability or failure states.
• Integration with GCP/AEP Ecosystem:
  Optimize the "Last Mile" of ingestion by tuning the connection between Dataflow and AEP APIs, managing back-pressure, retries, and error-handling (Dead Letter Queues).

Required Technical Stack:

• Dataflow Expert:
  5+ years of hands-on experience with
  Apache Beam
  (Dataflow) including both Batch and Streaming execution models.
• Languages:
  Expert-level
  Scala
  (functional programming) and
  Python
  .
• Orchestration:
  Advanced
  Apache Airflow
  (DAG design, Custom Operators, XComs).
• GCP Core:
  BigQuery (Storage Write API), Pub/Sub, and Cloud IAM.
• AEP Knowledge:
  Experience with AEP Ingestion APIs and XDM Schema Registry.
• Overall Pipeline Cost optimization