Overview

This intensive summer school offers a unique opportunity to delve into the cutting-edge intersection of Artificial Intelligence (AI) and Earth Observation (EO), combined with the essential skills for managing large-scale data and workflows in modern computing environments. Participants will gain theoretical knowledge and practical experience in applying AI techniques to analyze EO data, optimizing AI performance, managing complex workflows with Kubernetes, and handling massive datasets. The program is designed for researchers, professionals, and students eager to explore the intersection of AI and environmental science.

DaFab workshop will be held at High Performance, Edge And Cloud computing conference HiPEAC 2025 in Barcelona. Project members will showcase project plans and technologies and present an on-line demonstration of existing prototype. The workshop will be held on January 20th in Barcelona.

Introduction

In the realm of both scientific research and production environments, efficiently managing and utilizing metadata is crucial. Metadata serves as the backbone for data discovery, organization, and retrieval, enabling effective data usage across various fields. This is particularly important in areas like Earth Observation (EO), where vast amounts of satellite data need to be processed and analysed to monitor and understand our planet.

The DaFab project, an ambitious initiative, aims to enhance the exploitation of Copernicus data through advanced AI and High-Performance Computing (HPC) technologies. By integrating these technologies, DaFab seeks to improve the timeliness, accuracy, and accessibility of EO data. At the heart of this endeavour lies Rucio, a robust data management system developed by CERN. Rucio’s role is pivotal in achieving key objectives of the project such as creating a unified, searchable catalogue of interlinked EO metadata, improving metadata ingestion and retrieval speeds, and facilitating seamless integration with AI-driven workflows and HPC systems.

Multi-Site Workflow Orchestration in the DAFAB Project

DAFAB will design and implement a workflow orchestration system that enhances multi-site application deployment and data discovery. The workflow system will enable applications to express their computations as a graph and declare the data needed at each step in a high-level query language. Workflows will then execute across multiple sites, whether cloud-based (Kubernetes) or high-performance computing (Slurm) environments. By enabling transparent data access and seamless execution of workflow stages, this system shifts the burden of application deployment and data discovery to the platform itself, significantly accelerating development timelines.