The AI/ML Marketplace is a software library embedded within the Dashboard, offering a curated collection of machine learning algorithms and models for users to access and deploy. It supports seamless integration of AI and ML capabilities, enabling users to address specific data-driven challenges effectively.
Technical Features and Technologies
The marketplace hosts a repository of pre-trained machine learning algorithms covering a wide range of tasks. New algorithms, such as Data Fusion and others developed by project partners, will be regularly added. Each model comes with detailed information provided by its developer, including its purpose, accuracy metrics, input requirements, and output formats, enabling users to make informed decisions.
Users can select the algorithm that best suits their needs and incorporate it into their workflows. Personalisation features allow the system to recommend algorithms based on the user’s previous selections and feedback, with a history of events and actions maintained for convenience. Version control ensures that users can choose specific algorithm versions and stay informed about updates and improvements.
On the technical side, the marketplace’s backend is developed using Python, ensuring robustness and scalability. Metadata, user preferences, and versioning information are stored in a dedicated database. APIs facilitate communication with the machine learning algorithms and enable retrieval of relevant data. Algorithms deployed on the HES-SO inference server are wrapped in a standard format for smooth integration.
The user interface provides an intuitive dashboard where users can browse, search, and select ML algorithms with ease. Additionally, a visual workflow editor enables the design and execution of complex workflows by combining multiple algorithms through a drag-and-drop interface.
To assist users, each AI/ML model and workflow is represented with clear icons and concise descriptions, alongside defined input and output data formats to ensure compatibility. Any limitations or prerequisites are also highlighted, along with recommendations for optimal use. Predefined workflows are reusable, increasing efficiency for end-users.
Workflow and Data Integration
To maintain consistency, all AI/ML algorithms and workflows follow strict input standards, allowing smooth data processing. Data pre-processing may be necessary before entering fusion or ML pipeline phases. Users can download workflow files published by others from S3 buckets, with detailed metadata accompanying each workflow.
This seamless integration simplifies access to machine learning tools and enhances data processing and visualisation capabilities within the platform.
Background
The AI/ML Marketplace builds on the latest advances in AI and machine learning integration, data access, and user interface design. It leverages microservice-based architecture and modular APIs to ensure scalability, flexibility, and secure management of individual services. This approach enables dedicated handling of specific tasks, facilitating easier deployment and maintenance.
Integration with PostgreSQL databases and the workflow editor provides users with direct access to a rich set of AI/ML functionalities within the dashboard. Security measures, including OAuth 2.0, JSON Web Tokens (JWT), and HTTPS, safeguard data communication and API endpoints.
Key software components integrated into the AI/ML Marketplace include user-friendly workflow editors for designing data pipelines, data visualisation tools to present processed Earth Observation data effectively, and robust user management systems to control access rights.
Technologies and Frameworks Used
The development relies on modern technologies including ReactJS for the frontend, C# and ASP.NET Core for backend services, and Kafka for messaging and data streaming.
Input and Output
The system accepts YAML files representing workflows, AI/ML algorithms, and their configurations as input. The output consists of similarly structured YAML files for use by the workflow editor.
Figure 1: AI/ML Marketplace Communication and Integration – System Architecture