ai.deesup.com

This is a task queue based Machine Learning application that aims to provide data predictions based on Deesup datasets.

ai.deesup.com service ports:

• 5672: rabbitmq service
• 15672: rabbitmq service
• 15673: rabbitmq service
• 5555: flower service
• 9090: prometheus service
• 3000: grafana service
• 8000: flask service

Celery

Celery is a task queue with batteries included. It’s easy to use so that you can get started without learning the full complexities of the problem it solves.
There's a worker that executes tasks, and there are clients that submit tasks to the worker.
Tasks may also be submitted using REST API provided by Flower.

srcelery/celery_worker.py => entrypoint
srcelery/tasks => tasks definitions
srcelery/client => celery client scripts to perform tasks
datasets => datasets loaded by google_storage.fetch_latest_dataset, make sure volume in local has open permissions

Celery uses Pickle serializer instead the default JSON serializer because it supports pandas dataframes.
Note the Pickle serializer deals with untrusted data, so you should never accept serialized data from untrusted sources or deserialize it with pickle.loads() without using the safe kwarg.

In order to run a celery client is necessary to run the module with -m flag or the module wouldn't be able to use imports.

• python -m srcelery.client. (ex. python3 -m srcelery.client.pipeline)

RabbitMQ

RabbitMQ is a message broker: it accepts and forwards messages. RabbitMQ makes Celery able to operate at large scale and high throughput.
It exposes a REST API for management/monitoring and a management UI.
In dev environment, the management UI is available at http://localhost:15672/ with guest/guest credentials.

PORTS:

• 15672: management UI => HTTP GET :15672
• 5672: AMQP port => amqp://:@:5672/
• 5673: backend port => rpc://:@:5673/

API:

• curl -u guest:guest http://localhost:15672/api/whoami => { "name":"guest", "tags":"administrator" }

Flower

Flower is a web based tool for monitoring and administrating Celery clusters.

Flower API documentation is available at /api/ endpoint:

• POST /api/task/async-apply/<task_name>/ => works only if celery_worker is inside flower container, otherwise use send-task
• POST /api/task/send-task/<task_name>/
• GET /api/tasks

Prometheus

Prometheus is a monitoring system and time series database. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.
Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets the user select and aggregate time series data in real time.

PORTS:

• 9090: Prometheus UI => HTTP GET :9090

Grafana

Gradana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.
Grafana supports querying Prometheus and makes it easy to create and edit dashboards, query data, create alerts and share dashboards with other users.
In dev environment, the management UI is available at http://localhost:3000/ with admin/admin credentials.

PORTS:

• 3000: Grafana UI => HTTP GET :3000

Deesup data

Deesup Vendure service exposes a GraphQL API exportProducts that builds and uploads an xlsx export to Cloud Storage.
ai.deesup.com is able to retrieve the latest dataset from Cloud Storage and to load it.

Products dataframe structures:

• id: number
• name: string | null
• slug: string | null
• description: string | null
• category.alpha: string | null
• category.beta: string | null
• category.gamma: string | null
• category.delta: string | null
• views.count: number | null
• wishlist.count: number | null
• timestamp.created: number | Date | null
• price: number | null
• price.original: number | null
• 'price.starting': number | null
• 'price.limit': number | null
• 'sold.individually': Binary
• 'items.per.stock'?: number | null
• 'quantity': number | null
• 'seller.type': 'private' | 'professional' | null
• weight: number | null
• height: number | null
• width: number | null
• depth: number | null
• heighSeat: number | null
• 'lamp.included': Binary
• 'out.of.production': Binary
• designer: string | null
• material: string | null
• brand: string | null
• color: string | null
• style: string | null
• conditions: string | null

Flask

Flask web interface is exposed on port 8000 and is the web entrypoint of the application.
It's possible to run Flask outside Docker using command:

python3 flask/app.py