portfolio.rottigni.tech

At midnight every day, my portfolio SQLite database is automatically updated through a dedicated Docker service on my VPS.

This service runs a Node.js cron job that composes and upserts data from various sources, including Vimeo, GitHub, GitLab APIs, puppeteer web scraping, and static sources hosted onhttps://storage.rottigni.tech/.

The updates are applied to the SQLite database using@prisma/clientand are reflected on both the VPS file system and the Nuxt3 container, which stay up-to-date through Docker volumes.

The seeder container generates logs during the seeding process, including logs.log, errors.log, and debug.log, which are sent to my private storage athttps://s3.rottigni.tech/portfolio.rottigni.tech/seeding/logs/on each run.

The container also creates a backup of the SQLite3 database before each seed operation, stored athttps://s3.rottigni.tech/portfolio.rottigni.tech/seeding/backup/,, to ensure data safety.

One notable feature of the seeder is its ability to use my private storage as a cached data provider.

When the seeder requests data from an original provider, it uploads a new JSON API response file to my storage containing in its filename the UTC timestamp of the fetch

When caching is enabled, the seeder reads the latest file from storage instead of requesting data from the original provider.

All of this is orchestrated through a seed.config.ts file located in the root of the seeding directory.

This file specifies which entities and relations should be seeded, what should be cached, and paths of where data should be uploaded.

Overall, this system provides automated updates to my portfolio SQLite database while ensuring data safety and minimizing requests to original providers through the use of cached data.