move image storage to s3/local s3

add nodejs version check
update packages
2022-10-02 17:29:43 +02:00 · 2022-10-02 14:08:20 +02:00 · 2022-09-26 17:10:02 +02:00 · 2022-09-24 12:23:59 +02:00 · 2022-09-24 12:23:56 +02:00 · 2022-09-23 22:23:07 +02:00
243 changed files with 6944 additions and 7714 deletions
--- a/.github/FUNDING.yml
+++ b/.github/FUNDING.yml
@ -1,3 +1,3 @@
 # These are supported funding model platforms

-github: caramelfur
+github: rubikscraft
--- a/.nvmrc
+++ b/.nvmrc
@ -1 +1 @@
-v20
+v18.8
--- a/.vscode/settings.json
+++ b/.vscode/settings.json
@ -1,5 +1,14 @@
 {
  "vsicons.presets.angular": true,
  "angular.log": "verbose",
-  "discord.enabled": true
+  "discord.enabled": true,
+  "files.exclude": {
+    "**/.git": true,
+    "**/.svn": true,
+    "**/.hg": true,
+    "**/CVS": true,
+    "**/.DS_Store": true,
+    "**/Thumbs.db": true,
+    "**/node_modules": true
+  }
 }
--- a/.vscode/tasks.json
+++ b/.vscode/tasks.json
@ -55,7 +55,7 @@
    {
      "type": "shell",
      "label": "Start postgres",
-      "command": "yarn devdb:start",
+      "command": "yarn devdb:up",
      "options": {
        "cwd": "${cwd}",
        "shell": {
--- a/.yarn/versions/6db1bf03.yml
+++ b/.yarn/versions/6db1bf03.yml
@ -1,4 +0,0 @@
-undecided:
-  - root-workspace-0b6124
-  - picsur-backend
-  - picsur-frontend
--- a/.yarn/versions/9e39fdb4.yml
+++ b/.yarn/versions/9e39fdb4.yml
@ -1,4 +0,0 @@
-undecided:
-  - root-workspace-0b6124
-  - picsur-backend
-  - picsur-frontend
--- a/.yarn/versions/e0bbb8ad.yml
+++ b/.yarn/versions/e0bbb8ad.yml
@ -1,4 +0,0 @@
-undecided:
-  - root-workspace-0b6124
-  - picsur-backend
-  - picsur-frontend
--- a/2
+++ b/2
@ -630,7 +630,7 @@ state the exclusion of warranty; and each file should have at least
 the "copyright" line and a pointer to where the full notice is found.

    Picsur - An easy to use, selfhostable image sharing service like Imgur
-    Copyright (C) 2022  Caramel <picsur@caramelfur.dev>
+    Copyright (C) 2022  Rubikscraft <contact@rubikscraft.nl>

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU Affero General Public License as published
--- a/README.md
+++ b/README.md
@ -26,9 +26,9 @@ The images are deleted every five minutes, and the maximum filesize is 16MB. But
 ## Features

 Here is a list of done features, and what is planned.
-For a more detailed list, you can always visit [the project](https://github.com/CaramelFur/Picsur/projects/1).
+For a more detailed list, you can always visit [the project](https://github.com/rubikscraft/Picsur/projects/1).

-Every featured marked here should work in the latest release.
+Every feature marked here should work in the latest release.

 - [x] Uploading and viewing images
 - [x] Anonymous uploads
@ -63,11 +63,10 @@ Every featured marked here should work in the latest release.

 - [ ] White mode
 - [ ] Public gallery
- [ ] Albums

 ## Bugs

-If you encounter any bugs or oddities, please open an issue [here](https://github.com/CaramelFur/Picsur/issues). Cause without feedback I'll never know they exists.
+If you encounter any bugs or oddities, please open an issue [here](https://github.com/rubikscraft/Picsur/issues). Cause without feedback I'll never know they exists.

 ## Star

@ -75,13 +74,7 @@ If you like this project, don't forget to give it a star. It tells me that I'm n

 ## Faq

-### Is this project maintained?
-
-Yes it still is. If I were to stop maintaining it, I would archive the repository.
-
-However I do not have a lot of time on my hands, so updates are not always as frequent as I would like them to be.
-
-### Why do my images dissapear of the public instance?
+### Why do my images disappear off the public instance?

 The public instance is only a demo, and therefore only keeps images for 5 minutes. This is to prevent the server from running out of disk space, and to prevent people from using it to host questionable images.

@ -101,16 +94,12 @@ If you want to enable this however, you can do so by going to `settings -> gener

 Do keep in mind here, that the exif data will NOT be removed from the original image. So make sure you do not accidentally share sensitive data.

-### This service says its supports the QOI format, what is this?
+### This service says it supports the QOI format, what is it?

-QOI is a new lossless image format that is designed to be very fast to encode and decode. All while still offering good compression ratios. This is the primary format the server will store images in when uploaded.
+QOI is a new lossless image format that is designed to be very fast to encode and decode. All the while still offering good compression ratios. This is the primary format the server will store images in when uploaded.

 You can [read more about QOI here](https://qoiformat.org/).

-### What is the default admin login?
-
-The default username is `admin`, and the default password is set from the `PICSUR_ADMIN_PASSWORD` environment variable.
-
 ## Running your own instance

 You easily run this service yourself via Docker. Here is an example docker-compose file:
@ -119,7 +108,7 @@ You easily run this service yourself via Docker. Here is an example docker-compo
 version: '3'
 services:
  picsur:
-    image: ghcr.io/caramelfur/picsur:latest
+    image: ghcr.io/rubikscraft/picsur:latest
    container_name: picsur
    ports:
      - '8080:8080'
@ -133,7 +122,6 @@ services:
      # PICSUR_DB_PASSWORD: picsur
      # PICSUR_DB_DATABASE: picsur

-      ## The default username is admin, this is not modifyable
      # PICSUR_ADMIN_PASSWORD: picsur

      ## Optional, random secret will be generated if not set
@ -164,13 +152,12 @@ volumes:

 ## Thanks

- @chennin for monthly donating 4$
 - @awg13 for donating 5$

 ## Api

 Here is a usually up to date documentation of the api:

-[![Run in Postman](https://run.pstmn.io/button.svg)](https://www.postman.com/caramel-team/workspace/picsur/collection/1841871-78e559b6-4f39-4092-87c3-92fa29547d03)
+[![Run in Postman](https://run.pstmn.io/button.svg)](https://www.postman.com/rubikscraft-team/workspace/picsur/collection/1841871-78e559b6-4f39-4092-87c3-92fa29547d03)

 If you wish to build your own frontend or app for picsur, this will surely come in handy. Also take a look at the `./shared` folder in the source code, as it contains typescript schema definitions for the api.
--- a/backend/.eslintrc.cjs
+++ b/backend/.eslintrc.cjs
@ -1,9 +0,0 @@
-module.exports = {
-  parserOptions: {
-    project: './tsconfig.json',
-    tsconfigRootDir: __dirname,
-    sourceType: 'module',
-  },
-  extends: ['../.eslintrc.cjs'],
-  root: false,
-};
--- a/backend/.eslintrc.js
+++ b/backend/.eslintrc.js
@ -1,22 +1,24 @@
 module.exports = {
  parser: '@typescript-eslint/parser',
-
+  parserOptions: {
+    project: 'tsconfig.json',
+    sourceType: 'module',
+  },
  plugins: ['@typescript-eslint/eslint-plugin'],
  extends: [
    'plugin:@typescript-eslint/recommended',
    'plugin:prettier/recommended',
  ],
+  root: true,
  env: {
    node: true,
    jest: true,
  },
-  ignorePatterns: ['.eslintrc.cjs', 'dist', '*.exclude.*'],
+  ignorePatterns: ['.eslintrc.js'],
  rules: {
    '@typescript-eslint/interface-name-prefix': 'off',
    '@typescript-eslint/explicit-function-return-type': 'off',
    '@typescript-eslint/explicit-module-boundary-types': 'off',
    '@typescript-eslint/no-explicit-any': 'off',
-    '@typescript-eslint/ban-ts-comment': 'off',
  },
-  root: true,
 };
--- a/backend/package.json
+++ b/backend/package.json
@ -1,85 +1,101 @@
 {
  "name": "picsur-backend",
-  "version": "0.5.2",
+  "version": "0.4.0",
  "description": "Backend for Picsur",
  "license": "GPL-3.0",
-  "repository": "https://github.com/caramelfur/Picsur",
-  "author": "Caramel <picsur@caramelfur.dev>",
+  "repository": "https://github.com/rubikscraft/Picsur",
+  "author": "Rubikscraft <contact@rubikscraft.nl>",
  "type": "module",
  "main": "dist/main.js",
  "scripts": {
    "prebuild": "rimraf dist",
    "build": "nest build",
-    "start": "nest start --exec \"node --experimental-loader=extensionless\"",
-    "start:dev": "yarn clean && nest start --watch --exec \"node --experimental-loader=extensionless\"",
-    "start:debug": "nest start --debug --watch --exec \"node --experimental-loader=extensionless\"",
-    "start:prod": "node --experimental-loader=extensionless dist/main",
+    "start": "nest start --exec \"node --es-module-specifier-resolution=node\"",
+    "start:dev": "yarn clean && nest start --watch --exec \"node --es-module-specifier-resolution=node\"",
+    "start:debug": "yarn clean && nest start --debug --watch --exec \"node --es-module-specifier-resolution=node\"",
+    "start:prod": "node --es-module-specifier-resolution=node dist/main",
    "typeorm": "typeorm-ts-node-esm",
    "migrate": "PICSUR_PRODUCTION=\"true\" yarn typeorm migration:generate -d ./src/datasource.ts",
    "format": "prettier --write \"src/**/*.ts\"",
    "clean": "rimraf dist",
+    "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
    "purge": "rm -rf dist && rm -rf node_modules"
  },
  "dependencies": {
-    "@fastify/helmet": "^10.1.1",
-    "@fastify/multipart": "^7.6.1",
-    "@fastify/reply-from": "^9.3.0",
-    "@fastify/static": "^6.10.2",
-    "@nestjs/common": "^10.0.0",
-    "@nestjs/config": "^2.3.4",
-    "@nestjs/core": "^10.0.0",
-    "@nestjs/jwt": "^10.1.0",
-    "@nestjs/passport": "^9.0.3",
-    "@nestjs/platform-fastify": "^10.0.0",
-    "@nestjs/schedule": "^3.0.0",
-    "@nestjs/serve-static": "^4.0.0",
-    "@nestjs/throttler": "^4.0.0",
+    "@aws-sdk/client-s3": "^3.181.0",
+    "@fastify/helmet": "^10.0.1",
+    "@fastify/multipart": "^7.2.0",
+    "@fastify/reply-from": "^8.3.0",
+    "@fastify/static": "^6.5.0",
+    "@nestjs/bull": "^0.6.1",
+    "@nestjs/common": "^9.1.2",
+    "@nestjs/config": "^2.2.0",
+    "@nestjs/core": "^9.1.2",
+    "@nestjs/jwt": "^9.0.0",
+    "@nestjs/passport": "^9.0.0",
+    "@nestjs/platform-fastify": "^9.1.2",
+    "@nestjs/platform-socket.io": "^9.1.2",
+    "@nestjs/schedule": "^2.1.0",
+    "@nestjs/serve-static": "^3.0.0",
+    "@nestjs/throttler": "^3.0.0",
    "@nestjs/typeorm": "^9.0.1",
-    "bcrypt": "^5.1.0",
+    "@nestjs/websockets": "^9.1.2",
+    "bcrypt": "^5.0.1",
    "bmp-img": "^1.2.1",
+    "bull": "^4.10.0",
    "cors": "^2.8.5",
-    "extensionless": "^1.4.5",
-    "file-type": "^18.5.0",
+    "file-type": "^18.0.0",
+    "get-stream": "^6.0.1",
    "is-docker": "^3.0.0",
-    "ms": "2.1.3",
-    "node-fetch": "^3.3.1",
-    "p-timeout": "^6.1.2",
+    "ms": "^2.1.3",
+    "node-fetch": "^3.2.10",
+    "p-timeout": "^6.0.0",
    "passport": "^0.6.0",
    "passport-headerapikey": "^1.2.2",
-    "passport-jwt": "^4.0.1",
+    "passport-jwt": "^4.0.0",
    "passport-local": "^1.0.0",
    "passport-strategy": "^1.0.0",
-    "pg": "^8.11.0",
+    "pg": "^8.8.0",
    "picsur-shared": "*",
    "posix.js": "^0.1.1",
-    "qoi-img": "^2.1.1",
+    "qoi-img": "^2.1.0",
    "reflect-metadata": "^0.1.13",
-    "rimraf": "^5.0.1",
-    "rxjs": "^7.8.1",
-    "sharp": "^0.32.1",
+    "rimraf": "^3.0.2",
+    "rxjs": "^7.5.7",
+    "semver": "^7.3.7",
+    "sharp": "^0.31.1",
    "stream-parser": "^0.3.1",
    "thunks": "^4.9.6",
-    "typeorm": "0.3.16",
-    "zod": "^3.21.4"
+    "typeorm": "0.3.10",
+    "uuid": "^9.0.0",
+    "zod": "^3.19.1"
  },
  "devDependencies": {
-    "@nestjs/cli": "^10.0.1",
-    "@nestjs/schematics": "^10.0.1",
-    "@nestjs/testing": "^10.0.0",
+    "@nestjs/cli": "^9.1.4",
+    "@nestjs/schematics": "^9.0.3",
+    "@nestjs/testing": "^9.1.2",
    "@types/bcrypt": "^5.0.0",
-    "@types/cors": "^2.8.13",
+    "@types/bull": "^3.15.9",
+    "@types/cors": "^2.8.12",
    "@types/multer": "^1.4.7",
-    "@types/node": "^20.3.1",
-    "@types/passport-jwt": "^3.0.8",
-    "@types/passport-local": "^1.0.35",
+    "@types/node": "^18.7.23",
+    "@types/passport-jwt": "^3.0.7",
+    "@types/passport-local": "^1.0.34",
    "@types/passport-strategy": "^0.2.35",
-    "@types/sharp": "^0.32.0",
+    "@types/semver": "^7.3.12",
+    "@types/sharp": "^0.31.0",
    "@types/supertest": "^2.0.12",
-    "prettier": "^2.8.8",
+    "@types/uuid": "^8.3.4",
+    "@typescript-eslint/eslint-plugin": "^5.38.1",
+    "@typescript-eslint/parser": "^5.38.1",
+    "eslint": "^8.24.0",
+    "eslint-config-prettier": "^8.5.0",
+    "eslint-plugin-prettier": "^4.2.1",
+    "prettier": "^2.7.1",
    "source-map-support": "^0.5.21",
-    "ts-loader": "^9.4.3",
+    "ts-loader": "^9.4.1",
    "ts-node": "^10.9.1",
-    "tsconfig-paths": "^4.2.0",
-    "typescript": "^5.1.3"
+    "tsconfig-paths": "^4.1.0",
+    "typescript": "4.8.4"
  }
 }
--- a/backend/src/app.module.ts
+++ b/backend/src/app.module.ts
@ -1,8 +1,17 @@
-import { MiddlewareConsumer, Module, NestModule } from '@nestjs/common';
+import { BullModule } from '@nestjs/bull';
+import {
+  Logger,
+  MiddlewareConsumer,
+  Module,
+  NestModule,
+  OnModuleInit,
+} from '@nestjs/common';
 import { ScheduleModule } from '@nestjs/schedule';
 import { ServeStaticModule } from '@nestjs/serve-static';
 import cors from 'cors';
 import { IncomingMessage, ServerResponse } from 'http';
+import semver from 'semver';
+import { BullConfigService } from './config/early/bull.config.service';
 import { EarlyConfigModule } from './config/early/early-config.module';
 import { ServeStaticConfigService } from './config/early/serve-static.config.service';
 import { DatabaseModule } from './database/database.module';
@ -13,6 +22,8 @@ import { DemoManagerModule } from './managers/demo/demo.module';
 import { UsageManagerModule } from './managers/usage/usage.module';
 import { PicsurRoutesModule } from './routes/routes.module';

+const supportedNodeVersions = ['^16.17.0', '^18.6.0'];
+
 const mainCorsConfig = cors({
  origin: '<origin>',
 });
@ -30,7 +41,7 @@ const imageCorsConfig = cors({
 const imageCorsOverride = (
  req: IncomingMessage,
  res: ServerResponse,
-  next: () => void,
+  next: Function,
 ) => {
  res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');

@ -45,6 +56,11 @@ const imageCorsOverride = (
      imports: [EarlyConfigModule],
    }),
    ScheduleModule.forRoot(),
+    BullModule.forRootAsync({
+      useExisting: BullConfigService,
+      imports: [EarlyConfigModule],
+    }),
+
    DatabaseModule,
    AuthManagerModule,
    UsageManagerModule,
@ -53,9 +69,24 @@ const imageCorsOverride = (
    PicsurLayersModule,
  ],
 })
-export class AppModule implements NestModule {
+export class AppModule implements NestModule, OnModuleInit {
+  private readonly logger = new Logger(AppModule.name);
+
  configure(consumer: MiddlewareConsumer) {
    consumer.apply(mainCorsConfig).exclude('/i').forRoutes('/');
    consumer.apply(imageCorsConfig, imageCorsOverride).forRoutes('/i');
  }
+
+  onModuleInit() {
+    const nodeVersion = process.version;
+    if (!supportedNodeVersions.some((v) => semver.satisfies(nodeVersion, v))) {
+      this.logger.error(
+        `Unsupported Node version: ${nodeVersion}. Transcoding performance will be severely degraded.`,
+      );
+
+      this.logger.log(
+        `Supported Node versions: ${supportedNodeVersions.join(', ')}`,
+      );
+    }
+  }
 }
--- a/backend/src/collections/apikey-db/apikey-db.service.ts
+++ b/backend/src/collections/apikey-db/apikey-db.service.ts
@ -1,11 +1,6 @@
 import { Injectable, Logger } from '@nestjs/common';
 import { InjectRepository } from '@nestjs/typeorm';
-import {
-  AsyncFailable,
-  Fail,
-  FT,
-  HasFailed,
-} from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, Fail, FT, HasFailed } from 'picsur-shared/dist/types';
 import { FindResult } from 'picsur-shared/dist/types/find-result';
 import { generateRandomString } from 'picsur-shared/dist/util/random';
 import { Repository } from 'typeorm';
--- a/backend/src/collections/file-s3/file-s3.module.ts
+++ b/backend/src/collections/file-s3/file-s3.module.ts
@ -0,0 +1,10 @@
+import { Module } from '@nestjs/common';
+import { EarlyConfigModule } from '../../config/early/early-config.module';
+import { FileS3Service } from './file-s3.service';
+
+@Module({
+  imports: [EarlyConfigModule],
+  providers: [FileS3Service],
+  exports: [FileS3Service],
+})
+export class FileS3Module {}
--- a/backend/src/collections/file-s3/file-s3.service.ts
+++ b/backend/src/collections/file-s3/file-s3.service.ts
@ -0,0 +1,122 @@
+import {
+  CreateBucketCommand,
+  DeleteObjectCommand,
+  DeleteObjectsCommand,
+  GetObjectCommand,
+  ListBucketsCommand,
+  PutObjectCommand,
+  S3Client,
+} from '@aws-sdk/client-s3';
+import { Injectable, Logger, OnModuleInit } from '@nestjs/common';
+import { buffer as streamToBuffer } from 'get-stream';
+import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types';
+import { Readable } from 'stream';
+import { S3ConfigService } from '../../config/early/s3.config.service';
+
+@Injectable()
+export class FileS3Service implements OnModuleInit {
+  private readonly logger = new Logger(FileS3Service.name);
+
+  private S3: Promise<S3Client> = this.loadS3();
+
+  constructor(private readonly s3config: S3ConfigService) {}
+
+  onModuleInit() {
+    this.loadS3();
+  }
+
+  public async putFile(key: string, data: Buffer): AsyncFailable<string> {
+    const S3 = await this.S3;
+
+    const request = new PutObjectCommand({
+      Bucket: this.s3config.getS3Bucket(),
+      Key: key,
+      Body: data,
+    });
+
+    try {
+      await S3.send(request);
+      return key;
+    } catch (e) {
+      return Fail(FT.Database, e);
+    }
+  }
+
+  public async getFile(key: string): AsyncFailable<Buffer> {
+    const S3 = await this.S3;
+
+    const request = new GetObjectCommand({
+      Bucket: this.s3config.getS3Bucket(),
+      Key: key,
+    });
+
+    try {
+      const result = await S3.send(request);
+      if (!result.Body) return Fail(FT.NotFound, 'File not found');
+
+      if (result.Body instanceof Blob) {
+        return Buffer.from(await result.Body.arrayBuffer());
+      }
+      return streamToBuffer(result.Body as Readable);
+    } catch (e) {
+      return Fail(FT.Database, e);
+    }
+  }
+
+  public async deleteFile(key: string): AsyncFailable<true> {
+    const S3 = await this.S3;
+
+    const request = new DeleteObjectCommand({
+      Bucket: this.s3config.getS3Bucket(),
+      Key: key,
+    });
+
+    try {
+      await S3.send(request);
+      return true;
+    } catch (e) {
+      return Fail(FT.Database, e);
+    }
+  }
+
+  public async deleteFiles(keys: string[]): AsyncFailable<true> {
+    const S3 = await this.S3;
+
+    const request = new DeleteObjectsCommand({
+      Bucket: this.s3config.getS3Bucket(),
+      Delete: {
+        Objects: keys.map((key) => ({ Key: key })),
+      },
+    });
+
+    try {
+      await S3.send(request);
+      return true;
+    } catch (e) {
+      return Fail(FT.Database, e);
+    }
+  }
+
+  private async loadS3(): Promise<S3Client> {
+    const S3 = new S3Client(this.s3config.getS3Config());
+
+    try {
+      // Create bucket if it doesn't exist
+      const bucket = this.s3config.getS3Bucket();
+
+      // List buckets
+      const listBuckets = await S3.send(new ListBucketsCommand({}));
+
+      const bucketExists = listBuckets.Buckets?.some((b) => b.Name === bucket);
+      if (!bucketExists) {
+        this.logger.verbose(`Creating S3 Bucket ${bucket}`);
+        await S3.send(new CreateBucketCommand({ Bucket: bucket }));
+      } else {
+        this.logger.verbose(`Using existing S3 Bucket ${bucket}`);
+      }
+    } catch (e) {
+      this.logger.error(e);
+    }
+    return S3;
+  }
+}
--- a/backend/src/collections/image-db/image-db.module.ts
+++ b/backend/src/collections/image-db/image-db.module.ts
@ -3,6 +3,7 @@ import { TypeOrmModule } from '@nestjs/typeorm';
 import { EImageDerivativeBackend } from '../../database/entities/images/image-derivative.entity';
 import { EImageFileBackend } from '../../database/entities/images/image-file.entity';
 import { EImageBackend } from '../../database/entities/images/image.entity';
+import { FileS3Module } from '../file-s3/file-s3.module';
 import { ImageDBService } from './image-db.service';
 import { ImageFileDBService } from './image-file-db.service';

@ -13,6 +14,7 @@ import { ImageFileDBService } from './image-file-db.service';
      EImageFileBackend,
      EImageDerivativeBackend,
    ]),
+    FileS3Module,
  ],
  providers: [ImageDBService, ImageFileDBService],
  exports: [ImageDBService, ImageFileDBService],
--- a/backend/src/collections/image-db/image-db.service.ts
+++ b/backend/src/collections/image-db/image-db.service.ts
@ -1,6 +1,6 @@
 import { Injectable } from '@nestjs/common';
 import { InjectRepository } from '@nestjs/typeorm';
-import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types';
 import { FindResult } from 'picsur-shared/dist/types/find-result';
 import { generateRandomString } from 'picsur-shared/dist/util/random';
 import { In, LessThan, Repository } from 'typeorm';
--- a/backend/src/collections/image-db/image-file-db.service.ts
+++ b/backend/src/collections/image-db/image-file-db.service.ts
@ -1,44 +1,59 @@
-import { Injectable } from '@nestjs/common';
+import { Injectable, Logger } from '@nestjs/common';
 import { InjectRepository } from '@nestjs/typeorm';
 import { ImageEntryVariant } from 'picsur-shared/dist/dto/image-entry-variant.enum';
-import {
-  AsyncFailable,
-  Fail,
-  FT,
-  HasFailed,
-} from 'picsur-shared/dist/types/failable';
-import { LessThan, Repository } from 'typeorm';
+import { AsyncFailable, Fail, FT, HasFailed } from 'picsur-shared/dist/types';
+import { In, IsNull, LessThan, Repository } from 'typeorm';
+import { v4 as uuidv4 } from 'uuid';
 import { EImageDerivativeBackend } from '../../database/entities/images/image-derivative.entity';
 import { EImageFileBackend } from '../../database/entities/images/image-file.entity';
+import { FileS3Service } from '../file-s3/file-s3.service';

 const A_DAY_IN_SECONDS = 24 * 60 * 60;

@Injectable()
 export class ImageFileDBService {
+  private readonly logger = new Logger(ImageFileDBService.name);
+
  constructor(
    @InjectRepository(EImageFileBackend)
    private readonly imageFileRepo: Repository<EImageFileBackend>,

    @InjectRepository(EImageDerivativeBackend)
    private readonly imageDerivativeRepo: Repository<EImageDerivativeBackend>,
+
+    private readonly s3Service: FileS3Service,
  ) {}

+  public async getData(
+    file: EImageFileBackend | EImageDerivativeBackend,
+  ): AsyncFailable<Buffer> {
+    const result = await this.s3Service.getFile(file.s3key);
+    if (HasFailed(result)) return result;
+
+    return result;
+  }
+
  public async setFile(
    imageId: string,
    variant: ImageEntryVariant,
    file: Buffer,
    filetype: string,
  ): AsyncFailable<true> {
+    const s3key = uuidv4();
+
    const imageFile = new EImageFileBackend();
    imageFile.image_id = imageId;
    imageFile.variant = variant;
    imageFile.filetype = filetype;
-    imageFile.data = file;
+    imageFile.s3key = s3key;

    try {
      await this.imageFileRepo.upsert(imageFile, {
        conflictPaths: ['image_id', 'variant'],
      });
+
+      const s3result = await this.s3Service.putFile(s3key, file);
+      if (HasFailed(s3result)) return s3result;
    } catch (e) {
      return Fail(FT.Database, e);
    }
@ -78,7 +93,7 @@ export class ImageFileDBService {
    }
  }

-  public async deleteFile(
+  public async orphanFile(
    imageId: string,
    variant: ImageEntryVariant,
  ): AsyncFailable<EImageFileBackend> {
@ -89,8 +104,9 @@ export class ImageFileDBService {

      if (!found) return Fail(FT.NotFound, 'Image not found');

-      await this.imageFileRepo.delete({ image_id: imageId, variant: variant });
-      return found;
+      found.image_id = null;
+
+      return await this.imageFileRepo.save(found);
    } catch (e) {
      return Fail(FT.Database, e);
    }
@ -125,15 +141,22 @@ export class ImageFileDBService {
    filetype: string,
    file: Buffer,
  ): AsyncFailable<EImageDerivativeBackend> {
+    const s3key = uuidv4();
+
    const imageDerivative = new EImageDerivativeBackend();
    imageDerivative.image_id = imageId;
    imageDerivative.key = key;
    imageDerivative.filetype = filetype;
-    imageDerivative.data = file;
+    imageDerivative.s3key = s3key;
    imageDerivative.last_read = new Date();

    try {
-      return await this.imageDerivativeRepo.save(imageDerivative);
+      const result = await this.imageDerivativeRepo.save(imageDerivative);
+
+      const s3result = await this.s3Service.putFile(s3key, file);
+      if (HasFailed(s3result)) return s3result;
+
+      return result;
    } catch (e) {
      return Fail(FT.Database, e);
    }
@ -151,10 +174,12 @@ export class ImageFileDBService {
      if (!derivative) return null;

      // Ensure read time updated to within 1 day precision
-      const yesterday = new Date(Date.now() - A_DAY_IN_SECONDS * 1000);
-      if (derivative.last_read > yesterday) {
+      const aMinuteAgo = new Date(Date.now() - 60 * 1000);
+      if (derivative.last_read > aMinuteAgo) {
        derivative.last_read = new Date();
-        return await this.imageDerivativeRepo.save(derivative);
+        this.imageDerivativeRepo.save(derivative).then((r) => {
+          if (HasFailed(r)) r.print(this.logger);
+        });
      }

      return derivative;
@ -176,4 +201,47 @@ export class ImageFileDBService {
      return Fail(FT.Database, e);
    }
  }
+
+  public async cleanupOrphanedDerivatives(): AsyncFailable<number> {
+    return this.cleanupRepoWithS3(this.imageDerivativeRepo);
+  }
+
+  public async cleanupOrphanedFiles(): AsyncFailable<number> {
+    return this.cleanupRepoWithS3(this.imageFileRepo);
+  }
+
+  private async cleanupRepoWithS3(
+    repo: Repository<{ image_id: string | null; s3key: string }>,
+  ): AsyncFailable<number> {
+    try {
+      let remaining = Infinity;
+      let processed = 0;
+      while (remaining > 0) {
+        const orphaned = await repo.findAndCount({
+          where: {
+            image_id: IsNull(),
+          },
+          select: ['s3key'],
+          take: 100,
+        });
+        if (orphaned[1] === 0) break;
+        remaining = orphaned[1] - orphaned[0].length;
+
+        const keys = orphaned[0].map((d) => d.s3key);
+
+        const s3result = await this.s3Service.deleteFiles(keys);
+        if (HasFailed(s3result)) return s3result;
+
+        const result = await repo.delete({
+          s3key: In(keys),
+        });
+
+        processed += result.affected ?? 0;
+      }
+
+      return processed;
+    } catch (e) {
+      return Fail(FT.Database, e);
+    }
+  }
 }
--- a/backend/src/collections/preference-db/preference-common.service.ts
+++ b/backend/src/collections/preference-db/preference-common.service.ts
@ -10,7 +10,7 @@ import {
  Failable,
  FT,
  HasFailed,
-} from 'picsur-shared/dist/types/failable';
+} from 'picsur-shared/dist/types';

 type Enum = Record<string, string>;
 type EnumValue<E> = E[keyof E];
--- a/backend/src/collections/preference-db/sys-preference-db.service.ts
+++ b/backend/src/collections/preference-db/sys-preference-db.service.ts
@ -11,12 +11,7 @@ import {
  SysPreferenceValidators,
  SysPreferenceValueTypes,
 } from 'picsur-shared/dist/dto/sys-preferences.enum';
-import {
-  AsyncFailable,
-  Fail,
-  FT,
-  HasFailed,
-} from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, Fail, FT, HasFailed } from 'picsur-shared/dist/types';
 import { Repository } from 'typeorm';
 import {
  ESysPreferenceBackend,
@ -42,7 +37,7 @@ export class SysPreferenceDbService {
    value: PrefValueType,
  ): AsyncFailable<DecodedSysPref> {
    // Validate
-    const sysPreference = await this.encodeSysPref(key, value);
+    let sysPreference = await this.encodeSysPref(key, value);
    if (HasFailed(sysPreference)) return sysPreference;

    // Set
@ -65,7 +60,7 @@ export class SysPreferenceDbService {

  public async getPreference(key: string): AsyncFailable<DecodedSysPref> {
    // Validate
-    const validatedKey = this.prefCommon.validatePrefKey(key, SysPreference);
+    let validatedKey = this.prefCommon.validatePrefKey(key, SysPreference);
    if (HasFailed(validatedKey)) return validatedKey;

    // See the comment in 'mutex-fallback.ts' for why we are using a mutex here
@ -76,7 +71,6 @@ export class SysPreferenceDbService {
        try {
          existing = await this.sysPreferenceRepository.findOne({
            where: { key: validatedKey as SysPreference },
-            cache: 60000,
          });
          if (!existing) return null;
        } catch (e) {
@ -117,7 +111,7 @@ export class SysPreferenceDbService {
    key: string,
    type: PrefValueTypeStrings,
  ): AsyncFailable<PrefValueType> {
-    const pref = await this.getPreference(key);
+    let pref = await this.getPreference(key);
    if (HasFailed(pref)) return pref;
    if (pref.type !== type)
      return Fail(FT.UsrValidation, 'Invalid preference type');
@ -127,7 +121,7 @@ export class SysPreferenceDbService {

  public async getAllPreferences(): AsyncFailable<DecodedSysPref[]> {
    // TODO: We are fetching each value invidually, we should fetch all at once
-    const internalSysPrefs = await Promise.all(
+    let internalSysPrefs = await Promise.all(
      SysPreferenceList.map((key) => this.getPreference(key)),
    );
    if (internalSysPrefs.some((pref) => HasFailed(pref))) {
@ -163,7 +157,7 @@ export class SysPreferenceDbService {
      return Fail(FT.UsrValidation, undefined, valueValidated.error);
    }

-    const verifySysPreference = new ESysPreferenceBackend();
+    let verifySysPreference = new ESysPreferenceBackend();
    verifySysPreference.key = validated.key;
    verifySysPreference.value = validated.value;

--- a/backend/src/collections/preference-db/usr-preference-db.service.ts
+++ b/backend/src/collections/preference-db/usr-preference-db.service.ts
@ -11,12 +11,7 @@ import {
  UsrPreferenceValidators,
  UsrPreferenceValueTypes,
 } from 'picsur-shared/dist/dto/usr-preferences.enum';
-import {
-  AsyncFailable,
-  Fail,
-  FT,
-  HasFailed,
-} from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, Fail, FT, HasFailed } from 'picsur-shared/dist/types';
 import { Repository } from 'typeorm';
 import {
  EUsrPreferenceBackend,
@ -43,7 +38,7 @@ export class UsrPreferenceDbService {
    value: PrefValueType,
  ): AsyncFailable<DecodedUsrPref> {
    // Validate
-    const usrPreference = await this.encodeUsrPref(userid, key, value);
+    let usrPreference = await this.encodeUsrPref(userid, key, value);
    if (HasFailed(usrPreference)) return usrPreference;

    // Set
@ -71,7 +66,7 @@ export class UsrPreferenceDbService {
    key: string,
  ): AsyncFailable<DecodedUsrPref> {
    // Validate
-    const validatedKey = this.prefCommon.validatePrefKey(key, UsrPreference);
+    let validatedKey = this.prefCommon.validatePrefKey(key, UsrPreference);
    if (HasFailed(validatedKey)) return validatedKey;

    // See the comment in 'mutex-fallback.ts' for why we are using a mutex here
@ -82,7 +77,6 @@ export class UsrPreferenceDbService {
        try {
          existing = await this.usrPreferenceRepository.findOne({
            where: { key: validatedKey as UsrPreference, user_id: userid },
-            cache: 60000,
          });
          if (!existing) return null;
        } catch (e) {
@ -150,7 +144,7 @@ export class UsrPreferenceDbService {
    key: string,
    type: PrefValueTypeStrings,
  ): AsyncFailable<PrefValueType> {
-    const pref = await this.getPreference(userid, key);
+    let pref = await this.getPreference(userid, key);
    if (HasFailed(pref)) return pref;
    if (pref.type !== type)
      return Fail(FT.UsrValidation, 'Invalid preference type');
@ -162,7 +156,7 @@ export class UsrPreferenceDbService {
    userid: string,
  ): AsyncFailable<DecodedUsrPref[]> {
    // TODO: We are fetching each value invidually, we should fetch all at once
-    const internalSysPrefs = await Promise.all(
+    let internalSysPrefs = await Promise.all(
      UsrPreferenceList.map((key) => this.getPreference(userid, key)),
    );
    if (internalSysPrefs.some((pref) => HasFailed(pref))) {
@ -204,7 +198,7 @@ export class UsrPreferenceDbService {
      return Fail(FT.UsrValidation, undefined, valueValidated.error);
    }

-    const verifySysPreference = new EUsrPreferenceBackend();
+    let verifySysPreference = new EUsrPreferenceBackend();
    verifySysPreference.key = validated.key;
    verifySysPreference.value = validated.value;
    verifySysPreference.user_id = userid;
--- a/backend/src/collections/role-db/role-db.module.ts
+++ b/backend/src/collections/role-db/role-db.module.ts
@ -1,6 +1,6 @@
 import { Logger, Module, OnModuleInit } from '@nestjs/common';
 import { TypeOrmModule } from '@nestjs/typeorm';
-import { HasFailed } from 'picsur-shared/dist/types/failable';
+import { HasFailed } from 'picsur-shared/dist/types';
 import { EarlyConfigModule } from '../../config/early/early-config.module';
 import { HostConfigService } from '../../config/early/host.config.service';
 import { ERoleBackend } from '../../database/entities/users/role.entity';
--- a/backend/src/collections/role-db/role-db.service.ts
+++ b/backend/src/collections/role-db/role-db.service.ts
@ -7,7 +7,7 @@ import {
  FT,
  HasFailed,
  HasSuccess,
-} from 'picsur-shared/dist/types/failable';
+} from 'picsur-shared/dist/types';
 import { makeUnique } from 'picsur-shared/dist/util/unique';
 import { In, Repository } from 'typeorm';
 import { ERoleBackend } from '../../database/entities/users/role.entity';
@ -33,7 +33,7 @@ export class RoleDbService {
    if (await this.exists(name))
      return Fail(FT.Conflict, 'Role already exists');

-    const role = new ERoleBackend();
+    let role = new ERoleBackend();
    role.name = name;
    role.permissions = permissions;

@ -105,7 +105,7 @@ export class RoleDbService {
    role: string | ERoleBackend,
    permissions: Permissions,
    // Extra bypass for internal use
-    allowImmutable = false,
+    allowImmutable: boolean = false,
  ): AsyncFailable<ERoleBackend> {
    const roleToModify = await this.resolve(role);
    if (HasFailed(roleToModify)) return roleToModify;
@ -166,7 +166,7 @@ export class RoleDbService {
    return HasSuccess(await this.findOne(name));
  }

-  public async nukeSystemRoles(IAmSure = false): AsyncFailable<true> {
+  public async nukeSystemRoles(IAmSure: boolean = false): AsyncFailable<true> {
    if (!IAmSure)
      return Fail(
        FT.SysValidation,
--- a/backend/src/collections/system-state-db/system-state-db.service.ts
+++ b/backend/src/collections/system-state-db/system-state-db.service.ts
@ -1,6 +1,6 @@
 import { Injectable, Logger } from '@nestjs/common';
 import { InjectRepository } from '@nestjs/typeorm';
-import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types';
 import { Repository } from 'typeorm';
 import { ESystemStateBackend } from '../../database/entities/system/system-state.entity';

--- a/backend/src/collections/user-db/user-db.module.ts
+++ b/backend/src/collections/user-db/user-db.module.ts
@ -1,6 +1,6 @@
 import { Logger, Module, OnModuleInit } from '@nestjs/common';
 import { TypeOrmModule } from '@nestjs/typeorm';
-import { HasFailed } from 'picsur-shared/dist/types/failable';
+import { HasFailed } from 'picsur-shared/dist/types';
 import { generateRandomString } from 'picsur-shared/dist/util/random';
 import { AuthConfigService } from '../../config/early/auth.config.service';
 import { EarlyConfigModule } from '../../config/early/early-config.module';
--- a/backend/src/collections/user-db/user-db.service.ts
+++ b/backend/src/collections/user-db/user-db.service.ts
@ -8,7 +8,7 @@ import {
  FT,
  HasFailed,
  HasSuccess,
-} from 'picsur-shared/dist/types/failable';
+} from 'picsur-shared/dist/types';
 import { FindResult } from 'picsur-shared/dist/types/find-result';
 import { makeUnique } from 'picsur-shared/dist/util/unique';
 import { Repository } from 'typeorm';
@ -53,7 +53,7 @@ export class UserDbService {
    const strength = await this.getBCryptStrength();
    const hashedPassword = await bcrypt.hash(password, strength);

-    const user = new EUserBackend();
+    let user = new EUserBackend();
    user.username = username;
    user.hashed_password = hashedPassword;
    if (byPassRoleCheck) {
@ -208,7 +208,7 @@ export class UserDbService {
    username: string,
    // Also fetch fields that aren't normally sent to the client
    // (e.g. hashed password)
-    getPrivate = false,
+    getPrivate: boolean = false,
  ): AsyncFailable<EUserBackend> {
    try {
      const found = await this.usersRepository.findOne({
--- a/backend/src/config/early/bull.config.service.ts
+++ b/backend/src/config/early/bull.config.service.ts
@ -0,0 +1,34 @@
+import {
+  BullRootModuleOptions,
+  SharedBullConfigurationFactory,
+} from '@nestjs/bull';
+import { Injectable } from '@nestjs/common';
+import { RedisConfigService } from './redis.config.service';
+
+@Injectable()
+export class BullConfigService implements SharedBullConfigurationFactory {
+  constructor(private readonly redisConfig: RedisConfigService) {}
+
+  async createSharedConfiguration(): Promise<BullRootModuleOptions> {
+    const options: BullRootModuleOptions = {
+      url: this.redisConfig.getRedisUrl(),
+      redis: {
+        lazyConnect: false,
+      },
+      defaultJobOptions: {
+        attempts: 3,
+        backoff: {
+          delay: 500,
+          type: 'fixed',
+        },
+        removeOnFail: {
+          age: 1000 * 60 * 60 * 24 * 7, // 7 days
+        },
+        removeOnComplete: {
+          age: 1000 * 60 * 60 * 24 * 7, // 7 days
+        },
+      },
+    };
+    return options;
+  }
+}
--- a/backend/src/config/early/early-config.module.ts
+++ b/backend/src/config/early/early-config.module.ts
@ -1,10 +1,12 @@
 import { Module } from '@nestjs/common';
 import { ConfigModule } from '@nestjs/config';
 import { AuthConfigService } from './auth.config.service';
+import { BullConfigService } from './bull.config.service';
 import { EarlyJwtConfigService } from './early-jwt.config.service';
 import { HostConfigService } from './host.config.service';
 import { MultipartConfigService } from './multipart.config.service';
 import { RedisConfigService } from './redis.config.service';
+import { S3ConfigService } from './s3.config.service';
 import { ServeStaticConfigService } from './serve-static.config.service';
 import { TypeOrmConfigService } from './type-orm.config.service';

@ -23,6 +25,8 @@ import { TypeOrmConfigService } from './type-orm.config.service';
    AuthConfigService,
    MultipartConfigService,
    RedisConfigService,
+    BullConfigService,
+    S3ConfigService,
  ],
  exports: [
    ConfigModule,
@ -33,6 +37,8 @@ import { TypeOrmConfigService } from './type-orm.config.service';
    AuthConfigService,
    MultipartConfigService,
    RedisConfigService,
+    BullConfigService,
+    S3ConfigService,
  ],
 })
 export class EarlyConfigModule {}
--- a/backend/src/config/early/s3.config.service.ts
+++ b/backend/src/config/early/s3.config.service.ts
@ -0,0 +1,73 @@
+import { S3ClientConfig } from '@aws-sdk/client-s3';
+import { Injectable, Logger } from '@nestjs/common';
+import { ConfigService } from '@nestjs/config';
+import { ParseString } from 'picsur-shared/dist/util/parse-simple';
+import { EnvPrefix } from '../config.static';
+
+@Injectable()
+export class S3ConfigService {
+  private readonly logger = new Logger(S3ConfigService.name);
+
+  constructor(private readonly configService: ConfigService) {
+    if (this.getS3Endpoint())
+      this.logger.log('Custom S3 Endpoint: ' + this.getS3Endpoint());
+
+    this.logger.log('S3 Region: ' + this.getS3Region());
+    this.logger.log('S3 Bucket: ' + this.getS3Bucket());
+
+    this.logger.verbose('S3 Access Key: ' + this.getS3AccessKey());
+    this.logger.verbose('S3 Secret Key: ' + this.getS3SecretKey());
+  }
+
+  public getS3Config(): S3ClientConfig {
+    return {
+      credentials: {
+        accessKeyId: this.getS3AccessKey(),
+        secretAccessKey: this.getS3SecretKey(),
+      },
+      endpoint: this.getS3Endpoint() ?? undefined,
+      region: this.getS3Region(),
+      tls: this.getS3TLS(),
+    };
+  }
+
+  public getS3Endpoint(): string | null {
+    return ParseString(this.configService.get(`${EnvPrefix}S3_ENDPOINT`), null);
+  }
+
+  public getS3TLS(): boolean | undefined {
+    const endpoint = this.getS3Endpoint();
+    if (endpoint) {
+      return endpoint.startsWith('https');
+    }
+    return undefined;
+  }
+
+  public getS3Bucket(): string {
+    return ParseString(
+      this.configService.get(`${EnvPrefix}S3_BUCKET`),
+      'picsur',
+    );
+  }
+
+  public getS3Region(): string {
+    return ParseString(
+      this.configService.get(`${EnvPrefix}S3_REGION`),
+      'us-east-1',
+    );
+  }
+
+  public getS3AccessKey(): string {
+    return ParseString(
+      this.configService.get(`${EnvPrefix}S3_ACCESS_KEY`),
+      'picsur',
+    );
+  }
+
+  public getS3SecretKey(): string {
+    return ParseString(
+      this.configService.get(`${EnvPrefix}S3_SECRET_KEY`),
+      'picsur',
+    );
+  }
+}
--- a/backend/src/config/early/type-orm.config.service.ts
+++ b/backend/src/config/early/type-orm.config.service.ts
@ -2,10 +2,11 @@ import { Injectable, Logger } from '@nestjs/common';
 import { ConfigService } from '@nestjs/config';
 import { TypeOrmModuleOptions, TypeOrmOptionsFactory } from '@nestjs/typeorm';
 import { ParseInt, ParseString } from 'picsur-shared/dist/util/parse-simple';
-import { EntityList } from '../../database/entities/index';
-import { MigrationList } from '../../database/migrations/index';
+import { EntityList } from '../../database/entities';
+import { MigrationList } from '../../database/migrations';
 import { DefaultName, EnvPrefix } from '../config.static';
 import { HostConfigService } from './host.config.service';
+import { RedisConfigService } from './redis.config.service';

@Injectable()
 export class TypeOrmConfigService implements TypeOrmOptionsFactory {
@ -14,6 +15,7 @@ export class TypeOrmConfigService implements TypeOrmOptionsFactory {
  constructor(
    private readonly configService: ConfigService,
    private readonly hostService: HostConfigService,
+    private readonly redisConfig: RedisConfigService,
  ) {
    const varOptions = this.getTypeOrmServerOptions();

@ -48,10 +50,10 @@ export class TypeOrmConfigService implements TypeOrmOptionsFactory {
    return varOptions;
  }

-  public createTypeOrmOptions() {
+  public createTypeOrmOptions(connectionName?: string) {
    const varOptions = this.getTypeOrmServerOptions();
    return {
-      type: 'postgres' as const,
+      type: 'postgres' as 'postgres',
      synchronize: !this.hostService.isProduction(),

      migrationsRun: true,
@ -66,6 +68,13 @@ export class TypeOrmConfigService implements TypeOrmOptionsFactory {
        entitiesDir: 'src/database/entities',
      },

+      cache: {
+        duration: 60000,
+        type: 'ioredis',
+        alwaysEnabled: false,
+        options: this.redisConfig.getRedisUrl(),
+      },
+
      ...varOptions,
    } as TypeOrmModuleOptions;
  }
--- a/backend/src/config/late/info.config.service.ts
+++ b/backend/src/config/late/info.config.service.ts
@ -1,6 +1,6 @@
 import { Injectable, Logger } from '@nestjs/common';
 import { SysPreference } from 'picsur-shared/dist/dto/sys-preferences.enum';
-import { HasFailed } from 'picsur-shared/dist/types/failable';
+import { HasFailed } from 'picsur-shared/dist/types';
 import { SysPreferenceDbService } from '../../collections/preference-db/sys-preference-db.service';

@Injectable()
--- a/backend/src/config/late/jwt.config.service.ts
+++ b/backend/src/config/late/jwt.config.service.ts
@ -1,7 +1,7 @@
 import { FactoryProvider, Injectable, Logger } from '@nestjs/common';
 import { JwtModuleOptions, JwtOptionsFactory } from '@nestjs/jwt';
 import ms from 'ms';
-import { ThrowIfFailed } from 'picsur-shared/dist/types/failable';
+import { ThrowIfFailed } from 'picsur-shared/dist/types';
 import { SysPreferenceDbService } from '../../collections/preference-db/sys-preference-db.service';

@Injectable()
--- a/backend/src/config/late/usage.config.service.ts
+++ b/backend/src/config/late/usage.config.service.ts
@ -1,11 +1,6 @@
 import { Injectable } from '@nestjs/common';
 import { SysPreference } from 'picsur-shared/dist/dto/sys-preferences.enum';
-import {
-  AsyncFailable,
-  Fail,
-  FT,
-  HasFailed,
-} from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, Fail, FT, HasFailed } from 'picsur-shared/dist/types';
 import { URLRegex, UUIDRegex } from 'picsur-shared/dist/util/common-regex';
 import { SysPreferenceDbService } from '../../collections/preference-db/sys-preference-db.service';
 import { ReportInterval, ReportUrl } from '../config.static';
--- a/backend/src/database/entities/images/image-derivative.entity.ts
+++ b/backend/src/database/entities/images/image-derivative.entity.ts
@ -4,7 +4,7 @@ import {
  Index,
  JoinColumn,
  ManyToOne,
-  PrimaryGeneratedColumn,
+  PrimaryColumn,
  Unique,
 } from 'typeorm';
 import { EImageBackend } from './image.entity';
@ -12,22 +12,24 @@ import { EImageBackend } from './image.entity';
@Entity()
@Unique(['image_id', 'key'])
 export class EImageDerivativeBackend {
-  @PrimaryGeneratedColumn('uuid')
-  private _id?: string;
+  @PrimaryColumn({ type: 'uuid', nullable: false })
+  @Index()
+  s3key: string;

  // We do a little trickery
  @Index()
  @ManyToOne(() => EImageBackend, (image) => image.derivatives, {
-    nullable: false,
-    onDelete: 'CASCADE',
+    nullable: true,
+    onDelete: 'SET NULL',
  })
  @JoinColumn({ name: 'image_id' })
  private _image?: any;

  @Column({
    name: 'image_id',
+    nullable: true,
  })
-  image_id: string;
+  image_id: string | null;

  @Index()
  @Column({ nullable: false })
@ -42,8 +44,4 @@ export class EImageDerivativeBackend {
    nullable: false,
  })
  last_read: Date;
-
-  // Binary data
-  @Column({ type: 'bytea', nullable: false })
-  data: Buffer;
 }
--- a/backend/src/database/entities/images/image-file.entity.ts
+++ b/backend/src/database/entities/images/image-file.entity.ts
@ -5,7 +5,7 @@ import {
  Index,
  JoinColumn,
  ManyToOne,
-  PrimaryGeneratedColumn,
+  PrimaryColumn,
  Unique,
 } from 'typeorm';
 import { EImageBackend } from './image.entity';
@ -13,22 +13,24 @@ import { EImageBackend } from './image.entity';
@Entity()
@Unique(['image_id', 'variant'])
 export class EImageFileBackend {
-  @PrimaryGeneratedColumn('uuid')
-  private _id?: string;
+  @PrimaryColumn({ type: 'uuid', nullable: false })
+  @Index()
+  s3key: string;

  // We do a little trickery
  @Index()
  @ManyToOne(() => EImageBackend, (image) => image.files, {
-    nullable: false,
-    onDelete: 'CASCADE',
+    nullable: true,
+    onDelete: 'SET NULL',
  })
  @JoinColumn({ name: 'image_id' })
  private _image?: any;

  @Column({
    name: 'image_id',
+    nullable: true,
  })
-  image_id: string;
+  image_id: string | null;

  @Index()
  @Column({ nullable: false, enum: ImageEntryVariant })
@ -36,8 +38,4 @@ export class EImageFileBackend {

  @Column({ nullable: false })
  filetype: string;
-
-  // Binary data
-  @Column({ type: 'bytea', nullable: false })
-  data: Buffer;
 }
--- a/backend/src/database/entities/system/sys-preference.entity.ts
+++ b/backend/src/database/entities/system/sys-preference.entity.ts
@ -1,6 +1,6 @@
 import { IsEntityID } from 'picsur-shared/dist/validators/entity-id.validator';
 import { Column, Entity, Index, PrimaryGeneratedColumn } from 'typeorm';
-import * as z from 'zod';
+import z from 'zod';

 export const ESysPreferenceSchema = z.object({
  id: IsEntityID().optional(),
--- a/backend/src/database/entities/system/usr-preference.entity.ts
+++ b/backend/src/database/entities/system/usr-preference.entity.ts
@ -8,7 +8,7 @@ import {
  PrimaryGeneratedColumn,
  Unique,
 } from 'typeorm';
-import * as z from 'zod';
+import z from 'zod';
 import { EUserBackend } from '../users/user.entity';

 export const EUsrPreferenceSchema = z.object({
--- a/backend/src/database/migrations/1672154027079-V_0_5_0_a.ts
+++ b/backend/src/database/migrations/1672154027079-V_0_5_0_a.ts
@ -1,21 +0,0 @@
-import { MigrationInterface, QueryRunner } from 'typeorm';
-
-export class V050A1672154027079 implements MigrationInterface {
-  name = 'V050A1672154027079';
-
-  public async up(queryRunner: QueryRunner): Promise<void> {
-    await queryRunner.query(
-      `CREATE TABLE "e_system_state_backend" ("id" uuid NOT NULL DEFAULT uuid_generate_v4(), "key" character varying NOT NULL, "value" character varying NOT NULL, CONSTRAINT "UQ_f11f1605928b497b24f4b3ecc1f" UNIQUE ("key"), CONSTRAINT "PK_097ea165dadc8c14237481afd64" PRIMARY KEY ("id"))`,
-    );
-    await queryRunner.query(
-      `CREATE INDEX "IDX_f11f1605928b497b24f4b3ecc1" ON "e_system_state_backend" ("key") `,
-    );
-  }
-
-  public async down(queryRunner: QueryRunner): Promise<void> {
-    await queryRunner.query(
-      `DROP INDEX "public"."IDX_f11f1605928b497b24f4b3ecc1"`,
-    );
-    await queryRunner.query(`DROP TABLE "e_system_state_backend"`);
-  }
-}
--- a/backend/src/database/migrations/index.ts
+++ b/backend/src/database/migrations/index.ts
@ -1,19 +1,15 @@
-import { MigrationInterface } from 'typeorm';
 import { V030A1661692206479 } from './1661692206479-V_0_3_0_a';
 import { V032A1662029904716 } from './1662029904716-V_0_3_2_a';
 import { V040A1662314197741 } from './1662314197741-V_0_4_0_a';
 import { V040B1662485374471 } from './1662485374471-V_0_4_0_b';
 import { V040C1662535484200 } from './1662535484200-V_0_4_0_c';
 import { V040D1662728275448 } from './1662728275448-V_0_4_0_d';
-import { V050A1672154027079 } from './1672154027079-V_0_5_0_a';
-import { Newable } from 'picsur-shared/dist/types/newable.js';

-export const MigrationList: Newable<MigrationInterface>[] = [
+export const MigrationList: Function[] = [
  V030A1661692206479,
  V032A1662029904716,
  V040A1662314197741,
  V040B1662485374471,
  V040C1662535484200,
  V040D1662728275448,
-  V050A1672154027079,
 ];
--- a/backend/src/decorators/image-id/image-full-id.pipe.ts
+++ b/backend/src/decorators/image-id/image-full-id.pipe.ts
@ -1,12 +1,12 @@
-import { Injectable, PipeTransform } from '@nestjs/common';
+import { ArgumentMetadata, Injectable, PipeTransform } from '@nestjs/common';
 import { Ext2FileType } from 'picsur-shared/dist/dto/mimes.dto';
-import { FT, Fail, HasFailed } from 'picsur-shared/dist/types/failable';
+import { Fail, FT, HasFailed } from 'picsur-shared/dist/types';
 import { UUIDRegex } from 'picsur-shared/dist/util/common-regex';
 import { ImageFullId } from '../../models/constants/image-full-id.const';

@Injectable()
 export class ImageFullIdPipe implements PipeTransform<string, ImageFullId> {
-  transform(value: string): ImageFullId {
+  transform(value: string, metadata: ArgumentMetadata): ImageFullId {
    const split = value.split('.');
    if (split.length === 2) {
      const [id, ext] = split;
--- a/backend/src/decorators/image-id/image-id.pipe.ts
+++ b/backend/src/decorators/image-id/image-id.pipe.ts
@ -1,10 +1,10 @@
-import { Injectable, PipeTransform } from '@nestjs/common';
-import { FT, Fail } from 'picsur-shared/dist/types/failable';
+import { ArgumentMetadata, Injectable, PipeTransform } from '@nestjs/common';
+import { Fail, FT } from 'picsur-shared/dist/types';
 import { UUIDRegex } from 'picsur-shared/dist/util/common-regex';

@Injectable()
 export class ImageIdPipe implements PipeTransform<string, string> {
-  transform(value: string): string {
+  transform(value: string, metadata: ArgumentMetadata): string {
    if (UUIDRegex.test(value)) return value;
    throw Fail(FT.UsrValidation, 'Invalid image id');
  }
--- a/backend/src/decorators/multipart/postfile.pipe.ts
+++ b/backend/src/decorators/multipart/postfile.pipe.ts
@ -1,7 +1,7 @@
 import { Multipart, MultipartFile } from '@fastify/multipart';
 import { Injectable, Logger, PipeTransform, Scope } from '@nestjs/common';
 import { FastifyRequest } from 'fastify';
-import { Fail, FT } from 'picsur-shared/dist/types/failable';
+import { Fail, FT } from 'picsur-shared/dist/types';
 import { MultipartConfigService } from '../../config/early/multipart.config.service';

@Injectable({ scope: Scope.REQUEST })
@ -12,7 +12,7 @@ export class PostFilePipe implements PipeTransform {
    private readonly multipartConfigService: MultipartConfigService,
  ) {}

-  async transform({ request }: { request: FastifyRequest }) {
+  async transform({ request, data }: { data: any; request: FastifyRequest }) {
    if (!request.isMultipart()) throw Fail(FT.UsrValidation, 'Invalid file');

    // Only one file is allowed
--- a/backend/src/decorators/multipart/postfiles.pipe.ts
+++ b/backend/src/decorators/multipart/postfiles.pipe.ts
@ -1,7 +1,13 @@
 import { MultipartFile } from '@fastify/multipart';
-import { Injectable, Logger, PipeTransform, Scope } from '@nestjs/common';
+import {
+  ArgumentMetadata,
+  Injectable,
+  Logger,
+  PipeTransform,
+  Scope,
+} from '@nestjs/common';
 import { FastifyRequest } from 'fastify';
-import { FT, Fail } from 'picsur-shared/dist/types/failable';
+import { Fail, FT } from 'picsur-shared/dist/types';
 import { MultipartConfigService } from '../../config/early/multipart.config.service';

 export type FileIterator = AsyncIterableIterator<MultipartFile>;
@ -14,7 +20,10 @@ export class MultiPartPipe implements PipeTransform {
    private readonly multipartConfigService: MultipartConfigService,
  ) {}

-  async transform({ request, data }: { data: any; request: FastifyRequest }) {
+  async transform<T extends Object>(
+    { request, data }: { data: any; request: FastifyRequest },
+    metadata: ArgumentMetadata,
+  ) {
    const filesLimit = typeof data === 'number' ? data : undefined;

    if (!request.isMultipart()) throw Fail(FT.UsrValidation, 'Invalid files');
--- a/backend/src/decorators/permissions.decorator.ts
+++ b/backend/src/decorators/permissions.decorator.ts
@ -4,7 +4,7 @@ import {
  SetMetadata,
  UseGuards,
 } from '@nestjs/common';
-import { Fail, FT } from 'picsur-shared/dist/types/failable';
+import { Fail, FT } from 'picsur-shared/dist/types';
 import { CombineFCDecorators } from 'picsur-shared/dist/util/decorator';
 import { LocalAuthGuard } from '../managers/auth/guards/local-auth.guard';
 import { Permission, Permissions } from '../models/constants/permissions.const';
--- a/backend/src/decorators/request-user.decorator.ts
+++ b/backend/src/decorators/request-user.decorator.ts
@ -1,5 +1,5 @@
 import { createParamDecorator, ExecutionContext } from '@nestjs/common';
-import { Fail, FT } from 'picsur-shared/dist/types/failable';
+import { Fail, FT } from 'picsur-shared/dist/types';
 import AuthFastifyRequest from '../models/interfaces/authrequest.dto';

 export const ReqUser = createParamDecorator(
--- a/backend/src/decorators/returns.decorator.ts
+++ b/backend/src/decorators/returns.decorator.ts
@ -6,12 +6,12 @@ import { Newable } from 'picsur-shared/dist/types/newable';
 type ReturnsMethodDecorator<ReturnType> = <
  T extends (...args: any) => ReturnType | Promise<ReturnType>,
 >(
-  target: object,
+  target: Object,
  propertyKey: string | symbol,
  descriptor: TypedPropertyDescriptor<T>,
 ) => TypedPropertyDescriptor<T> | void;

-export function Returns<N extends object>(
+export function Returns<N extends Object>(
  newable: Newable<N>,
 ): ReturnsMethodDecorator<N> {
  return SetMetadata('returns', newable);
--- a/backend/src/layers/exception/exception.filter.ts
+++ b/backend/src/layers/exception/exception.filter.ts
@ -23,7 +23,7 @@ import {

@Catch()
 export class MainExceptionFilter implements ExceptionFilter {
-  private static readonly logger = new Logger('MainExceptionFilter');
+  private static readonly logger = new Logger(MainExceptionFilter.name);

  catch(exception: Failure, host: ArgumentsHost) {
    const ctx = host.switchToHttp();
--- a/backend/src/layers/success/success.interceptor.ts
+++ b/backend/src/layers/success/success.interceptor.ts
@ -9,7 +9,7 @@ import {
 import { Reflector } from '@nestjs/core';
 import { FastifyReply } from 'fastify';
 import { ApiAnySuccessResponse } from 'picsur-shared/dist/dto/api/api.dto';
-import { Fail, FT } from 'picsur-shared/dist/types/failable';
+import { Fail, FT } from 'picsur-shared/dist/types';
 import { ZodDtoStatic } from 'picsur-shared/dist/util/create-zod-dto';
 import { map, Observable } from 'rxjs';

@ -20,7 +20,7 @@ export interface ZodValidationInterceptorOptions {
 }

@Injectable()
-export class SuccessInterceptor implements NestInterceptor {
+export class SuccessInterceptor<T> implements NestInterceptor {
  private readonly logger = new Logger();

  // TODO: make work
@ -82,7 +82,7 @@ export class SuccessInterceptor implements NestInterceptor {
      );
    }

-    const schema = schemaStatic.zodSchema;
+    let schema = schemaStatic.zodSchema;

    const parseResult = schema.safeParse(data);
    if (!parseResult.success) {
@ -105,7 +105,7 @@ export class SuccessInterceptor implements NestInterceptor {
    const response = context.switchToHttp().getResponse<FastifyReply>();

    const newResponse: ApiAnySuccessResponse = {
-      success: true as const, // really typescript
+      success: true as true, // really typescript
      statusCode: response.statusCode,
      timestamp: new Date().toISOString(),
      timeMs: Math.round(response.getResponseTime()),
--- a/backend/src/layers/throttler/PicsurThrottler.guard.ts
+++ b/backend/src/layers/throttler/PicsurThrottler.guard.ts
@ -1,10 +1,10 @@
-import { Injectable } from '@nestjs/common';
+import { ExecutionContext, Injectable } from '@nestjs/common';
 import { ThrottlerGuard } from '@nestjs/throttler';
-import { FT, Fail } from 'picsur-shared/dist/types/failable';
+import { Fail, FT } from 'picsur-shared/dist/types';

@Injectable()
 export class PicsurThrottlerGuard extends ThrottlerGuard {
-  protected override throwThrottlingException(): void {
+  protected override throwThrottlingException(context: ExecutionContext): void {
    throw Fail(FT.RateLimit);
  }
 }
--- a/backend/src/layers/validate/zod-validator.pipe.ts
+++ b/backend/src/layers/validate/zod-validator.pipe.ts
@ -9,7 +9,7 @@ import {
  Optional,
  PipeTransform,
 } from '@nestjs/common';
-import { Fail, FT } from 'picsur-shared/dist/types/failable';
+import { Fail, FT } from 'picsur-shared/dist/types';
 import { ZodDtoStatic } from 'picsur-shared/dist/util/create-zod-dto';

 export interface ZodValidationPipeOptions {
@ -30,7 +30,7 @@ export class ZodValidationPipe implements PipeTransform {
  public transform(value: unknown, metadata: ArgumentMetadata): unknown {
    if (!this.validateCustom && metadata.type === 'custom') return value;

-    const zodSchema = (metadata?.metatype as ZodDtoStatic)?.zodSchema;
+    let zodSchema = (metadata?.metatype as ZodDtoStatic)?.zodSchema;

    if (zodSchema) {
      const parseResult = zodSchema.safeParse(value);
--- a/backend/src/managers/auth/auth.service.ts
+++ b/backend/src/managers/auth/auth.service.ts
@ -2,7 +2,7 @@ import { Injectable, Logger } from '@nestjs/common';
 import { JwtService } from '@nestjs/jwt';
 import { JwtData, JwtDataSchema } from 'picsur-shared/dist/dto/jwt.dto';
 import { EUser } from 'picsur-shared/dist/entities/user.entity';
-import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types';

@Injectable()
 export class AuthManagerService {
--- a/backend/src/managers/auth/guards/apikey.strategy.ts
+++ b/backend/src/managers/auth/guards/apikey.strategy.ts
@ -2,7 +2,7 @@ import { Injectable, Logger } from '@nestjs/common';
 import { PassportStrategy } from '@nestjs/passport';
 import { HeaderAPIKeyStrategy } from 'passport-headerapikey';
 import { EUser, EUserSchema } from 'picsur-shared/dist/entities/user.entity';
-import { HasFailed } from 'picsur-shared/dist/types/failable';
+import { HasFailed } from 'picsur-shared/dist/types';
 import { IsApiKey } from 'picsur-shared/dist/validators/api-key.validator';
 import { ApiKeyDbService } from '../../../collections/apikey-db/apikey-db.service';
 import { EUserBackend2EUser } from '../../../models/transformers/user.transformer';
@ -23,7 +23,7 @@ export class ApiKeyStrategy extends PassportStrategy(
      false,
      (
        apikey: string,
-        verified: (err: Error | null, user?: object, info?: object) => void,
+        verified: (err: Error | null, user?: Object, info?: Object) => void,
      ) => {
        this.validate(apikey)
          .then((user) => {
--- a/backend/src/managers/auth/guards/guest.strategy.ts
+++ b/backend/src/managers/auth/guards/guest.strategy.ts
@ -8,12 +8,11 @@ import { ReqType } from './reqtype';

 class GuestPassportStrategy extends Strategy {
  // Will be overridden by the nest implementation
-  // eslint-disable-next-line @typescript-eslint/no-unused-vars
  async validate(req: ReqType): Promise<any> {
    return undefined;
  }

-  override async authenticate(req: ReqType) {
+  override async authenticate(req: ReqType, options?: any) {
    const user = await this.validate(req);
    this.success(user);
  }
@ -29,7 +28,7 @@ export class GuestStrategy extends PassportStrategy(
  }

  // Return the guest user created by the guestservice
-  override async validate(): Promise<EUser> {
+  override async validate(payload: any): Promise<EUser> {
    return EUserBackend2EUser(await this.guestService.getGuestUser());
  }
 }
--- a/backend/src/managers/auth/guards/jwt.strategy.ts
+++ b/backend/src/managers/auth/guards/jwt.strategy.ts
@ -3,7 +3,7 @@ import { PassportStrategy } from '@nestjs/passport';
 import { ExtractJwt, Strategy as JwtPassportStrategy } from 'passport-jwt';
 import { JwtDataSchema } from 'picsur-shared/dist/dto/jwt.dto';
 import { EUser } from 'picsur-shared/dist/entities/user.entity';
-import { ThrowIfFailed } from 'picsur-shared/dist/types/failable';
+import { ThrowIfFailed } from 'picsur-shared/dist/types';
 import { UserDbService } from '../../../collections/user-db/user-db.service';
 import { EUserBackend2EUser } from '../../../models/transformers/user.transformer';

--- a/backend/src/managers/auth/guards/local-auth.strategy.ts
+++ b/backend/src/managers/auth/guards/local-auth.strategy.ts
@ -2,10 +2,7 @@ import { Injectable } from '@nestjs/common';
 import { PassportStrategy } from '@nestjs/passport';
 import { Strategy } from 'passport-local';
 import { EUser } from 'picsur-shared/dist/entities/user.entity';
-import {
-  AsyncFailable,
-  ThrowIfFailed,
-} from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, ThrowIfFailed } from 'picsur-shared/dist/types';
 import { UserDbService } from '../../../collections/user-db/user-db.service';
 import { EUserBackend2EUser } from '../../../models/transformers/user.transformer';

--- a/backend/src/managers/auth/guards/main.guard.ts
+++ b/backend/src/managers/auth/guards/main.guard.ts
@ -9,7 +9,7 @@ import {
  FT,
  HasFailed,
  ThrowIfFailed,
-} from 'picsur-shared/dist/types/failable';
+} from 'picsur-shared/dist/types';
 import { makeUnique } from 'picsur-shared/dist/util/unique';
 import { UserDbService } from '../../../collections/user-db/user-db.service';
 import { Permissions } from '../../../models/constants/permissions.const';
--- a/backend/src/managers/auth/guest.service.ts
+++ b/backend/src/managers/auth/guest.service.ts
@ -1,5 +1,5 @@
 import { Injectable } from '@nestjs/common';
-import { HasFailed } from 'picsur-shared/dist/types/failable';
+import { HasFailed } from 'picsur-shared/dist/types';
 import { UserDbService } from '../../collections/user-db/user-db.service';
 import { EUserBackend } from '../../database/entities/users/user.entity';

--- a/backend/src/managers/image/convert.consumer.ts
+++ b/backend/src/managers/image/convert.consumer.ts
@ -0,0 +1,97 @@
+import { OnQueueError, OnQueueFailed, Process, Processor } from '@nestjs/bull';
+import { Logger } from '@nestjs/common';
+import type { Job } from 'bull';
+import { ImageRequestParams } from 'picsur-shared/dist/dto/api/image.dto';
+import { SysPreference } from 'picsur-shared/dist/dto/sys-preferences.enum';
+import { IsFailure, ThrowIfFailed } from 'picsur-shared/dist/types';
+import { ParseFileType } from 'picsur-shared/dist/util/parse-mime';
+import { ImageFileDBService } from '../../collections/image-db/image-file-db.service';
+import { SysPreferenceDbService } from '../../collections/preference-db/sys-preference-db.service';
+import { ImageConverterService } from './image-converter.service';
+import { ImageManagerService } from './image-manager.service';
+import { ImageConvertQueueID } from './image.queue';
+
+// This contains the job to convert an image to a derivative and store it
+
+export interface ImageConvertJobData {
+  uniqueKey: string;
+  imageId: string;
+  fileType: string;
+  options: ImageRequestParams;
+}
+export type ImageConvertJob = Job<ImageConvertJobData>;
+
+@Processor(ImageConvertQueueID)
+export class ConvertConsumer {
+  private readonly logger = new Logger(ConvertConsumer.name);
+
+  constructor(
+    private readonly imageFilesService: ImageFileDBService,
+    private readonly imageConverter: ImageConverterService,
+    private readonly sysPref: SysPreferenceDbService,
+    private readonly imageService: ImageManagerService,
+  ) {}
+
+  @Process()
+  async convertImage(job: ImageConvertJob): Promise<void> {
+    const { imageId, fileType, options, uniqueKey } = job.data;
+
+    // Get file type
+    const targetFileType = ThrowIfFailed(ParseFileType(fileType));
+
+    // Get preferences
+    const allow_editing = ThrowIfFailed(
+      await this.sysPref.getBooleanPreference(SysPreference.AllowEditing),
+    );
+
+    // Get master image
+    const masterImage = ThrowIfFailed(
+      await this.imageService.getMaster(imageId),
+    );
+    const masterImageData = ThrowIfFailed(
+      await this.imageService.getData(masterImage),
+    );
+    const sourceFileType = ThrowIfFailed(ParseFileType(masterImage.filetype));
+
+    // Conver timage
+    const startTime = Date.now();
+    const convertResult = ThrowIfFailed(
+      await this.imageConverter.convert(
+        masterImageData,
+        sourceFileType,
+        targetFileType,
+        allow_editing ? options : {},
+      ),
+    );
+
+    this.logger.verbose(
+      `Converted ${imageId} from ${sourceFileType.identifier} to ${
+        targetFileType.identifier
+      } in ${Date.now() - startTime}ms`,
+    );
+
+    ThrowIfFailed(
+      await this.imageFilesService.addDerivative(
+        imageId,
+        uniqueKey,
+        convertResult.filetype,
+        convertResult.image,
+      ),
+    );
+  }
+
+  @OnQueueError()
+  async handleError(error: any) {
+    if (IsFailure(error)) error.print(this.logger);
+    else this.logger.error(error);
+  }
+
+  @OnQueueFailed()
+  async handleFailed(job: Job, error: any) {
+    if (IsFailure(error))
+      error.print(this.logger, {
+        prefix: `[JOB ${job.id}]`,
+      });
+    else this.logger.error(error);
+  }
+}
--- a/backend/src/managers/image/convert.service.ts
+++ b/backend/src/managers/image/convert.service.ts
@ -0,0 +1,103 @@
+import { InjectQueue } from '@nestjs/bull';
+import { Injectable } from '@nestjs/common';
+import Crypto from 'crypto';
+import { ImageRequestParams } from 'picsur-shared/dist/dto/api/image.dto';
+import {
+  AsyncFailable,
+  Fail,
+  FT,
+  HasFailed,
+  ThrowIfFailed,
+} from 'picsur-shared/dist/types';
+import { ImageFileDBService } from '../../collections/image-db/image-file-db.service';
+import { EImageDerivativeBackend } from '../../database/entities/images/image-derivative.entity';
+import { ImageConvertJob } from './convert.consumer';
+import * as ImageQueue from './image.queue';
+
+@Injectable()
+export class ConvertService {
+  constructor(
+    @InjectQueue(ImageQueue.ImageConvertQueueID)
+    private readonly imageQueue: ImageQueue.ImageConvertQueue,
+    private readonly imageFilesService: ImageFileDBService,
+  ) {}
+
+  public async convertJob(
+    imageId: string,
+    fileType: string,
+    options: ImageRequestParams,
+  ): AsyncFailable<ImageConvertJob> {
+    const jobID = this.getConvertHash(imageId, { fileType, ...options });
+
+    /*
+    Jobs with the same ID don't get executed, we abuse this by passing it a hash of the input parameters.
+    This way, if the same image is requested with the same parameters, we don't have to convert it again.
+    Since it will always produce the same output with the same inputs
+    */
+
+    let job: ImageConvertJob;
+    try {
+      job = (await this.imageQueue.add(
+        {
+          imageId,
+          fileType,
+          options,
+          uniqueKey: jobID,
+        },
+        {
+          jobId: jobID,
+        },
+      )) as ImageConvertJob;
+    } catch (e) {
+      return Fail(FT.Internal, e);
+    }
+
+    if (!job.id) return Fail(FT.Internal, undefined, 'Failed to queue job');
+    return job;
+  }
+
+  public async convertPromise(
+    imageId: string,
+    fileType: string,
+    options: ImageRequestParams,
+  ): AsyncFailable<EImageDerivativeBackend> {
+    const uniqueKey = this.getConvertHash(imageId, { fileType, ...options });
+
+    const startime = Date.now();
+    const findExisting = ThrowIfFailed(
+      await this.imageFilesService.getDerivative(imageId, uniqueKey),
+    );
+    if (findExisting !== null) {
+      console.log('Found existing derivative in ' + (Date.now() - startime));
+      return findExisting;
+    }
+
+    const job = await this.convertJob(imageId, fileType, options);
+    if (HasFailed(job)) return job;
+
+    try {
+      await job.finished();
+    } catch (e) {
+      return Fail(FT.Internal, 'Failed to convert image', e);
+    }
+
+    const findResult = ThrowIfFailed(
+      await this.imageFilesService.getDerivative(imageId, uniqueKey),
+    );
+    if (findResult !== null) {
+      console.log('Found new derivative');
+      return findResult;
+    }
+
+    return Fail(FT.Internal, 'Failed to convert image');
+  }
+
+  private getConvertHash(imageID: string, options: object) {
+    // Return a sha256 hash of the stringified options
+    const stringified = JSON.stringify(options) + '-' + imageID;
+    const hash = Crypto.createHash('sha256');
+    hash.update(stringified);
+    const digest = hash.digest('hex');
+    return digest;
+  }
+}
--- a/backend/src/managers/image/image-converter.service.ts
+++ b/backend/src/managers/image/image-converter.service.ts
@ -6,12 +6,7 @@ import {
  SupportedFileTypeCategory,
 } from 'picsur-shared/dist/dto/mimes.dto';
 import { SysPreference } from 'picsur-shared/dist/dto/sys-preferences.enum';
-import {
-  AsyncFailable,
-  Fail,
-  FT,
-  HasFailed,
-} from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, Fail, FT, HasFailed } from 'picsur-shared/dist/types';
 import { SharpOptions } from 'sharp';
 import { SysPreferenceDbService } from '../../collections/preference-db/sys-preference-db.service';
 import { SharpWrapper } from '../../workers/sharp.wrapper';
--- a/backend/src/managers/image/image-manager.module.ts
+++ b/backend/src/managers/image/image-manager.module.ts
@ -1,25 +1,51 @@
+import { BullModule } from '@nestjs/bull';
 import { Logger, Module, OnModuleInit } from '@nestjs/common';
 import { Interval } from '@nestjs/schedule';
 import ms from 'ms';
 import { SysPreference } from 'picsur-shared/dist/dto/sys-preferences.enum';
-import { HasFailed } from 'picsur-shared/dist/types/failable';
+import { HasFailed } from 'picsur-shared/dist/types';
 import { ImageDBModule } from '../../collections/image-db/image-db.module';
 import { ImageDBService } from '../../collections/image-db/image-db.service';
 import { ImageFileDBService } from '../../collections/image-db/image-file-db.service';
 import { PreferenceDbModule } from '../../collections/preference-db/preference-db.module';
 import { SysPreferenceDbService } from '../../collections/preference-db/sys-preference-db.service';
+import { ConvertConsumer } from './convert.consumer';
+import { ConvertService } from './convert.service';
 import { ImageConverterService } from './image-converter.service';
-import { ImageProcessorService } from './image-processor.service';
-import { ImageManagerService } from './image.service';
+import { ImageManagerService } from './image-manager.service';
+import { ImageConvertQueueID, ImageIngestQueueID } from './image.queue';
+import { IngestConsumer } from './ingest.consumer';
+import { IngestService } from './ingest.service';

@Module({
-  imports: [ImageDBModule, PreferenceDbModule],
+  imports: [
+    ImageDBModule,
+    PreferenceDbModule,
+    BullModule.registerQueue({
+      name: ImageConvertQueueID,
+      defaultJobOptions: {
+        removeOnComplete: true,
+        removeOnFail: true,
+      },
+    }),
+    BullModule.registerQueue({
+      name: ImageIngestQueueID,
+    }),
+  ],
  providers: [
    ImageManagerService,
-    ImageProcessorService,
    ImageConverterService,
+    IngestConsumer,
+    ConvertConsumer,
+    IngestService,
+    ConvertService,
+  ],
+  exports: [
+    ImageManagerService,
+    ImageConverterService,
+    IngestService,
+    ConvertService,
  ],
-  exports: [ImageManagerService, ImageConverterService],
 })
 export class ImageManagerModule implements OnModuleInit {
  private readonly logger = new Logger(ImageManagerModule.name);
@ -38,6 +64,7 @@ export class ImageManagerModule implements OnModuleInit {
  private async imageManagerCron() {
    await this.cleanupDerivatives();
    await this.cleanupExpired();
+    await this.cleanupOrphanedFiles();
  }

  private async cleanupDerivatives() {
@ -75,4 +102,25 @@ export class ImageManagerModule implements OnModuleInit {
    if (cleanedUp > 0)
      this.logger.log(`Cleaned up ${cleanedUp} expired images`);
  }
+
+  private async cleanupOrphanedFiles() {
+    const cleanedUpDerivatives =
+      await this.imageFileDB.cleanupOrphanedDerivatives();
+
+    if (HasFailed(cleanedUpDerivatives)) {
+      cleanedUpDerivatives.print(this.logger);
+      return;
+    }
+
+    const cleanedUpFiles = await this.imageFileDB.cleanupOrphanedFiles();
+    if (HasFailed(cleanedUpFiles)) {
+      cleanedUpFiles.print(this.logger);
+      return;
+    }
+
+    if (cleanedUpDerivatives > 0 || cleanedUpFiles > 0)
+      this.logger.log(
+        `Cleaned up ${cleanedUpDerivatives} orphaned derivatives and ${cleanedUpFiles} orphaned files`,
+      );
+  }
 }
--- a/backend/src/managers/image/image-manager.service.ts
+++ b/backend/src/managers/image/image-manager.service.ts
@ -0,0 +1,111 @@
+import { Injectable, Logger } from '@nestjs/common';
+import { ImageEntryVariant } from 'picsur-shared/dist/dto/image-entry-variant.enum';
+import { FileType } from 'picsur-shared/dist/dto/mimes.dto';
+import { AsyncFailable, Fail, FT, HasFailed } from 'picsur-shared/dist/types';
+import { FindResult } from 'picsur-shared/dist/types/find-result';
+import { ParseFileType } from 'picsur-shared/dist/util/parse-mime';
+import { ImageDBService } from '../../collections/image-db/image-db.service';
+import { ImageFileDBService } from '../../collections/image-db/image-file-db.service';
+import { EImageDerivativeBackend } from '../../database/entities/images/image-derivative.entity';
+import { EImageFileBackend } from '../../database/entities/images/image-file.entity';
+import { EImageBackend } from '../../database/entities/images/image.entity';
+
+@Injectable()
+export class ImageManagerService {
+  private readonly logger = new Logger(ImageManagerService.name);
+
+  constructor(
+    private readonly imagesService: ImageDBService,
+    private readonly imageFilesService: ImageFileDBService,
+  ) {}
+
+  public async findOne(id: string): AsyncFailable<EImageBackend> {
+    return await this.imagesService.findOne(id, undefined);
+  }
+
+  public async findMany(
+    count: number,
+    page: number,
+    userid: string | undefined,
+  ): AsyncFailable<FindResult<EImageBackend>> {
+    return await this.imagesService.findMany(count, page, userid);
+  }
+
+  public async update(
+    id: string,
+    userid: string | undefined,
+    options: Partial<Pick<EImageBackend, 'file_name' | 'expires_at'>>,
+  ): AsyncFailable<EImageBackend> {
+    if (options.expires_at !== undefined && options.expires_at !== null) {
+      if (options.expires_at < new Date()) {
+        return Fail(FT.UsrValidation, 'Expiration date must be in the future');
+      }
+    }
+    return await this.imagesService.update(id, userid, options);
+  }
+
+  public async deleteMany(
+    ids: string[],
+    userid: string | undefined,
+  ): AsyncFailable<EImageBackend[]> {
+    return await this.imagesService.delete(ids, userid);
+  }
+
+  public async deleteWithKey(
+    imageId: string,
+    key: string,
+  ): AsyncFailable<EImageBackend> {
+    return await this.imagesService.deleteWithKey(imageId, key);
+  }
+
+  // File getters ==============================================================
+
+  public async getMaster(imageId: string): AsyncFailable<EImageFileBackend> {
+    return this.imageFilesService.getFile(imageId, ImageEntryVariant.MASTER);
+  }
+
+  public async getMasterFileType(imageId: string): AsyncFailable<FileType> {
+    const mime = await this.imageFilesService.getFileTypes(imageId);
+    if (HasFailed(mime)) return mime;
+
+    if (mime['master'] === undefined)
+      return Fail(FT.NotFound, 'No master file');
+
+    return ParseFileType(mime['master']);
+  }
+
+  public async getOriginal(imageId: string): AsyncFailable<EImageFileBackend> {
+    return this.imageFilesService.getFile(imageId, ImageEntryVariant.ORIGINAL);
+  }
+
+  public async getOriginalFileType(imageId: string): AsyncFailable<FileType> {
+    const filetypes = await this.imageFilesService.getFileTypes(imageId);
+    if (HasFailed(filetypes)) return filetypes;
+
+    if (filetypes['original'] === undefined)
+      return Fail(FT.NotFound, 'No original file');
+
+    return ParseFileType(filetypes['original']);
+  }
+
+  public async getData(image: EImageFileBackend | EImageDerivativeBackend) {
+    return await this.imageFilesService.getData(image);
+  }
+
+  public async getFileMimes(imageId: string): AsyncFailable<{
+    [ImageEntryVariant.MASTER]: string;
+    [ImageEntryVariant.ORIGINAL]: string | undefined;
+  }> {
+    const result = await this.imageFilesService.getFileTypes(imageId);
+    if (HasFailed(result)) return result;
+
+    if (result[ImageEntryVariant.MASTER] === undefined) {
+      return Fail(FT.NotFound, 'No master file found');
+    }
+
+    return {
+      [ImageEntryVariant.MASTER]: result[ImageEntryVariant.MASTER]!,
+      [ImageEntryVariant.ORIGINAL]: result[ImageEntryVariant.ORIGINAL],
+    };
+  }
+}
--- a/backend/src/managers/image/image-processor.service.ts
+++ b/backend/src/managers/image/image-processor.service.ts
@ -1,55 +0,0 @@
-import { Injectable } from '@nestjs/common';
-import {
-  FileType,
-  ImageFileType,
-  SupportedFileTypeCategory,
-} from 'picsur-shared/dist/dto/mimes.dto';
-
-import {
-  AsyncFailable,
-  Fail,
-  FT,
-  HasFailed,
-} from 'picsur-shared/dist/types/failable';
-import { ParseFileType } from 'picsur-shared/dist/util/parse-mime';
-import { ImageConverterService } from './image-converter.service';
-import { ImageResult } from './imageresult';
-
-@Injectable()
-export class ImageProcessorService {
-  constructor(private readonly imageConverter: ImageConverterService) {}
-
-  public async process(
-    image: Buffer,
-    filetype: FileType,
-  ): AsyncFailable<ImageResult> {
-    if (filetype.category === SupportedFileTypeCategory.Image) {
-      return await this.processStill(image, filetype);
-    } else if (filetype.category === SupportedFileTypeCategory.Animation) {
-      return await this.processAnimation(image, filetype);
-    } else {
-      return Fail(FT.SysValidation, 'Unsupported mime type');
-    }
-  }
-
-  private async processStill(
-    image: Buffer,
-    filetype: FileType,
-  ): AsyncFailable<ImageResult> {
-    const outputFileType = ParseFileType(ImageFileType.QOI);
-    if (HasFailed(outputFileType)) return outputFileType;
-
-    return this.imageConverter.convert(image, filetype, outputFileType, {});
-  }
-
-  private async processAnimation(
-    image: Buffer,
-    filetype: FileType,
-  ): AsyncFailable<ImageResult> {
-    // Webps and gifs are stored as is for now
-    return {
-      image: image,
-      filetype: filetype.identifier,
-    };
-  }
-}
--- a/backend/src/managers/image/image.queue.ts
+++ b/backend/src/managers/image/image.queue.ts
@ -0,0 +1,9 @@
+import { Queue } from 'bull';
+import { ImageConvertJobData } from './convert.consumer';
+import { ImageIngestJobData } from './ingest.consumer';
+
+export const ImageConvertQueueID = 'image-convert-queue';
+export const ImageIngestQueueID = 'image-ingest-queue';
+
+export type ImageConvertQueue = Queue<ImageConvertJobData>;
+export type ImageIngestQueue = Queue<ImageIngestJobData>;
--- a/backend/src/managers/image/image.service.ts
+++ b/backend/src/managers/image/image.service.ts
@ -1,281 +0,0 @@
-import { Injectable, Logger } from '@nestjs/common';
-import { createHash } from 'crypto';
-import { fileTypeFromBuffer, FileTypeResult } from 'file-type';
-import { ImageRequestParams } from 'picsur-shared/dist/dto/api/image.dto';
-import { ImageEntryVariant } from 'picsur-shared/dist/dto/image-entry-variant.enum';
-import {
-  AnimFileType,
-  FileType,
-  ImageFileType,
-  Mime2FileType,
-} from 'picsur-shared/dist/dto/mimes.dto';
-import { SysPreference } from 'picsur-shared/dist/dto/sys-preferences.enum';
-import { UsrPreference } from 'picsur-shared/dist/dto/usr-preferences.enum';
-import {
-  AsyncFailable,
-  Fail,
-  FT,
-  HasFailed,
-} from 'picsur-shared/dist/types/failable';
-import { FindResult } from 'picsur-shared/dist/types/find-result';
-import { ParseFileType } from 'picsur-shared/dist/util/parse-mime';
-import { IsQOI } from 'qoi-img';
-import { ImageDBService } from '../../collections/image-db/image-db.service';
-import { ImageFileDBService } from '../../collections/image-db/image-file-db.service';
-import { SysPreferenceDbService } from '../../collections/preference-db/sys-preference-db.service';
-import { UsrPreferenceDbService } from '../../collections/preference-db/usr-preference-db.service';
-import { EImageDerivativeBackend } from '../../database/entities/images/image-derivative.entity';
-import { EImageFileBackend } from '../../database/entities/images/image-file.entity';
-import { EImageBackend } from '../../database/entities/images/image.entity';
-import { MutexFallBack } from '../../util/mutex-fallback';
-import { ImageConverterService } from './image-converter.service';
-import { ImageProcessorService } from './image-processor.service';
-import { WebPInfo } from './webpinfo/webpinfo';
-
-@Injectable()
-export class ImageManagerService {
-  private readonly logger = new Logger(ImageManagerService.name);
-
-  constructor(
-    private readonly imagesService: ImageDBService,
-    private readonly imageFilesService: ImageFileDBService,
-    private readonly processService: ImageProcessorService,
-    private readonly convertService: ImageConverterService,
-    private readonly userPref: UsrPreferenceDbService,
-    private readonly sysPref: SysPreferenceDbService,
-  ) {}
-
-  public async findOne(id: string): AsyncFailable<EImageBackend> {
-    return await this.imagesService.findOne(id, undefined);
-  }
-
-  public async findMany(
-    count: number,
-    page: number,
-    userid: string | undefined,
-  ): AsyncFailable<FindResult<EImageBackend>> {
-    return await this.imagesService.findMany(count, page, userid);
-  }
-
-  public async update(
-    id: string,
-    userid: string | undefined,
-    options: Partial<Pick<EImageBackend, 'file_name' | 'expires_at'>>,
-  ): AsyncFailable<EImageBackend> {
-    if (options.expires_at !== undefined && options.expires_at !== null) {
-      if (options.expires_at < new Date()) {
-        return Fail(FT.UsrValidation, 'Expiration date must be in the future');
-      }
-    }
-    return await this.imagesService.update(id, userid, options);
-  }
-
-  public async deleteMany(
-    ids: string[],
-    userid: string | undefined,
-  ): AsyncFailable<EImageBackend[]> {
-    return await this.imagesService.delete(ids, userid);
-  }
-
-  public async deleteWithKey(
-    imageId: string,
-    key: string,
-  ): AsyncFailable<EImageBackend> {
-    return await this.imagesService.deleteWithKey(imageId, key);
-  }
-
-  public async upload(
-    userid: string,
-    filename: string,
-    image: Buffer,
-    withDeleteKey: boolean,
-  ): AsyncFailable<EImageBackend> {
-    const fileType = await this.getFileTypeFromBuffer(image);
-    if (HasFailed(fileType)) return fileType;
-
-    // Check if need to save orignal
-    const keepOriginal = await this.userPref.getBooleanPreference(
-      userid,
-      UsrPreference.KeepOriginal,
-    );
-    if (HasFailed(keepOriginal)) return keepOriginal;
-
-    // Process
-    const processResult = await this.processService.process(image, fileType);
-    if (HasFailed(processResult)) return processResult;
-
-    // Strip extension from filename
-    const name = (() => {
-      const index = filename.lastIndexOf('.');
-      if (index === -1) return filename;
-      return filename.substring(0, index);
-    })();
-
-    // Save processed to db
-    const imageEntity = await this.imagesService.create(
-      userid,
-      name,
-      withDeleteKey,
-    );
-    if (HasFailed(imageEntity)) return imageEntity;
-
-    const imageFileEntity = await this.imageFilesService.setFile(
-      imageEntity.id,
-      ImageEntryVariant.MASTER,
-      processResult.image,
-      processResult.filetype,
-    );
-    if (HasFailed(imageFileEntity)) return imageFileEntity;
-
-    if (keepOriginal) {
-      const originalFileEntity = await this.imageFilesService.setFile(
-        imageEntity.id,
-        ImageEntryVariant.ORIGINAL,
-        image,
-        fileType.identifier,
-      );
-      if (HasFailed(originalFileEntity)) return originalFileEntity;
-    }
-
-    return imageEntity;
-  }
-
-  public async getConverted(
-    imageId: string,
-    fileType: string,
-    options: ImageRequestParams,
-  ): AsyncFailable<EImageDerivativeBackend> {
-    const targetFileType = ParseFileType(fileType);
-    if (HasFailed(targetFileType)) return targetFileType;
-
-    const converted_key = this.getConvertHash({ mime: fileType, ...options });
-
-    const allow_editing = await this.sysPref.getBooleanPreference(
-      SysPreference.AllowEditing,
-    );
-    if (HasFailed(allow_editing)) return allow_editing;
-
-    return MutexFallBack(
-      converted_key,
-      () => {
-        return this.imageFilesService.getDerivative(imageId, converted_key);
-      },
-      async () => {
-        const masterImage = await this.getMaster(imageId);
-        if (HasFailed(masterImage)) return masterImage;
-
-        const sourceFileType = ParseFileType(masterImage.filetype);
-        if (HasFailed(sourceFileType)) return sourceFileType;
-
-        const startTime = Date.now();
-        const convertResult = await this.convertService.convert(
-          masterImage.data,
-          sourceFileType,
-          targetFileType,
-          allow_editing ? options : {},
-        );
-        if (HasFailed(convertResult)) return convertResult;
-
-        this.logger.verbose(
-          `Converted ${imageId} from ${sourceFileType.identifier} to ${
-            targetFileType.identifier
-          } in ${Date.now() - startTime}ms`,
-        );
-
-        return await this.imageFilesService.addDerivative(
-          imageId,
-          converted_key,
-          convertResult.filetype,
-          convertResult.image,
-        );
-      },
-    );
-  }
-
-  // File getters ==============================================================
-
-  public async getMaster(imageId: string): AsyncFailable<EImageFileBackend> {
-    return this.imageFilesService.getFile(imageId, ImageEntryVariant.MASTER);
-  }
-
-  public async getMasterFileType(imageId: string): AsyncFailable<FileType> {
-    const mime = await this.imageFilesService.getFileTypes(imageId);
-    if (HasFailed(mime)) return mime;
-
-    if (mime['master'] === undefined)
-      return Fail(FT.NotFound, 'No master file');
-
-    return ParseFileType(mime['master']);
-  }
-
-  public async getOriginal(imageId: string): AsyncFailable<EImageFileBackend> {
-    return this.imageFilesService.getFile(imageId, ImageEntryVariant.ORIGINAL);
-  }
-
-  public async getOriginalFileType(imageId: string): AsyncFailable<FileType> {
-    const filetypes = await this.imageFilesService.getFileTypes(imageId);
-    if (HasFailed(filetypes)) return filetypes;
-
-    if (filetypes['original'] === undefined)
-      return Fail(FT.NotFound, 'No original file');
-
-    return ParseFileType(filetypes['original']);
-  }
-
-  public async getFileMimes(imageId: string): AsyncFailable<{
-    [ImageEntryVariant.MASTER]: string;
-    [ImageEntryVariant.ORIGINAL]: string | undefined;
-  }> {
-    const result = await this.imageFilesService.getFileTypes(imageId);
-    if (HasFailed(result)) return result;
-
-    if (result[ImageEntryVariant.MASTER] === undefined) {
-      return Fail(FT.NotFound, 'No master file found');
-    }
-
-    return {
-      [ImageEntryVariant.MASTER]: result[ImageEntryVariant.MASTER],
-      [ImageEntryVariant.ORIGINAL]: result[ImageEntryVariant.ORIGINAL],
-    };
-  }
-
-  // Util stuff ==================================================================
-
-  private async getFileTypeFromBuffer(image: Buffer): AsyncFailable<FileType> {
-    const filetypeResult: FileTypeResult | undefined = await fileTypeFromBuffer(
-      image,
-    );
-
-    let mime: string | undefined;
-    if (filetypeResult === undefined) {
-      if (IsQOI(image)) mime = 'image/x-qoi';
-    } else {
-      mime = filetypeResult.mime;
-    }
-
-    if (mime === undefined) mime = 'other/unknown';
-
-    let filetype: string | undefined;
-    if (mime === 'image/webp') {
-      const header = await WebPInfo.from(image);
-      if (header.summary.isAnimated) filetype = AnimFileType.WEBP;
-      else filetype = ImageFileType.WEBP;
-    }
-    if (filetype === undefined) {
-      const parsed = Mime2FileType(mime);
-      if (HasFailed(parsed)) return parsed;
-      filetype = parsed;
-    }
-
-    return ParseFileType(filetype);
-  }
-
-  private getConvertHash(options: object) {
-    // Return a sha256 hash of the stringified options
-    const stringified = JSON.stringify(options);
-    const hash = createHash('sha256');
-    hash.update(stringified);
-    const digest = hash.digest('hex');
-    return digest;
-  }
-}
--- a/backend/src/managers/image/ingest.consumer.ts
+++ b/backend/src/managers/image/ingest.consumer.ts
@ -0,0 +1,140 @@
+import { OnQueueError, OnQueueFailed, Process, Processor } from '@nestjs/bull';
+import { Logger } from '@nestjs/common';
+import type { Job } from 'bull';
+import { ImageEntryVariant } from 'picsur-shared/dist/dto/image-entry-variant.enum';
+import {
+  FileType,
+  ImageFileType,
+  SupportedFileTypeCategory,
+} from 'picsur-shared/dist/dto/mimes.dto';
+import {
+  AsyncFailable,
+  Fail,
+  FT,
+  HasFailed,
+  IsFailure,
+  ThrowIfFailed,
+} from 'picsur-shared/dist/types';
+import { ParseFileType } from 'picsur-shared/dist/util/parse-mime';
+import { ImageDBService } from '../../collections/image-db/image-db.service';
+import { ImageFileDBService } from '../../collections/image-db/image-file-db.service';
+import { EImageBackend } from '../../database/entities/images/image.entity';
+import { ImageConverterService } from '../image/image-converter.service';
+import { ImageResult } from '../image/imageresult';
+import { ImageIngestQueueID } from './image.queue';
+
+export interface ImageIngestJobData {
+  imageID: string;
+  storeOriginal: boolean;
+}
+export type ImageIngestJob = Job<ImageIngestJobData>;
+
+@Processor(ImageIngestQueueID)
+export class IngestConsumer {
+  private readonly logger = new Logger(IngestConsumer.name);
+
+  constructor(
+    private readonly imagesService: ImageDBService,
+    private readonly imageFilesService: ImageFileDBService,
+    private readonly imageConverter: ImageConverterService,
+  ) {}
+
+  @Process({
+    concurrency: 5,
+  })
+  async ingestImage(job: ImageIngestJob): Promise<EImageBackend> {
+    const { imageID, storeOriginal } = job.data;
+
+    // Already start the query for the image, we only need it when returning
+    const imagePromise = this.imagesService.findOne(imageID, undefined);
+
+    this.logger.verbose(
+      `Ingesting image ${imageID} and store original: ${storeOriginal}`,
+    );
+
+    const ingestFile = ThrowIfFailed(
+      await this.imageFilesService.getFile(imageID, ImageEntryVariant.INGEST),
+    );
+    const ingestFileData = ThrowIfFailed(
+      await this.imageFilesService.getData(ingestFile),
+    );
+    const ingestFiletype = ThrowIfFailed(ParseFileType(ingestFile.filetype));
+
+    const processed = ThrowIfFailed(
+      await this.process(ingestFileData, ingestFiletype),
+    );
+
+    const masterPromise = this.imageFilesService.setFile(
+      imageID,
+      ImageEntryVariant.MASTER,
+      processed.image,
+      processed.filetype,
+    );
+
+    const originalPromise = storeOriginal
+      ? this.imageFilesService.migrateFile(
+          imageID,
+          ImageEntryVariant.INGEST,
+          ImageEntryVariant.ORIGINAL,
+        )
+      : this.imageFilesService.orphanFile(imageID, ImageEntryVariant.INGEST);
+
+    const results = await Promise.all([masterPromise, originalPromise]);
+    results.map((r) => ThrowIfFailed(r));
+
+    const image = ThrowIfFailed(await imagePromise);
+
+    this.logger.verbose(`Ingested image ${imageID}`);
+
+    return image;
+  }
+
+  private async process(
+    image: Buffer,
+    filetype: FileType,
+  ): AsyncFailable<ImageResult> {
+    if (filetype.category === SupportedFileTypeCategory.Image) {
+      return await this.processStill(image, filetype);
+    } else if (filetype.category === SupportedFileTypeCategory.Animation) {
+      return await this.processAnimation(image, filetype);
+    } else {
+      return Fail(FT.SysValidation, 'Unsupported mime type');
+    }
+  }
+
+  private async processStill(
+    image: Buffer,
+    filetype: FileType,
+  ): AsyncFailable<ImageResult> {
+    const outputFileType = ParseFileType(ImageFileType.QOI);
+    if (HasFailed(outputFileType)) return outputFileType;
+
+    return this.imageConverter.convert(image, filetype, outputFileType, {});
+  }
+
+  private async processAnimation(
+    image: Buffer,
+    filetype: FileType,
+  ): AsyncFailable<ImageResult> {
+    // Webps and gifs are stored as is for now
+    return {
+      image: image,
+      filetype: filetype.identifier,
+    };
+  }
+
+  @OnQueueError()
+  async handleError(error: any) {
+    if (IsFailure(error)) error.print(this.logger);
+    else this.logger.error(error);
+  }
+
+  @OnQueueFailed()
+  async handleFailed(job: Job, error: any) {
+    if (IsFailure(error))
+      error.print(this.logger, {
+        prefix: `[JOB ${job.id}]`,
+      });
+    else this.logger.error(error);
+  }
+}
--- a/backend/src/managers/image/ingest.service.ts
+++ b/backend/src/managers/image/ingest.service.ts
@ -0,0 +1,177 @@
+import { InjectQueue } from '@nestjs/bull';
+import { Injectable, Logger } from '@nestjs/common';
+import { fileTypeFromBuffer, FileTypeResult } from 'file-type';
+import { ImageEntryVariant } from 'picsur-shared/dist/dto/image-entry-variant.enum';
+import {
+  AnimFileType,
+  FileType,
+  ImageFileType,
+  Mime2FileType,
+} from 'picsur-shared/dist/dto/mimes.dto';
+import { UsrPreference } from 'picsur-shared/dist/dto/usr-preferences.enum';
+import { AsyncFailable, Fail, FT, HasFailed } from 'picsur-shared/dist/types';
+import { ParseFileType } from 'picsur-shared/dist/util/parse-mime';
+import { IsQOI } from 'qoi-img';
+import { ImageDBService } from '../../collections/image-db/image-db.service';
+import { ImageFileDBService } from '../../collections/image-db/image-file-db.service';
+import { UsrPreferenceDbService } from '../../collections/preference-db/usr-preference-db.service';
+import { EImageBackend } from '../../database/entities/images/image.entity';
+import { WebPInfo } from '../image/webpinfo/webpinfo';
+import * as ImageQueue from './image.queue';
+import { ImageIngestJob } from './ingest.consumer';
+import { v4 as uuidv4 } from 'uuid';
+
+@Injectable()
+export class IngestService {
+  private readonly logger = new Logger(IngestService.name);
+
+  constructor(
+    @InjectQueue(ImageQueue.ImageIngestQueueID)
+    private readonly imageQueue: ImageQueue.ImageIngestQueue,
+    private readonly imagesService: ImageDBService,
+    private readonly imageFilesService: ImageFileDBService,
+    private readonly userPref: UsrPreferenceDbService,
+  ) {}
+
+  public async uploadJob(
+    userid: string,
+    filename: string,
+    image: Buffer,
+    withDeleteKey: boolean,
+  ): AsyncFailable<[ImageIngestJob, EImageBackend]> {
+    const fileType = await this.getFileTypeFromBuffer(image);
+    if (HasFailed(fileType)) return fileType;
+
+    // Check if need to save orignal
+    const keepOriginal = await this.userPref.getBooleanPreference(
+      userid,
+      UsrPreference.KeepOriginal,
+    );
+    if (HasFailed(keepOriginal)) return keepOriginal;
+
+    // Strip extension from filename
+    const name = (() => {
+      const index = filename.lastIndexOf('.');
+      if (index === -1) return filename;
+      return filename.substring(0, index);
+    })();
+
+    // Save unprocessed image to be processed by worker
+    const imageEntity = await this.imagesService.create(
+      userid,
+      name,
+      withDeleteKey,
+    );
+    if (HasFailed(imageEntity)) return imageEntity;
+
+    {
+      const imageFileEntity = await this.imageFilesService.setFile(
+        imageEntity.id,
+        ImageEntryVariant.INGEST,
+        image,
+        fileType.identifier,
+      );
+      if (HasFailed(imageFileEntity)) return imageFileEntity;
+    }
+
+    try {
+      const job = (await this.imageQueue.add(
+        {
+          imageID: imageEntity.id,
+          storeOriginal: keepOriginal,
+        },
+        {
+          jobId: uuidv4(),
+        },
+      )) as ImageIngestJob;
+      if (!job.id) return Fail(FT.Internal, undefined, 'Failed to queue job');
+
+      return [job, imageEntity];
+    } catch (e) {
+      return Fail(FT.Internal, e);
+    }
+  }
+
+  public async uploadPromise(
+    userid: string,
+    filename: string,
+    image: Buffer,
+    withDeleteKey: boolean,
+  ): AsyncFailable<EImageBackend> {
+    const result = await this.uploadJob(userid, filename, image, withDeleteKey);
+    if (HasFailed(result)) return result;
+
+    const [job, imageEntity] = result;
+
+    try {
+      await job.finished();
+      return imageEntity;
+    } catch (e) {
+      return Fail(FT.Internal, 'Failed to process image', e);
+    }
+  }
+
+  public async getProgress(jobsIds: string[]): AsyncFailable<{
+    progress: number;
+    failed: string[];
+  }> {
+    try {
+      const jobs = await Promise.all(
+        jobsIds.map((id) => this.imageQueue.getJob(id)),
+      );
+
+      const cleanJobs: ImageIngestJob[] = jobs.filter(
+        (job) => job !== null,
+      ) as ImageIngestJob[];
+
+      if (cleanJobs.length === 0) return { progress: 1, failed: [] };
+
+      const statefulJobs = await Promise.all(
+        cleanJobs.map(async (job) => ({ job, state: await job.getState() })),
+      );
+
+      const progress =
+        statefulJobs.filter(
+          (job) => job.state === 'completed' || job.state === 'failed',
+        ).length / cleanJobs.length;
+
+      return {
+        progress,
+        failed: statefulJobs
+          .filter((job) => job.state === 'failed')
+          .map((job) => job.job.id.toString()),
+      };
+    } catch (e) {
+      return Fail(FT.Internal, e);
+    }
+  }
+
+  private async getFileTypeFromBuffer(image: Buffer): AsyncFailable<FileType> {
+    const filetypeResult: FileTypeResult | undefined = await fileTypeFromBuffer(
+      image,
+    );
+
+    let mime: string | undefined;
+    if (filetypeResult === undefined) {
+      if (IsQOI(image)) mime = 'image/x-qoi';
+    } else {
+      mime = filetypeResult.mime;
+    }
+
+    if (mime === undefined) mime = 'other/unknown';
+
+    let filetype: string | undefined;
+    if (mime === 'image/webp') {
+      const header = await WebPInfo.from(image);
+      if (header.summary.isAnimated) filetype = AnimFileType.WEBP;
+      else filetype = ImageFileType.WEBP;
+    }
+    if (filetype === undefined) {
+      const parsed = Mime2FileType(mime);
+      if (HasFailed(parsed)) return parsed;
+      filetype = parsed;
+    }
+
+    return ParseFileType(filetype);
+  }
+}
--- a/backend/src/managers/image/webpinfo/webpinfo.ts
+++ b/backend/src/managers/image/webpinfo/webpinfo.ts
@ -1,5 +1,3 @@
-/* eslint-disable @typescript-eslint/no-unused-vars */
-/* eslint-disable @typescript-eslint/no-empty-function */
 // @ts-nocheck

 /*
@ -312,8 +310,8 @@ export class WebPInfo extends StreamParserWritable {
    ['VP8', 'VP8L', 'ANMF'].map((t) => [t, true] as [ChunkType, true]),
  );

-  private offset = 0;
-  private maxSeekableOffset = -1; // same as file size - 1
+  private offset: number = 0;
+  private maxSeekableOffset: number = -1; // same as file size - 1

  private pending?: {
    size: number;
--- a/backend/src/managers/usage/usage.service.ts
+++ b/backend/src/managers/usage/usage.service.ts
@ -1,8 +1,8 @@
 import { Injectable, Logger } from '@nestjs/common';
 import isDocker from 'is-docker';
 import fetch from 'node-fetch';
-import * as os from 'os';
-import { FallbackIfFailed, HasFailed } from 'picsur-shared/dist/types/failable';
+import os from 'os';
+import { FallbackIfFailed, HasFailed } from 'picsur-shared/dist/types';
 import { UUIDRegex } from 'picsur-shared/dist/util/common-regex';
 import { ImageDBService } from '../../collections/image-db/image-db.service';
 import { SystemStateDbService } from '../../collections/system-state-db/system-state-db.service';
@ -23,7 +23,6 @@ interface UsageData {
  architecture: string;
  cpu_count: number;
  ram_total: number;
-  hostname: string;

  is_docker: boolean;
  is_production: boolean;
@ -151,7 +150,6 @@ export class UsageService {
      architecture: process.arch,
      cpu_count: os.cpus().length,
      ram_total: Math.floor(os.totalmem() / 1024 / 1024),
-      hostname: os.hostname(),
      is_docker: isDocker(),
      is_production: this.hostConfig.isProduction(),
    };
--- a/backend/src/models/constants/roles.const.ts
+++ b/backend/src/models/constants/roles.const.ts
@ -24,7 +24,7 @@ export const UndeletableRolesList: string[] = UndeletableRolesTuple;
 export const SystemRolesList = UndeletableRolesList;

 // Defaults
-type SystemRole = (typeof UndeletableRolesTuple)[number];
+type SystemRole = typeof UndeletableRolesTuple[number];
 const SystemRoleDefaultsTyped: {
  [key in SystemRole]: Permissions;
 } = {
--- a/backend/src/routes/api/apikeys/apikeys.controller.ts
+++ b/backend/src/routes/api/apikeys/apikeys.controller.ts
@ -12,7 +12,7 @@ import {
  ApiKeyUpdateResponse,
 } from 'picsur-shared/dist/dto/api/apikeys.dto';
 import { Permission } from 'picsur-shared/dist/dto/permissions.enum';
-import { ThrowIfFailed } from 'picsur-shared/dist/types/failable';
+import { ThrowIfFailed } from 'picsur-shared/dist/types';
 import { ApiKeyDbService } from '../../../collections/apikey-db/apikey-db.service';
 import {
  HasPermission,
--- a/backend/src/routes/api/experiment/experiment.controller.ts
+++ b/backend/src/routes/api/experiment/experiment.controller.ts
@ -1,16 +1,17 @@
-import { Controller } from '@nestjs/common';
+import { Controller, Get, Logger } from '@nestjs/common';
 import { NoPermissions } from '../../../decorators/permissions.decorator';
+import { ReturnsAnything } from '../../../decorators/returns.decorator';
+
@Controller('api/experiment')
@NoPermissions()
 export class ExperimentController {
-  // @Get()
-  // @Returns(UserInfoResponse)
-  // async testRoute(
-  //   @Request() req: AuthFastifyRequest,
-  //   @Response({ passthrough: true }) res: FastifyReply,
-  // ): Promise<UserInfoResponse> {
-  //   res.header('Location', '/error/delete-success');
-  //   res.code(302);
-  //   return req.user;
-  // }
+  private readonly logger = new Logger(ExperimentController.name);
+
+  constructor() {}
+
+  @Get()
+  @ReturnsAnything()
+  async testRoute(): Promise<any> {
+    return 'ok';
+  }
 }
--- a/backend/src/routes/api/experiment/experiment.module.ts
+++ b/backend/src/routes/api/experiment/experiment.module.ts
@ -1,11 +1,13 @@
 import { Module } from '@nestjs/common';
+import { PicsurLoggerModule } from '../../../logger/logger.module';
+import { ImageManagerModule } from '../../../managers/image/image-manager.module';
 import { ExperimentController } from './experiment.controller';

 // This is comletely useless module, but is used for testing
 // TODO: remove when out of beta

@Module({
-  imports: [],
+  imports: [ImageManagerModule, PicsurLoggerModule],
  controllers: [ExperimentController],
 })
 export class ExperimentModule {}
--- a/backend/src/routes/api/info/info.controller.ts
+++ b/backend/src/routes/api/info/info.controller.ts
@ -11,7 +11,7 @@ import {
  SupportedImageFileTypes,
 } from 'picsur-shared/dist/dto/mimes.dto';
 import { TrackingState } from 'picsur-shared/dist/dto/tracking-state.enum';
-import { FallbackIfFailed } from 'picsur-shared/dist/types/failable';
+import { FallbackIfFailed } from 'picsur-shared/dist/types';
 import { HostConfigService } from '../../../config/early/host.config.service';
 import { InfoConfigService } from '../../../config/late/info.config.service';
 import { UsageConfigService } from '../../../config/late/usage.config.service';
--- a/backend/src/routes/api/pref/sys-pref.controller.ts
+++ b/backend/src/routes/api/pref/sys-pref.controller.ts
@ -6,7 +6,7 @@ import {
  UpdatePreferenceRequest,
  UpdatePreferenceResponse,
 } from 'picsur-shared/dist/dto/api/pref.dto';
-import { ThrowIfFailed } from 'picsur-shared/dist/types/failable';
+import { ThrowIfFailed } from 'picsur-shared/dist/types';
 import { SysPreferenceDbService } from '../../../collections/preference-db/sys-preference-db.service';
 import { RequiredPermissions } from '../../../decorators/permissions.decorator';
 import { Returns } from '../../../decorators/returns.decorator';
--- a/backend/src/routes/api/pref/usr-pref.controller.ts
+++ b/backend/src/routes/api/pref/usr-pref.controller.ts
@ -6,7 +6,7 @@ import {
  UpdatePreferenceRequest,
  UpdatePreferenceResponse,
 } from 'picsur-shared/dist/dto/api/pref.dto';
-import { ThrowIfFailed } from 'picsur-shared/dist/types/failable';
+import { ThrowIfFailed } from 'picsur-shared/dist/types';
 import { UsrPreferenceDbService } from '../../../collections/preference-db/usr-preference-db.service';
 import { RequiredPermissions } from '../../../decorators/permissions.decorator';
 import { ReqUserID } from '../../../decorators/request-user.decorator';
--- a/backend/src/routes/api/roles/roles.controller.ts
+++ b/backend/src/routes/api/roles/roles.controller.ts
@ -12,7 +12,7 @@ import {
  RoleUpdateResponse,
  SpecialRolesResponse,
 } from 'picsur-shared/dist/dto/api/roles.dto';
-import { Fail, FT, ThrowIfFailed } from 'picsur-shared/dist/types/failable';
+import { Fail, FT, ThrowIfFailed } from 'picsur-shared/dist/types';
 import { RoleDbService } from '../../../collections/role-db/role-db.service';
 import { UserDbService } from '../../../collections/user-db/user-db.service';
 import { RequiredPermissions } from '../../../decorators/permissions.decorator';
--- a/backend/src/routes/api/usage/usage.controller.ts
+++ b/backend/src/routes/api/usage/usage.controller.ts
@ -1,7 +1,7 @@
 import { Controller, Logger, Post, Req, Res } from '@nestjs/common';
 import { Throttle } from '@nestjs/throttler';
 import type { FastifyReply, FastifyRequest } from 'fastify';
-import { Fail, FT, ThrowIfFailed } from 'picsur-shared/dist/types/failable';
+import { Fail, FT, ThrowIfFailed } from 'picsur-shared/dist/types';
 import { UsageConfigService } from '../../../config/late/usage.config.service';
 import { NoPermissions } from '../../../decorators/permissions.decorator';
 import { ReturnsAnything } from '../../../decorators/returns.decorator';
--- a/backend/src/routes/api/user/user-manage.controller.ts
+++ b/backend/src/routes/api/user/user-manage.controller.ts
@ -13,7 +13,7 @@ import {
  UserUpdateRequest,
  UserUpdateResponse,
 } from 'picsur-shared/dist/dto/api/user-manage.dto';
-import { ThrowIfFailed } from 'picsur-shared/dist/types/failable';
+import { ThrowIfFailed } from 'picsur-shared/dist/types';
 import { UserDbService } from '../../../collections/user-db/user-db.service';
 import { RequiredPermissions } from '../../../decorators/permissions.decorator';
 import { Returns } from '../../../decorators/returns.decorator';
--- a/backend/src/routes/api/user/user.controller.ts
+++ b/backend/src/routes/api/user/user.controller.ts
@ -10,7 +10,7 @@ import {
  UserRegisterResponse,
 } from 'picsur-shared/dist/dto/api/user.dto';
 import type { EUser } from 'picsur-shared/dist/entities/user.entity';
-import { ThrowIfFailed } from 'picsur-shared/dist/types/failable';
+import { ThrowIfFailed } from 'picsur-shared/dist/types';
 import { UserDbService } from '../../../collections/user-db/user-db.service';
 import {
  NoPermissions,
--- a/backend/src/routes/image/image-manage.controller.ts
+++ b/backend/src/routes/image/image-manage.controller.ts
@ -16,17 +16,17 @@ import {
  ImageDeleteWithKeyResponse,
  ImageListRequest,
  ImageListResponse,
+  ImagesProgressRequest,
+  ImagesProgressResponse,
+  ImagesUploadResponse,
  ImageUpdateRequest,
  ImageUpdateResponse,
  ImageUploadResponse,
 } from 'picsur-shared/dist/dto/api/image-manage.dto';
 import { Permission } from 'picsur-shared/dist/dto/permissions.enum';
-import {
-  Fail,
-  FT,
-  HasFailed,
-  ThrowIfFailed,
-} from 'picsur-shared/dist/types/failable';
+import { EImage } from 'picsur-shared/dist/entities/image.entity';
+import { Fail, FT, HasFailed, ThrowIfFailed } from 'picsur-shared/dist/types';
+import { EImageBackend } from '../../database/entities/images/image.entity';
 import { PostFiles } from '../../decorators/multipart/multipart.decorator';
 import type { FileIterator } from '../../decorators/multipart/postfiles.pipe';
 import {
@ -34,15 +34,19 @@ import {
  RequiredPermissions,
 } from '../../decorators/permissions.decorator';
 import { ReqUserID } from '../../decorators/request-user.decorator';
-import { Returns } from '../../decorators/returns.decorator';
-import { ImageManagerService } from '../../managers/image/image.service';
+import { Returns, ReturnsAnything } from '../../decorators/returns.decorator';
+import { ImageManagerService } from '../../managers/image/image-manager.service';
+import { IngestService } from '../../managers/image/ingest.service';
 import { GetNextAsync } from '../../util/iterator';
@Controller('api/image')
@RequiredPermissions(Permission.ImageUpload)
 export class ImageManageController {
  private readonly logger = new Logger(ImageManageController.name);

-  constructor(private readonly imagesService: ImageManagerService) {}
+  constructor(
+    private readonly imagesService: ImageManagerService,
+    private readonly ingestService: IngestService,
+  ) {}

  @Post('upload')
  @Returns(ImageUploadResponse)
@ -62,7 +66,7 @@ export class ImageManageController {
    }

    const image = ThrowIfFailed(
-      await this.imagesService.upload(
+      await this.ingestService.uploadPromise(
        userid,
        file.filename,
        buffer,
@ -73,6 +77,54 @@ export class ImageManageController {
    return image;
  }

+  @Post('upload/bulk')
+  @Returns(ImagesUploadResponse)
+  @Throttle(20)
+  async uploadImages(
+    @PostFiles() multipart: FileIterator,
+    @ReqUserID() userid: string,
+    @HasPermission(Permission.ImageDeleteKey) withDeleteKey: boolean,
+  ): Promise<ImagesUploadResponse> {
+    let jobs: {
+      job_id: string;
+      image: EImage;
+    }[] = [];
+    for await (const file of multipart) {
+      const buffer = await file.toBuffer();
+      const filename = file.filename;
+
+      const [job, image] = ThrowIfFailed(
+        await this.ingestService.uploadJob(
+          userid,
+          filename,
+          buffer,
+          withDeleteKey,
+        ),
+      );
+
+      jobs.push({
+        job_id: job.id.toString(),
+        image: image,
+      });
+    }
+    if (jobs.length === 0) {
+      throw Fail(FT.BadRequest, 'No files uploaded');
+    }
+
+    return {
+      count: jobs.length,
+      results: jobs,
+    };
+  }
+
+  @Post('upload/status')
+  @Returns(ImagesProgressResponse)
+  async getImagesProgress(
+    @Body() body: ImagesProgressRequest,
+  ): Promise<ImagesProgressResponse> {
+    return ThrowIfFailed(await this.ingestService.getProgress(body.job_ids));
+  }
+
  @Post('list')
  @RequiredPermissions(Permission.ImageManage)
  @Returns(ImageListResponse)
--- a/backend/src/routes/image/image.controller.ts
+++ b/backend/src/routes/image/image.controller.ts
@ -7,17 +7,14 @@ import {
 } from 'picsur-shared/dist/dto/api/image.dto';
 import { ImageEntryVariant } from 'picsur-shared/dist/dto/image-entry-variant.enum';
 import { FileType2Mime } from 'picsur-shared/dist/dto/mimes.dto';
-import {
-  FT,
-  IsFailure,
-  ThrowIfFailed,
-} from 'picsur-shared/dist/types/failable';
+import { FT, IsFailure, ThrowIfFailed } from 'picsur-shared/dist/types';
 import { UserDbService } from '../../collections/user-db/user-db.service';
 import { ImageFullIdParam } from '../../decorators/image-id/image-full-id.decorator';
 import { ImageIdParam } from '../../decorators/image-id/image-id.decorator';
 import { RequiredPermissions } from '../../decorators/permissions.decorator';
 import { Returns } from '../../decorators/returns.decorator';
-import { ImageManagerService } from '../../managers/image/image.service';
+import { ConvertService } from '../../managers/image/convert.service';
+import { ImageManagerService } from '../../managers/image/image-manager.service';
 import type { ImageFullId } from '../../models/constants/image-full-id.const';
 import { Permission } from '../../models/constants/permissions.const';
 import { EUserBackend2EUser } from '../../models/transformers/user.transformer';
@ -33,6 +30,7 @@ export class ImageController {
  constructor(
    private readonly imagesService: ImageManagerService,
    private readonly userService: UserDbService,
+    private readonly convertService: ConvertService,
  ) {}

  @Head(':id')
@ -65,21 +63,23 @@ export class ImageController {
        const image = ThrowIfFailed(
          await this.imagesService.getOriginal(fullid.id),
        );
+        const data = ThrowIfFailed(await this.imagesService.getData(image));

        res.type(ThrowIfFailed(FileType2Mime(image.filetype)));
-        return image.data;
+        return data;
      }

      const image = ThrowIfFailed(
-        await this.imagesService.getConverted(
+        await this.convertService.convertPromise(
          fullid.id,
          fullid.filetype,
          params,
        ),
      );
+      const data = ThrowIfFailed(await this.imagesService.getData(image));

      res.type(ThrowIfFailed(FileType2Mime(image.filetype)));
-      return image.data;
+      return data;
    } catch (e) {
      if (!IsFailure(e) || e.getType() !== FT.NotFound) throw e;

--- a/backend/src/util/branding.ts
+++ b/backend/src/util/branding.ts
@ -1,6 +1,6 @@
 import { readFile } from 'fs/promises';
 import { resolve } from 'path';
-import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types';
 import { PackageRoot } from '../config/config.static';

 export const BrandingPath = resolve(PackageRoot, '../branding');
--- a/backend/src/util/iterator.ts
+++ b/backend/src/util/iterator.ts
@ -1,4 +1,4 @@
-import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types/failable';
+import { AsyncFailable, Fail, FT } from 'picsur-shared/dist/types';

 export async function GetNextAsync<T>(
  iterator: AsyncIterableIterator<T>,
--- a/backend/src/workers/sharp.wrapper.ts
+++ b/backend/src/workers/sharp.wrapper.ts
@ -1,7 +1,7 @@
 import { Logger } from '@nestjs/common';
 import { ChildProcess, fork } from 'child_process';
 import pTimeout from 'p-timeout';
-import { dirname, join as pathJoin } from 'path';
+import path from 'path';
 import { FileType } from 'picsur-shared/dist/dto/mimes.dto';
 import {
  AsyncFailable,
@ -9,7 +9,7 @@ import {
  Failable,
  FT,
  HasFailed,
-} from 'picsur-shared/dist/types/failable';
+} from 'picsur-shared/dist/types';
 import { Sharp, SharpOptions } from 'sharp';
 import {
  SharpWorkerFinishOptions,
@ -22,13 +22,13 @@ import {
 import { SharpResult } from './sharp/universal-sharp';

 const moduleURL = new URL(import.meta.url);
-const __dirname = dirname(moduleURL.pathname);
+const __dirname = path.dirname(moduleURL.pathname);

 export class SharpWrapper {
  private readonly workerID: number = Math.floor(Math.random() * 100000);
  private readonly logger: Logger = new Logger('SharpWrapper' + this.workerID);

-  private static readonly WORKER_PATH = pathJoin(
+  private static readonly WORKER_PATH = path.join(
    __dirname,
    './sharp',
    'sharp.worker.js',
--- a/backend/src/workers/sharp/sharp.worker.ts
+++ b/backend/src/workers/sharp/sharp.worker.ts
@ -1,5 +1,5 @@
 import { FileType } from 'picsur-shared/dist/dto/mimes.dto';
-import { setrlimit } from 'posix.js';
+import posix from 'posix.js';
 import { Sharp } from 'sharp';
 import {
  SharpWorkerFinishOptions,
@ -11,7 +11,7 @@ import {
 import { UniversalSharpIn, UniversalSharpOut } from './universal-sharp';

 export class SharpWorker {
-  private startTime = 0;
+  private startTime: number = 0;
  private sharpi: Sharp | null = null;

  constructor() {
@ -29,7 +29,7 @@ export class SharpWorker {
      return this.purge('MEMORY_LIMIT_MB environment variable is not set');
    }

-    setrlimit('data', {
+    posix.setrlimit('data', {
      soft: 1000 * 1000 * memoryLimit,
      hard: 1000 * 1000 * memoryLimit,
    });
--- a/backend/tsconfig.json
+++ b/backend/tsconfig.json
@ -3,13 +3,15 @@
  "include": ["src/**/*.ts", "src/**/*.d.ts"],
  "exclude": ["node_modules", "dist", "test", "**/*spec.ts"],
  "compilerOptions": {
-    "lib": ["es2022"],
-    "module": "ES2022",
-    "target": "es2022",
+    "target": "es2020",
+    "module": "es2020",

    "outDir": "./dist",
    "declaration": true,
    "sourceMap": true,
    "emitDecoratorMetadata": true
+  },
+  "ts-node": {
+    "experimentalSpecifierResolution": "node"
  }
 }
--- a/frontend/.browserslistrc
+++ b/frontend/.browserslistrc
@ -8,8 +8,8 @@
 # You can see what browsers were selected by your queries by running:
 #   npx browserslist

-last 2 Chrome version
-last 2 Firefox version
+last 1 Chrome version
+last 1 Firefox version
 last 2 Edge major versions
 last 2 Safari major versions
 last 2 iOS major versions
--- a/frontend/.eslintrc.cjs
+++ b/frontend/.eslintrc.cjs
@ -1,9 +0,0 @@
-module.exports = {
-  parserOptions: {
-    project: './tsconfig.base.json',
-    tsconfigRootDir: __dirname,
-    sourceType: 'module',
-  },
-  ignorePatterns: ['src/environments', 'custom-webpack.config.js'],
-  root: false,
-};
--- a/frontend/angular.json
+++ b/frontend/angular.json
@ -42,8 +42,7 @@
            "allowedCommonJsDependencies": [
              "ngx-auto-unsubscribe-decorator",
              "moment",
-              "platform",
-              "form-data"
+              "platform"
            ],
            "optimization": true,
            "webWorkerTsConfig": "tsconfig.worker.json",
@ -56,8 +55,8 @@
              "budgets": [
                {
                  "type": "initial",
-                  "maximumWarning": "1mb",
-                  "maximumError": "2mb"
+                  "maximumWarning": "500kb",
+                  "maximumError": "1mb"
                },
                {
                  "type": "anyComponentStyle",
@ -104,5 +103,6 @@
        }
      }
    }
-  }
+  },
+  "defaultProject": "picsur-frontend"
 }
--- a/frontend/custom-webpack.config.js
+++ b/frontend/custom-webpack.config.js
@ -1,23 +1,5 @@
 import webpack from 'webpack';
-// import { BundleAnalyzerPlugin } from 'webpack-bundle-analyzer';

 export default {
-  plugins: [
-    new webpack.ContextReplacementPlugin(/moment[\/\\]locale$/, /en/),
-    // new BundleAnalyzerPlugin(),
-  ],
-  module: {
-    rules: [
-      {
-        test: /\.m?js$/,
-        exclude: /node_modules/,
-        use: {
-          loader: 'babel-loader',
-          options: {
-            presets: [['@babel/preset-env']],
-          },
-        },
-      },
-    ],
-  },
+  plugins: [new webpack.ContextReplacementPlugin(/moment[\/\\]locale$/, /en/)],
 };
--- a/frontend/package.json
+++ b/frontend/package.json
@ -1,10 +1,10 @@
 {
  "name": "picsur-frontend",
-  "version": "0.5.2",
+  "version": "0.4.0",
  "description": "Frontend for Picsur",
  "license": "GPL-3.0",
-  "repository": "https://github.com/caramelfur/Picsur",
-  "author": "Caramel <picsur@caramelfur.dev>",
+  "repository": "https://github.com/rubikscraft/Picsur",
+  "author": "Rubikscraft <contact@rubikscraft.nl>",
  "type": "module",
  "scripts": {
    "ng": "ng",
@ -14,40 +14,34 @@
    "purge": "rm -rf dist && rm -rf node_modules && rm -rf .angular"
  },
  "devDependencies": {
-    "@angular-builders/custom-webpack": "^16.0.0",
-    "@angular-devkit/build-angular": "^16.1.0",
-    "@angular/animations": "^16.1.1",
-    "@angular/cdk": "^16.1.1",
-    "@angular/cli": "^16.1.0",
-    "@angular/common": "^16.1.1",
-    "@angular/compiler": "^16.1.1",
-    "@angular/compiler-cli": "^16.1.1",
-    "@angular/core": "^16.1.1",
-    "@angular/forms": "^16.1.1",
-    "@angular/material": "^16.1.1",
-    "@angular/platform-browser": "^16.1.1",
-    "@angular/platform-browser-dynamic": "^16.1.1",
-    "@angular/router": "^16.1.1",
-    "@babel/cli": "^7.22.5",
-    "@babel/core": "^7.22.5",
-    "@babel/preset-env": "^7.22.5",
-    "@fontsource/roboto": "^5.0.3",
-    "@ng-web-apis/common": "^2.1.0",
+    "@angular-builders/custom-webpack": "^14.0.1",
+    "@angular-devkit/build-angular": "14.2.4",
+    "@angular/animations": "^14.2.4",
+    "@angular/cdk": "^14.2.3",
+    "@angular/cli": "^14.2.4",
+    "@angular/common": "^14.2.4",
+    "@angular/compiler": "^14.2.4",
+    "@angular/compiler-cli": "^14.2.4",
+    "@angular/core": "^14.2.4",
+    "@angular/forms": "^14.2.4",
+    "@angular/material": "^14.2.3",
+    "@angular/platform-browser": "^14.2.4",
+    "@angular/platform-browser-dynamic": "^14.2.4",
+    "@angular/router": "^14.2.4",
+    "@fontsource/material-icons": "^4.5.4",
+    "@fontsource/material-icons-outlined": "^4.5.4",
+    "@fontsource/roboto": "^4.5.8",
+    "@ng-web-apis/common": "^2.0.1",
    "@ng-web-apis/resize-observer": "^2.0.0",
    "@ngui/common": "^1.0.0",
-    "@popperjs/core": "^2.11.8",
    "@types/ackee-tracker": "^5.0.2",
-    "@types/node": "^20.3.1",
+    "@types/node": "^18.7.23",
    "@types/resize-observer-browser": "^0.1.7",
-    "@types/validator": "^13.7.17",
+    "@types/validator": "^13.7.7",
    "ackee-tracker": "^5.1.0",
-    "axios": "^1.4.0",
-    "babel-loader": "^9.1.2",
-    "bootstrap": "^5.3.0",
-    "browserslist": "^4.21.8",
-    "caniuse-lite": "^1.0.30001503",
+    "bootstrap": "^5.2.1",
    "fuse.js": "^6.6.2",
-    "material-icons": "^1.13.8",
+    "jwt-decode": "^3.1.2",
    "moment": "^2.29.4",
    "ng-mat-select-infinite-scroll": "^4.0.0",
    "ngx-auto-unsubscribe-decorator": "^1.1.0",
@ -55,14 +49,13 @@
    "ngx-moment": "^6.0.2",
    "picsur-shared": "*",
    "reflect-metadata": "^0.1.13",
-    "rxjs": "^7.8.1",
-    "tslib": "^2.5.3",
-    "typescript": "^5.1.3",
-    "webpack-bundle-analyzer": "^4.9.0",
-    "zod": "^3.21.4",
-    "zone.js": "^0.13.1"
+    "rxjs": "~7.5.7",
+    "tslib": "^2.4.0",
+    "typescript": "4.8.4",
+    "zod": "^3.19.1",
+    "zone.js": "~0.11.8"
  },
  "dependencies": {
-    "@leteu/jwt-decoder": "^1.0.4"
+    "axios": "^0.27.2"
  }
 }
--- a/frontend/src/app/app.component.ts
+++ b/frontend/src/app/app.component.ts
@ -25,14 +25,14 @@ export class AppComponent implements OnInit {

  @ViewChild(MatSidenav) sidebar: MatSidenav;

-  loading = false;
+  loading: boolean = false;
  private loadingTimeout: number | null = null;

-  wrapContentWithContainer = true;
+  wrapContentWithContainer: boolean = true;
  sidebarPortal: Portal<any> | undefined = undefined;

-  isDesktop = false;
-  hasSidebar = false;
+  isDesktop: boolean = false;
+  hasSidebar: boolean = false;

  public constructor(
    private readonly router: Router,
@ -64,7 +64,7 @@ export class AppComponent implements OnInit {
      if (event instanceof NavigationEnd) {
        this.loadingEnd();
      }
-      if (event instanceof NavigationEnd) this.onNavigationEnd();
+      if (event instanceof NavigationEnd) this.onNavigationEnd(event);
      if (event instanceof NavigationError) this.onNavigationError(event);
    });
  }
@ -84,7 +84,7 @@ export class AppComponent implements OnInit {
      this.router.navigate(['/error/404'], { replaceUrl: true });
  }

-  private async onNavigationEnd() {
+  private async onNavigationEnd(event: NavigationEnd) {
    const data = this.routeData;
    this.wrapContentWithContainer = !data.noContainer;

--- a/frontend/src/app/components/copy-field/copy-field.component.html
+++ b/frontend/src/app/components/copy-field/copy-field.component.html
@ -1,8 +1,4 @@
-<mat-form-field
-  [appearance]="appearance"
-  [color]="color"
-  [subscriptSizing]="subscriptSizing"
->
+<mat-form-field appearance="outline" color="accent">
  <mat-label>{{ label }}</mat-label>
  <input
    matInput
--- a/frontend/src/app/components/copy-field/copy-field.component.ts
+++ b/frontend/src/app/components/copy-field/copy-field.component.ts
@ -1,12 +1,8 @@
+import { Clipboard } from '@angular/cdk/clipboard';
 import { Component, EventEmitter, Input, Output } from '@angular/core';
-import {
-  MatFormFieldAppearance,
-  SubscriptSizing,
-} from '@angular/material/form-field';
-import { FT, Fail } from 'picsur-shared/dist/types/failable';
-import { Logger } from '../../services/logger/logger.service';
-import { ClipboardService } from '../../util/clipboard.service';
-import { ErrorService } from '../../util/error-manager/error.service';
+import { Fail, FT } from 'picsur-shared/dist/types';
+import { Logger } from 'src/app/services/logger/logger.service';
+import { ErrorService } from 'src/app/util/error-manager/error.service';

@Component({
  selector: 'copy-field',
@ -17,26 +13,22 @@ export class CopyFieldComponent {
  private readonly logger = new Logger(CopyFieldComponent.name);

  // Two parameters: name, value
-  @Input() label = 'Loading...';
-  @Input() value = 'Loading...';
+  @Input() label: string = 'Loading...';
+  @Input() value: string = 'Loading...';

-  @Input() showHideButton = false;
-  @Input() hidden = false;
-
-  @Input() color: 'primary' | 'accent' | 'warn' = 'primary';
-  @Input() appearance: MatFormFieldAppearance = 'outline';
-  @Input() subscriptSizing: SubscriptSizing = 'fixed';
+  @Input() showHideButton: boolean = false;
+  @Input() hidden: boolean = false;

  @Output('copy') onCopy = new EventEmitter<string>();
  @Output('hide') onHide = new EventEmitter<boolean>();

  constructor(
-    private readonly clipboard: ClipboardService,
+    private readonly clipboard: Clipboard,
    private readonly errorService: ErrorService,
  ) {}

-  public async copy() {
-    if (await this.clipboard.copy(this.value)) {
+  public copy() {
+    if (this.clipboard.copy(this.value)) {
      this.errorService.info(`Copied ${this.label}!`);
      this.onCopy.emit(this.value);
      return;
--- a/frontend/src/app/components/copy-field/copy-field.module.ts
+++ b/frontend/src/app/components/copy-field/copy-field.module.ts
@ -1,11 +1,11 @@
+import { ClipboardModule } from '@angular/cdk/clipboard';
 import { CommonModule } from '@angular/common';
 import { NgModule } from '@angular/core';
 import { MatButtonModule } from '@angular/material/button';
 import { MatIconModule } from '@angular/material/icon';
 import { MatInputModule } from '@angular/material/input';
+import { ErrorManagerModule } from 'src/app/util/error-manager/error-manager.module';
 import { CopyFieldComponent } from './copy-field.component';
-import { ErrorManagerModule } from '../../util/error-manager/error-manager.module';
-
@NgModule({
  declarations: [CopyFieldComponent],
  imports: [
@ -15,6 +15,7 @@ import { ErrorManagerModule } from '../../util/error-manager/error-manager.modul
    MatInputModule,
    MatIconModule,
    MatButtonModule,
+    ClipboardModule,
  ],
  exports: [CopyFieldComponent],
 })
--- a/frontend/src/app/components/fab/normal/fab.component.ts
+++ b/frontend/src/app/components/fab/normal/fab.component.ts
@ -5,10 +5,11 @@ import { Component, Input } from '@angular/core';
  templateUrl: './fab.component.html',
 })
 export class FabComponent {
-  @Input('aria-label') ariaLabel = 'Floating Action Button';
-  @Input() icon = 'add';
-  @Input() color = 'primary';
+  @Input('aria-label') ariaLabel: string = 'Floating Action Button';
+  @Input() icon: string = 'add';
+  @Input() color: string = 'accent';
  @Input('tooltip') tooltip: string;
-  // eslint-disable-next-line @typescript-eslint/no-empty-function
  @Input() onClick: () => void = () => {};
+
+  constructor() {}
 }
--- a/frontend/src/app/components/fab/speed-dial/speed-dial-option.directive.ts
+++ b/frontend/src/app/components/fab/speed-dial/speed-dial-option.directive.ts
@ -1,5 +1,5 @@
 import { Directive, Host, Optional } from '@angular/core';
-import { MatMiniFabButton } from '@angular/material/button';
+import { MatButton } from '@angular/material/button';
 import { MatTooltip } from '@angular/material/tooltip';

@Directive({
@ -8,9 +8,9 @@ import { MatTooltip } from '@angular/material/tooltip';
 export class SpeedDialOptionDirective {
  constructor(
    @Host() @Optional() tooltip?: MatTooltip,
-    @Host() @Optional() button?: MatMiniFabButton,
+    @Host() @Optional() button?: MatButton,
  ) {
    if (tooltip) tooltip.position = 'left';
-    if (button) button.color = 'accent';
+    if (button) button.color = 'primary';
  }
 }
--- a/frontend/src/app/components/fab/speed-dial/speed-dial.component.html
+++ b/frontend/src/app/components/fab/speed-dial/speed-dial.component.html
@ -22,7 +22,7 @@
      [matTooltip]="tooltip"
      matTooltipPosition="left"
      [matTooltipDisabled]="!openManager.isOpen"
-      (click)="click()"
+      (click)="click($event)"
      aria-label=""
    >
      <mat-icon
--- a/Show more
+++ b/Show more
Author	SHA1	Message	Date
rubikscraft	4d3ca30efa	move image storage to s3/local s3	2022-10-02 17:29:43 +02:00
rubikscraft	d10ba06947	add nodejs version check	2022-10-02 14:08:20 +02:00
rubikscraft	b3a80f845a	update packages	2022-09-26 17:10:02 +02:00
rubikscraft	9c17b3ed35	Merge branch 'master' of github.com:rubikscraft/Picsur	2022-09-24 12:23:59 +02:00
rubikscraft	e45510f35a	add some memory logging	2022-09-24 12:23:56 +02:00
Rubikscraft	a7ce0d9b0c	Merge pull request #17 Fix minor grammatical errors in the README	2022-09-23 22:23:07 +02:00
Dyras	cb71f6dfc9	Fix minor grammatical errors in the README	2022-09-23 21:17:28 +02:00
rubikscraft	3e62412ef8	fix some bugs and try things	2022-09-20 21:01:45 +02:00
rubikscraft	d03d3f6ed4	add support for upload status report	2022-09-19 20:45:35 +02:00
rubikscraft	6e4ac465d4	switch to axios for frontend, progress reporting now works	2022-09-19 17:55:45 +02:00
rubikscraft	38c2b9d42e	make image converting a job too	2022-09-19 16:53:58 +02:00
rubikscraft	e622972e08	cmon, why the logger gotta be like this	2022-09-18 20:57:05 +02:00
rubikscraft	acc64711fc	add support for bull with redis	2022-09-18 17:32:18 +02:00
rubikscraft	20b4faa2ad	move entities	2022-09-18 16:05:37 +02:00
rubikscraft	6d5039d15c	use nestjs scheduler	2022-09-18 15:40:44 +02:00
rubikscraft	9b0ab6adef	change upload mechanisms	2022-09-18 15:16:34 +02:00
rubikscraft	471a66aa81	add ratelimits	2022-09-17 19:46:53 +02:00
rubikscraft	bc1d012322	run formatter	2022-09-17 18:02:07 +02:00
rubikscraft	b64f471d81	add usage reporting	2022-09-17 18:00:17 +02:00
rubikscraft	5908b52037	add loading bars for slow internet	2022-09-17 16:19:27 +02:00
rubikscraft	426670b5bb	fix some bugs	2022-09-17 14:43:37 +02:00
rubikscraft	c1a3e43615	Allow overriding of hostname	2022-09-17 14:31:17 +02:00
rubikscraft	32eda60bb4	improve preference behaviour	2022-09-17 14:09:37 +02:00
rubikscraft	5166e799d2	change settings layout	2022-09-16 22:09:24 +02:00
rubikscraft	f8ff054ce6	migrate to newer ms library	2022-09-16 19:14:25 +02:00
rubikscraft	e5eecebd51	improve preference management	2022-09-16 17:20:25 +02:00
rubikscraft	c8e1ff7fb7	Merge branch 'master' of github.com:rubikscraft/Picsur	2022-09-16 15:10:31 +02:00
rubikscraft	9536441caf	update deps	2022-09-16 15:10:29 +02:00
rubikscraft	bdbcf70082	update deps	2022-09-16 15:06:42 +02:00
rubikscraft	4709961f02	add proxy ip resolving	2022-09-16 15:05:13 +02:00
rubikscraft	e0887fa5b8	fix bug in caching time being ignored	2022-09-16 13:16:27 +02:00
rubikscraft	4e2f00545e	fix bug accepting qoi files	2022-09-16 10:41:11 +02:00
Rubikscraft	b28df28a4e	Update README.md	2022-09-15 10:27:21 +02:00
Rubikscraft	fa75b089c6	Update README.md	2022-09-14 11:50:39 +02:00
rubikscraft	32f3c409a9	dynamic usage	2022-09-14 10:39:57 +02:00
rubikscraft	3aa6208e40	add some route splitting	2022-09-14 10:17:08 +02:00
rubikscraft	9c68e9b5c0	play around with usage reporting	2022-09-13 20:34:35 +02:00
rubikscraft	9e9753a530	test some stuff	2022-09-13 00:13:49 +02:00
rubikscraft	35c8000589	update readme	2022-09-11 16:47:12 +02:00