Switch task queue to procrastinate based on psql (#11)

* Add openaleph-procrastinate. Bump versions to satisfy dependencies (poetry lock).

* 🧑‍💻 Add pre-commit, use requirements.txt, upgrade to python3.13

* 🧑‍💻 Add dev requirements only for test build

* 🔥 (github) Drop daily cache job

* ✅ (tests/test_pdf) Fix whitespace errors from test results

* 🔨 (make) Build before test

* 👷 Inline base build

* 🚧 Tweak builds and tags

* 👷 (github) Skip intermediate arm46 build for tests

* 👷 (github) Skip cache-from [tmp]

* Revert "👷 (github) Skip cache-from [tmp]"

This reverts commit 03f86fd.

* 👷 (github/docker) Try this

* 🚨 Apply black

* 👷 (github/docker) Don't use registry cache

* 🧪 (test_image) Skip gif test

* 👷 (github/docker) maybe this

* ✨ Boilerplate ingest task for procrastinate

* 🔧  Use pydantic_settings

* 📌 Use openaleph-procrastinate from git

* ⚰️ Drop TranscriptionSupport

* 🔥 Remove analysis part

* 🔥 Remove servicelayer worker

* ♻️ Refactor manager and supports to work with procrastinate

* 🚧 Make procrastinate task to work with manager

* ➖ languagecodes, pantomime -> rigour

* 🔊 Tweak global logging

* ♻️ Refactor cli with typer

* 🧪 Make tests work with procrastinate refactor

* 🩹 (ingestors/email) Use relative path

* ✨ (support/timestamp) Fall back to dateparser for unknown formats

* 🙈 Ignore more

* 🔥 Remove unused lid model

* 👷 (github) Tag base image properly

* 📦 (docker) Use entrypoint and run procrastinate worker

* 🧑‍💻 (contrib) Add non-docker debian install dependencies

* 🧪 Add end-to-end testing setup

* 🧪 (e2e) Working example

* 🔧 (settings) Move deferring settings up to openaleph-procrastinate

* 📌 requirements

* Pin Tesserocr to 2.6.2

* Add ENV LD_PRELOAD for Apple Silicone as comment

* Solve minor errors

* Bump openaleph_procrastinate version

* 🐛 (cli) Use defer settings correctly in debug mode

* ⬆️ openaleph-procrastinate v0.0.7

* 👽️ Adapt explicit defers from openaleph-procrastinate v0.0.7

* 👷 Tweak compose settings

* Bump openaleph-procrastinate version

* Add namespace to entities. Remove app user

* Add namespace info to test setup

* Explicitly set the testing DB to sqlite

* Pin procrastinate to 3.2.2 for tests

* Add transcription procrastinate task

* 📌 Pin procrastinate==3.2.2 for test docker build

* 🚧 (docker) Cleanup duplicated RUN

* 🚧 (cli) Adjust settings display

* 🔧 (tests) Properly set FTM_STORE_URI

* ⬆️ Dependencies

* ✨ Documentation

* 🔥 Drop google cloud vision support

* Always index entities after ingesting

* Replace get_dataset with get_fragments (ftmq.store)

* ⬆️ ftm(q) 4.1.x, openaleph-procrastinate 0.0.13

* 🐛 (support/email) Catch empty name

* 🎨 (support/transcription) Cleanup

* 🔥 Drop unused settings

* ⬆️ openaleph-procrastinate 0.0.14

* 🚧 (cli) Make foreign_id optional

* ✅ Add e2e testing with minio

* ⬆️ openaleph-procrastinate 0.0.16

* ⬆️ openaleph-procrastinate 0.0.16

* 💚 (github) Skip e2e

* ⬆️ ftmq 4.1.1, openaleph-procrastinate 0.0.18

* 🚧 (tests/e2e) Adjustments

* 🔖 Bump version: 3.24.0 → 5.0.0rc1

* 💚 (github) Enable e2e again

* ⬆️ openaleph-procrastinate 0.0.20

* 🚧 (ingestors/image) Explicitly close PIL obj after processing

* 🚧 (support/shell) Write to subprocess special DEVNULL

* 🚧 (ingestors/access) Wrap subprocess call in context manager

* 🚧 (ingestors/csv) Properly use context manager for file open

* 🚧 (support/ocr) Clean up OCR engine after use

* ⚗️ memray

* 🔧 (settings) Properly configure servicelayer tags

* 🚧 (tasks) Collect garbage, just in case

* 📌 Pin olefile<0.47 as this leaks crazy memory

* 📌 Fix RC version string

* ⬆️ All the things

* 🔖 Bump version: 5.0.0-rc1 → 5.0.0-rc2

* ⬆️ openaleph-procrastinate 0.0.25 and others

* 🔖 Bump version: 5.0.0-rc2 → 5.0.0-rc3

* 🩹 (tasks) Pass through batch (formerly job_id)

* 📌 Pin back tesserocr=2.6.2

* Dockerfile refactorings (tesserocr 2.6.2, openaleph-servicelayer etc.) (#16)

* Compile tesserocr with c++ 14; use openaleph-servicelayer

* Build tesserocr in Dockerfile.base; don't build Apple base docker image

* Separate test docker image

* Move tesserocr to ocr dependencies

* Only generate main requirements from pre-commit hook

* Move tesserocr to optional dependencies

* Add build-test to Makefile test, before running tests

* 🔖 Bump version: 5.0.0-rc3 → 5.0.0-rc4

* ⬆️ followthemoney 4.2.0

* ⬆️ ftmq 4.2.2 (psycopg3)

* ⬆️ openaleph-procrastinate 0.0.29

* 🔧 Ensure psycopg3 for sl tags db

* Temporarily disable daily ingest-file-base build

* Update poetry.lock

* 🔖 Bump version: 5.0.0-rc4 → 5.0.0-rc5

---------

Co-authored-by: Alex Ștefănescu <alex.stefanescu@pm.me>
Co-authored-by: Alex Ștefănescu <catileptic@users.noreply.github.com>

Author: simonwoerpel

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 3.24.0
+current_version = 5.0.0-rc5
 tag_name = {new_version}
 commit = True
 tag = True

.dockerignore

@@ -0,0 +1 @@
+.gitignore

.dockerignore

@@ -82,3 +82,7 @@ jobs:
           labels: ${{ steps.meta.outputs.labels }}
           cache-from: type=gha
           cache-to: type=gha,mode=max
+
+      - name: Run e2e test
+        working-directory: ./e2e
+        run: bash -c ./test_e2e.sh

.github/workflows/build.yml

@@ -2,8 +2,8 @@ name: Build ingest-file-base
 
 on:
   workflow_dispatch: {}
-  schedule:
-    - cron: "0 0 * * *"
+  # schedule:
+  #   - cron: "0 0 * * *"
   push:
     paths:
       - Dockerfile.base
@@ -28,7 +28,8 @@ jobs:
             type=ref,event=branch
             type=semver,pattern={{version}}
             type=sha
-            type=raw,value=latest
+            type=raw,value=cache
+            type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags') }}
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v2
         with:
@@ -44,7 +45,7 @@ jobs:
         with:
           context: .
           file: ./Dockerfile.base
-          platforms: linux/amd64,linux/arm64
+          platforms: linux/amd64 #,linux/arm64
           push: true
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}

.github/workflows/docker-base.yml

@@ -1,5 +1,10 @@
+contrib/*.bin
+data/archive
 data/model_type_prediction.ftz
+debug.sqlite3
 data/servicelayer-archive
+# documentation
+site
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

.gitignore

@@ -5,20 +5,21 @@
 # * Run "pre-commit install".
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v5.0.0
+    rev: v6.0.0
     hooks:
       - id: check-added-large-files
       - id: check-case-conflict
       - id: check-merge-conflict
       - id: check-symlinks
       - id: check-toml
       - id: check-yaml
+        exclude: "mkdocs.yml"
       - id: debug-statements
       - id: end-of-file-fixer
       - id: mixed-line-ending
         args: ["--fix=lf"]
       - id: trailing-whitespace
-        exclude: ".bumpversion.cfg"  # wtf
+        exclude: ".bumpversion.cfg" # wtf
 
   # - repo: https://github.com/asottile/pyupgrade
   #   rev: v3.10.1
@@ -78,7 +79,7 @@ repos:
     rev: 1.9.0
     hooks:
       - id: poetry-export
-        args: ["--without-hashes", "-o", "requirements.txt"]
+        args: ["--without-hashes", "--with", "main", "-o", "requirements.txt"]
       - id: poetry-export
         args:
           ["--without-hashes", "--only", "dev", "-o", "requirements-dev.txt"]

.pre-commit-config.yaml

@@ -1,15 +1,22 @@
 FROM ghcr.io/openaleph/ingest-file-base:latest
 
+# uncomment when running on Apple Silicon
+# ENV LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1
+ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgomp.so.1
+
 COPY . /ingestors
+RUN rm -rf /ingestors/tests
 WORKDIR /ingestors
-RUN pip3 install --no-cache-dir -r /ingestors/requirements.txt
-RUN pip3 install --no-cache-dir /ingestors
+
+RUN pip3 install --no-cache-dir --no-deps -r /ingestors/requirements.txt
+RUN pip3 install --no-deps --no-cache-dir /ingestors
 
 ENV ARCHIVE_TYPE=file \
     ARCHIVE_PATH=/data \
-    FTM_STORE_URI=postgresql://aleph:aleph@postgres/aleph \
+    OPENALEPH_DB_URI=postgresql://aleph:aleph@postgres/aleph \
     REDIS_URL=redis://redis:6379/0 \
     TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata
 
-USER app
-CMD ingestors process
+ENV PROCRASTINATE_APP="ingestors.tasks.app"
+
+CMD ["procrastinate", "worker", "-q", "ingest"]

Dockerfile

@@ -12,12 +12,12 @@ RUN apt-get -qq -y update \
     # python deps (mostly to install their dependencies)
     python3-pip python3-dev python3-pil \
     # tesseract
-    tesseract-ocr libtesseract-dev libleptonica-dev pkg-config\
+    tesseract-ocr libtesseract-dev libleptonica-dev pkg-config \
     # libraries
     libxslt1-dev libpq-dev libldap2-dev libsasl2-dev \
     zlib1g-dev libicu-dev libxml2-dev \
     # package tools
-    unrar p7zip-full \
+    unrar \
     # audio & video metadata
     libmediainfo-dev \
     # image processing, djvu
@@ -116,41 +116,22 @@ ENV LANG='en_US.UTF-8' \
     OMP_THREAD_LIMIT='1' \
     OPENBLAS_NUM_THREADS='1'
 
-ENV LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libgomp.so.1"
+# force compile tesserocr 2.6.2 with C++ 14
+# to make it compatible with Tesseract 5
+RUN pip download --no-binary=:all: "tesserocr==2.6.2" \
+    && tar -xzf tesserocr-2.6.2.tar.gz \
+    && sed -i "s/-std=c++11/-std=c++14/" tesserocr-2.6.2/setup.py \
+    && cd tesserocr-2.6.2 \
+    && CXXFLAGS="-std=c++14" pip install --no-cache-dir .
 
 # tesseract 5
 ENV TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata
 
 RUN groupadd -g 1000 -r app \
     && useradd -m -u 1000 -s /bin/false -g app app
 
-# Download the ftm-typepredict model
-RUN mkdir /models/ && \
-    curl -o "/models/model_type_prediction.ftz" "https://public.data.occrp.org/develop/models/types/type-08012020-7a69d1b.ftz"
-
 RUN pip3 install --no-cache-dir --prefer-binary --upgrade pip
 RUN pip3 install --no-cache-dir --prefer-binary --upgrade setuptools wheel
 
-# Install spaCy
-RUN pip3 install --no-cache-dir spacy
 # Install PyICU
 RUN pip3 install --no-binary=:pyicu: pyicu
-# Install TesserOCR
-RUN pip3 install --no-binary=:tesserocr: tesserocr
-
-# Install default (small) spaCy models
-RUN python3 -m spacy download en_core_web_sm
-RUN python3 -m spacy download de_core_news_sm
-RUN python3 -m spacy download fr_core_news_sm
-RUN python3 -m spacy download es_core_news_sm
-RUN python3 -m spacy download ru_core_news_sm
-RUN python3 -m spacy download pt_core_news_sm
-RUN python3 -m spacy download ro_core_news_sm
-RUN python3 -m spacy download mk_core_news_sm
-RUN python3 -m spacy download el_core_news_sm
-RUN python3 -m spacy download pl_core_news_sm
-RUN python3 -m spacy download it_core_news_sm
-RUN python3 -m spacy download lt_core_news_sm
-RUN python3 -m spacy download nl_core_news_sm
-RUN python3 -m spacy download nb_core_news_sm
-RUN python3 -m spacy download da_core_news_sm

Dockerfile.base

@@ -1,20 +1,26 @@
 FROM ghcr.io/openaleph/ingest-file-base:latest
 
+# uncomment when running on Apple Silicon
 # ENV LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1
+ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgomp.so.1
 
 COPY . /ingestors
 WORKDIR /ingestors
-RUN pip3 install --no-cache-dir -r /ingestors/requirements.txt
-RUN pip3 install --no-cache-dir /ingestors
 
-RUN pip3 install -r /ingestors/requirements-dev.txt
+RUN pip3 install --no-cache-dir --no-deps -r /ingestors/requirements.txt
+RUN pip3 install --no-deps --no-cache-dir /ingestors
+
+RUN pip3 install --no-deps -r /ingestors/requirements-dev.txt
+RUN pip3 install --no-cache-dir procrastinate==3.2.2
 RUN chown -R app:app /ingestors
 
 ENV ARCHIVE_TYPE=file \
     ARCHIVE_PATH=/data \
     FTM_STORE_URI=postgresql://aleph:aleph@postgres/aleph \
     REDIS_URL=redis://redis:6379/0 \
-    TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata
+    TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata \
+    DEBUG=1
+
+ENV PROCRASTINATE_APP="ingestors.tasks.app"
 
-USER app
-CMD ingestors process
+CMD ["pytest"]