Skip to content

Switch task queue to procrastinate based on psql #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 104 commits into from
Aug 31, 2025
Merged

Switch task queue to procrastinate based on psql #11

merged 104 commits into from
Aug 31, 2025

Conversation

simonwoerpel
Copy link
Member

@simonwoerpel simonwoerpel commented Jun 19, 2025 •
edited
Loading

this is work in progress

This is a refactor to swap out servicelayer worker (and redis queue) with procrastinate.
It as well separates the analysis code into ftm-analyze

TODO:

catileptic and others added 30 commits June 16, 2025 15:53
@catileptic @simonwoerpel
Add openaleph-procrastinate. Bump versions to satisfy dependencies (p…
190c5ff 
…oetry lock).
@simonwoerpel
🧑‍💻 Add pre-commit, use requirements.txt, upgrade to python3.13
a46a028
@simonwoerpel
🧑‍💻 Add dev requirements only for test build
109b53a
@simonwoerpel
🔥 (github) Drop daily cache job
43a02d1
@simonwoerpel
✅ (tests/test_pdf) Fix whitespace errors from test results
f977f90
@simonwoerpel
🔨 (make) Build before test
d6ad514
@simonwoerpel
👷 Inline base build
8ef339f
@simonwoerpel
🚧 Tweak builds and tags
9e376e4
@simonwoerpel
👷 (github) Skip intermediate arm46 build for tests
39a2385
@simonwoerpel
👷 (github) Skip cache-from [tmp]
03f86fd
@simonwoerpel
Revert "👷 (github) Skip cache-from [tmp]"
ff1a8c4 
This reverts commit 03f86fd.
@simonwoerpel
👷 (github/docker) Try this
4adcfb2
@simonwoerpel
🚨 Apply black
099aef5
@simonwoerpel
👷 (github/docker) Don't use registry cache
4071354
@simonwoerpel
🧪 (test_image) Skip gif test
dd8bb0c
@simonwoerpel
👷 (github/docker) maybe this
9adf413
@simonwoerpel
✨ Boilerplate ingest task for procrastinate
53d0a8a
@simonwoerpel
🔧 Use pydantic_settings
6457ebd
@simonwoerpel
📌 Use openaleph-procrastinate from git
90292be
@simonwoerpel
⚰️ Drop TranscriptionSupport
99046a8
@simonwoerpel
🔥 Remove analysis part
33a33f0
@simonwoerpel
🔥 Remove servicelayer worker
ef84cf7
@simonwoerpel
♻️ Refactor manager and supports to work with procrastinate
b1dadd8
@simonwoerpel
🚧 Make procrastinate task to work with manager
e64425b
@simonwoerpel
➖ languagecodes, pantomime -> rigour
13bbf14
@simonwoerpel
🔊 Tweak global logging
0dacf45
@simonwoerpel
♻️ Refactor cli with typer
bf2a15e
@simonwoerpel
🧪 Make tests work with procrastinate refactor
8873af6
@simonwoerpel
🩹 (ingestors/email) Use relative path
bf26349
@simonwoerpel
✨ (support/timestamp) Fall back to dateparser for unknown formats
bddb556
@simonwoerpel simonwoerpel changed the title [WIP] Feat/procrastinate Switch task queue to procrastinate based on psql Aug 11, 2025
simonwoerpel and others added 10 commits August 14, 2025 02:22
@simonwoerpel
⬆️ openaleph-procrastinate 0.0.25 and others
80b34ef
@simonwoerpel
🔖 Bump version: 5.0.0-rc2 → 5.0.0-rc3
538b3e7
@simonwoerpel
🩹 (tasks) Pass through batch (formerly job_id)
f7eb25f
@simonwoerpel
📌 Pin back tesserocr=2.6.2
5743921
@catileptic
Dockerfile refactorings (tesserocr 2.6.2, openaleph-servicelayer etc.) (
b4fa0d5 
#16)

* Compile tesserocr with c++ 14; use openaleph-servicelayer

* Build tesserocr in Dockerfile.base; don't build Apple base docker image

* Separate test docker image

* Move tesserocr to ocr dependencies

* Only generate main requirements from pre-commit hook

* Move tesserocr to optional dependencies

* Add build-test to Makefile test, before running tests

* 🔖 Bump version: 5.0.0-rc3 → 5.0.0-rc4
@simonwoerpel
⬆️ followthemoney 4.2.0
171d063
@simonwoerpel
⬆️ ftmq 4.2.2 (psycopg3)
896685f
@simonwoerpel
⬆️ openaleph-procrastinate 0.0.29
83a8098
@simonwoerpel
🔧 Ensure psycopg3 for sl tags db
d436fb5
@catileptic
Temporarily disable daily ingest-file-base build
a9f6af0
@catileptic catileptic merged commit aafc5bf into main Aug 31, 2025
2 of 4 checks passed
@catileptic catileptic deleted the feat/procrastinate branch August 31, 2025 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@catileptic catileptic catileptic requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@simonwoerpel @catileptic