Skip to content

Commit 4cca7c0

Browse files
Alessio Fabianigiohappyallyoucanmapdependabot[bot]t-book
authored
[Fixes GeoNode#8148] [Harvesting] Adapt / extend remote service model (GeoNode#8149)
* Bump urllib3 from 1.26.2 to 1.26.3 (GeoNode#6908) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.2 to 1.26.3. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/1.26.3/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.2...1.26.3) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni <toni.schoenbuchner@csgis.de> * [Fixes GeoNode#6880] Circle CI upload tests fail irregulary (GeoNode#6881) * [Fixes GeoNode#6880] Circle CI upload tests fail irregulary * CircleCI test fix: sometimes expires due to upload timeout in the test environment * - Avoid infinite loop on upload testing * Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment" This reverts commit 66139fd. Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it> Co-authored-by: afabiani <alessio.fabiani@gmail.com> * [Fixes GeoNode#6914] Remove "add to basket" tool for documents and maps (GeoNode#6915) * Added malnajdi as contributor * [Fixes GeoNode#6910] meaningful filename for document download (GeoNode#6911) * get meaningful document filenames on download * - Strip extension from document title before slugify it (e.g.: image.jpg instead of imagejpg.jpg) Co-authored-by: afabiani <alessio.fabiani@gmail.com> Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it> * - CircleCI Upload Tests: trying to reduce more the risk of infinite loop on "wait_for_progress" * [Fixes GeoNode#6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION (GeoNode#6913) * [Fixes GeoNode#6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION * [Fixes GeoNode#6916] unit test for GEOSERVER_LOCATION * Bump django-cors-headers from 3.6.0 to 3.7.0 (GeoNode#6901) Bumps [django-cors-headers](https://github.com/adamchainz/django-cors-headers) from 3.6.0 to 3.7.0. - [Release notes](https://github.com/adamchainz/django-cors-headers/releases) - [Changelog](https://github.com/adamchainz/django-cors-headers/blob/master/HISTORY.rst) - [Commits](adamchainz/django-cors-headers@3.6.0...3.7.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump amqp from 5.0.3 to 5.0.5 (GeoNode#6905) Bumps [amqp](https://github.com/celery/py-amqp) from 5.0.3 to 5.0.5. - [Release notes](https://github.com/celery/py-amqp/releases) - [Changelog](https://github.com/celery/py-amqp/blob/master/Changelog) - [Commits](celery/py-amqp@v5.0.3...v5.0.5) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pip from 21.0 to 21.0.1 (GeoNode#6900) Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1. - [Release notes](https://github.com/pypa/pip/releases) - [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst) - [Commits](pypa/pip@21.0...21.0.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump coverage from 5.3.1 to 5.4 (GeoNode#6903) Bumps [coverage](https://github.com/nedbat/coveragepy) from 5.3.1 to 5.4. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@coverage-5.3.1...coverage-5.4) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pytest from 6.2.1 to 6.2.2 (GeoNode#6907) Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.1 to 6.2.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst) - [Commits](pytest-dev/pytest@6.2.1...6.2.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump djangorestframework-gis from 0.16 to 0.17 (GeoNode#6902) Bumps [djangorestframework-gis](https://github.com/openwisp/django-rest-framework-gis) from 0.16 to 0.17. - [Release notes](https://github.com/openwisp/django-rest-framework-gis/releases) - [Changelog](https://github.com/openwisp/django-rest-framework-gis/blob/master/CHANGES.rst) - [Commits](openwisp/django-rest-framework-gis@v0.16.0...v0.17.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * - Algin setup.cfg to requirements.txt * [Fixes GeoNode#6922][REST API v2] Expose the curated thumbnail URL if it has… (GeoNode#6923) * [Fixes GeoNode#6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded * - Add REST APIs test suite to CircleCI * [Fixes GeoNode#6918] Removal of QGIS support (GeoNode#6919) * [Cleanup and Refactor] Remove QGIS server backend dependencies * [Cleanup and Refactor] Remove QGIS server backend dependencies * - Fix LGTM issues * allow Basic authenticated requests in LOCKDOWN mode * fix to avoid circular import * flake8 check fix * added tests * [Fixes GeoNode#6880] Circle CI upload tests fail irregulary (GeoNode#6881) * [Fixes GeoNode#6880] Circle CI upload tests fail irregulary * CircleCI test fix: sometimes expires due to upload timeout in the test environment * - Avoid infinite loop on upload testing * Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment" This reverts commit 66139fd. Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it> Co-authored-by: afabiani <alessio.fabiani@gmail.com> * [Fixes GeoNode#6914] Remove "add to basket" tool for documents and maps (GeoNode#6915) * Added malnajdi as contributor * Bump pip from 21.0 to 21.0.1 (GeoNode#6900) Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1. - [Release notes](https://github.com/pypa/pip/releases) - [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst) - [Commits](pypa/pip@21.0...21.0.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * - Algin setup.cfg to requirements.txt * [Fixes GeoNode#6922][REST API v2] Expose the curated thumbnail URL if it has… (GeoNode#6923) * [Fixes GeoNode#6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded * - Add REST APIs test suite to CircleCI * [Fixes GeoNode#6918] Removal of QGIS support (GeoNode#6919) * [Cleanup and Refactor] Remove QGIS server backend dependencies * [Cleanup and Refactor] Remove QGIS server backend dependencies * - Fix LGTM issues * allow Basic authenticated requests in LOCKDOWN mode * fix to avoid circular import * - Align to upstream master branch * [Fixes GeoNode#7945] Ingest harvested layer data to geonode * Refactor celery tasks in order to allow performing harvesting of selected harvestable resources * Implement copying of remote resources for the GeoNode legacy harvester * Improve harvesting session and the admin * Ensure `extension` is present for harvested GeoNode documents * Add missing default attributes to harvested resources * fixing tests * fixing tests * [Fixes GeoNode#8148] [Harvesting] Adapt / extend remote service model * - Make sure we set all the "Remote" attributes correctly for a Dataset * - Getting rid of outdated comments * fix migration files conflict * [LGTM] Fix Issues * [CircleCI] Test Fixes * Renamed `geonode` harvester module to `geonodeharvester` Hoping to make lgtm happy * Add migrations to rename geonode harvester module name * Merge branch '7945-harvest-layer-data' of https://github.com/meomancer/geonode into ISSUE_8148 # Conflicts: # geonode/harvesting/harvesters/base.py * Merge branch '7945-harvest-layer-data' of https://github.com/meomancer/geonode into ISSUE_8148 # Conflicts: # geonode/harvesting/harvesters/base.py * [Flake8] Autopep8 * - Fixing migrations * [Dependencies] Align "setup.cfg" to "requirements.txt" * [Remote Services] Harvest a single resource instead of triggering all the marked ones again. * - Few more sanity checks on the 'remote_service' property existence * [Dependencies] Align "setup.cfg" to "requirements.txt" * - Make sure we don't save a dataset twice in case the "final" upload step has been called again * - Using "GeonodeLegacyHarvester" on "Remote Services" of type "GN_WMS" * [Fixes GeoNode#7945] Ingest harvested layer data to geonode * Refactor celery tasks in order to allow performing harvesting of selected harvestable resources * Implement copying of remote resources for the GeoNode legacy harvester * Improve harvesting session and the admin * Ensure `extension` is present for harvested GeoNode documents * Add missing default attributes to harvested resources * fixing tests * fixing tests * fix migration files conflict * Renamed `geonode` harvester module to `geonodeharvester` Hoping to make lgtm happy * Add migrations to rename geonode harvester module name * Improve support for harvesting raster datasets and disable harvesting maps * - Make sure ll_bbox and default links have been generated for the geonode resources * [Harvesters - WMS] Harvesters should merge remote url parameters if specified Harvester [Harvesters - WMS] Harvesters should use WMS operations to fetch the GetCapabilities Harvester [Harvesters - WMS] GeoNode resources are not correctly defined Harvester [Harvesters - WMS] The GeoNode-WMS service legacy type is missing Harvester [Harvester] GeoNode Legacy Harvester should fetch the metadata and thumbnail from the Remote resources * [Fixes GeoNode#7945] Ingest harvested layer data to geonode * Improve harvesting session and the admin * fix migration files conflict * Initial work for implementing stoppable harvesting sessions * Implement aborting of harvesting celery tasks Add the `AsynchronousHarvestingSession` model, which is used to implement sessions for both refreshing of a harvester's harvestable resources and for the harvesting of remote resources. Refactor the `admin`, `api` and `tasks` to use this new model. Moved some functions out of `harvesting.utils` module in order to avoid circular imports * fix tests * fix conflicts * Remove accidental duplication of code that crept in during conflict resolution * Add a couple more tests * Update signature of get_resource method in WMS harvester worker * Uncomment `settings.py` line that designates the GeoNode test runner as the one to be used * - Tests Fixes * - Adapt to the new "AsyncHarvestingSession" model changes * - Better management of "Refreshing" and "Harvesting" sessions status from the Remote Service detail page * - Tests Fixes * - General improvements to the PR and fixes to the migrations * - Cleaning up models and getting rid of unused proxy stuff * - Tests fixes * - "upgradeservices" management command * fix filter icon * - Fixes and improvements to the harvester WMS resource discovery * - Tests fixes * - Create thumbs on geonode resources from WMS harvesters * - Update harvester on "rescan" service action * fix filtering with TableFilter * fix filtering views and html * - Create thumbs on geonode resources from WMS harvesters * fix unefined checkbox id * - Create thumbs on geonode resources from WMS harvesters * - Remote Services: create thumbnail uses the original SRID * [Pep8] Fixing Flake8 issues * - Make sure the "thumb" is shown only after the resource has been fully uploaded * - WMS Harvester: make sure we parse the correct CRS and BBOX from GetCapa XML response * - Fix thumbnail generation through the APIs (cherry picked from commit 1c07928) # Conflicts: # geonode/base/api/views.py Co-authored-by: Giovanni Allegri <giohappy@gmail.com> Co-authored-by: allyoucanmap <bovio.stefano@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni <toni.schoenbuchner@csgis.de> Co-authored-by: Florian Hoedt <gannebamm@gmail.com> Co-authored-by: Mohammed Y. Alnajdi <mohdnagfy@gmail.com> Co-authored-by: biegan <bieganowski.rev@gmail.com> Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com> Co-authored-by: meomancer <meomancer@gmail.com> Co-authored-by: Ricardo Garcia Silva <ricardo.garcia.silva@gmail.com>
1 parent 0867642 commit 4cca7c0

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

73 files changed

+1484
-1731
lines changed

.env

+1
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ GEONODE_DB_CONN_MAX_AGE=0
3737
GEONODE_DB_CONN_TOUT=5
3838
DEFAULT_BACKEND_DATASTORE=datastore
3939
BROKER_URL=amqp://guest:guest@rabbitmq:5672/
40+
CELERY_BEAT_SCHEDULER=celery.beat:PersistentScheduler
4041
ASYNC_SIGNALS=True
4142

4243
SITEURL=http://localhost/

.env_dev

+1
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ GEONODE_DB_CONN_MAX_AGE=0
3737
GEONODE_DB_CONN_TOUT=5
3838
DEFAULT_BACKEND_DATASTORE=datastore
3939
BROKER_URL=amqp://admin:admin@localhost:5672//
40+
CELERY_BEAT_SCHEDULER=celery.beat:PersistentScheduler
4041
ASYNC_SIGNALS=False
4142

4243
SITEURL=http://localhost:8000/

.env_local

+1
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ GEONODE_DB_CONN_MAX_AGE=0
3737
GEONODE_DB_CONN_TOUT=5
3838
DEFAULT_BACKEND_DATASTORE=datastore
3939
BROKER_URL=amqp://admin:admin@localhost:5672//
40+
CELERY_BEAT_SCHEDULER=celery.beat:PersistentScheduler
4041
ASYNC_SIGNALS=False
4142

4243
SITEURL=http://localhost/

.env_test

+1
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ GEONODE_DB_CONN_MAX_AGE=0
3737
GEONODE_DB_CONN_TOUT=5
3838
DEFAULT_BACKEND_DATASTORE=datastore
3939
BROKER_URL=amqp://guest:guest@rabbitmq:5672/
40+
CELERY_BEAT_SCHEDULER=celery.beat:PersistentScheduler
4041
ASYNC_SIGNALS=False
4142

4243
SITEURL=http://localhost:8001/

celery-cmd

+3-3
Original file line numberDiff line numberDiff line change
@@ -3,20 +3,20 @@
33
# Luca Pasquali <luca.pasquali@geo-solutions.it>
44
CELERY_BIN=${CELERY_BIN:-"$(which celery||echo celery)"}
55
CELERY_APP=${CELERY_APP:-"geonode.celery_app:app"}
6-
# CELERY__STATE_DB=${CELERY__STATE_DB:-"/mnt/volumes/statics/worker.state"}
6+
CELERY__STATE_DB=${CELERY__STATE_DB:-"/mnt/volumes/statics/worker.state"}
77
# expressed in KB
88
CELERY__MAX_MEMORY_PER_CHILD=${CELERY__MAX_MEMORY_PER_CHILD:-"200000"}
99
CELERY__AUTOSCALE_VALUES=${CELERY__AUTOSCALE_VALUES:-"2,4"}
1010
CELERY__MAX_TASKS_PER_CHILD=${CELERY__MAX_TASKS_PER_CHILD:-"10"}
1111
CELERY__OPTS=${CELERY__OPTS:-"--without-gossip --without-mingle -Ofair -B -E"}
12-
CELERY__BEAT_SCHEDULE=${CELERY__BEAT_SCHEDULE:-"django_celery_beat.schedulers:DatabaseScheduler"}
12+
CELERY__BEAT_SCHEDULE=${CELERY__BEAT_SCHEDULE:-"celery.beat:PersistentScheduler"}
1313
CELERY__LOG_LEVEL=${CELERY__LOG_LEVEL:-"INFO"}
1414
CELERY__LOG_FILE=${CELERY__LOG_FILE:-"/var/log/celery.log"}
1515
CELERY__WORKER_NAME=${CELERY__WORKER_NAME:-"worker1@%h"}
1616
CELERY__WORKER_CONCURRENCY=${CELERY__WORKER_CONCURRENCY:-"4"}
1717

1818
$CELERY_BIN -A $CELERY_APP worker --autoscale=$CELERY__AUTOSCALE_VALUES \
1919
--max-memory-per-child=$CELERY__MAX_MEMORY_PER_CHILD $CELERY__OPTS \
20-
-s $CELERY__BEAT_SCHEDULE \
20+
--statedb=$CELERY__STATE_DB --scheduler=$CELERY__BEAT_SCHEDULE \
2121
--loglevel=$CELERY__LOG_LEVEL -n $CELERY__WORKER_NAME -f $CELERY__LOG_FILE \
2222
--concurrency=$CELERY__WORKER_CONCURRENCY --max-tasks-per-child=$CELERY__MAX_TASKS_PER_CHILD

celery.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
#!/bin/bash
22
nohup celery -A geonode.celery_app:app beat -l DEBUG -f /var/log/celery.log &>/dev/null &
3-
nohup celery -A geonode.celery_app:app worker --without-gossip --without-mingle -Ofair -B -E -s django_celery_beat.schedulers:DatabaseScheduler --loglevel=INFO --concurrency=2 -n worker1@%h -f /var/log/celery.log &>/dev/null &
3+
nohup celery -A geonode.celery_app:app worker --without-gossip --without-mingle -Ofair -B -E --statedb=worker.state --scheduler=celery.beat:PersistentScheduler --loglevel=INFO --concurrency=2 -n worker1@%h -f /var/log/celery.log &>/dev/null &
44
nohup celery -A geonode.celery_app:app flower --auto_refresh=True --debug=False --broker=${BROKER_URL} --basic_auth=${ADMIN_USERNAME}:${ADMIN_PASSWORD} --address=0.0.0.0 --port=5555 &>/dev/null &

celery_dev.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@ set -a
22
. ./.env_dev
33
set +a
44

5-
celery -A geonode.celery_app:app worker --without-gossip --without-mingle -Ofair -B -E -s django_celery_beat.schedulers:DatabaseScheduler --loglevel=DEBUG --concurrency=2 -n worker1@%h
5+
celery -A geonode.celery_app:app worker --without-gossip --without-mingle -Ofair -B -E --statedb=worker.state --scheduler=celery.beat:PersistentScheduler --loglevel=DEBUG --concurrency=2 -n worker1@%h

geonode/base/api/tests.py

+9-4
Original file line numberDiff line numberDiff line change
@@ -1019,7 +1019,9 @@ def test_set_thumbnail_from_bbox_from_logged_user_for_existing_dataset(self, moc
10191019
response = self.client.post(url, data=payload, format='json')
10201020

10211021
expected = {
1022-
"thumbnail_url": "http://localhost:8000/mocked_url.jpg"
1022+
'message': 'Thumbnail correctly created.',
1023+
'success': True,
1024+
'thumbnail_url': 'http://localhost:8000/mocked_url.jpg'
10231025
}
10241026
self.assertEqual(response.status_code, 200)
10251027
self.assertEqual(expected, response.json())
@@ -1044,7 +1046,8 @@ def test_set_thumbnail_from_bbox_from_logged_user_for_not_existing_dataset(self)
10441046
response = self.client.post(url, data=payload, format='json')
10451047

10461048
expected = {
1047-
"message": f"Resource selected with id {dataset_id} does not exists"
1049+
'message': f'Resource selected with id {dataset_id} does not exists',
1050+
'success': False
10481051
}
10491052
self.assertEqual(response.status_code, 404)
10501053
self.assertEqual(expected, response.json())
@@ -1064,7 +1067,8 @@ def test_set_thumbnail_from_bbox_from_logged_user_for_existing_doc(self):
10641067
response = self.client.post(url, data=payload, format='json')
10651068

10661069
expected = {
1067-
"message": "Not implemented: Endpoint available only for Dataset and Maps"
1070+
'message': 'Not implemented: Endpoint available only for Dataset and Maps',
1071+
'success': False
10681072
}
10691073
self.assertEqual(response.status_code, 405)
10701074
self.assertEqual(expected, response.json())
@@ -1085,7 +1089,8 @@ def test_set_thumbnail_from_bbox_from_logged_user_for_existing_dataset_raise_exp
10851089
response = self.client.post(url, data=payload, format='json')
10861090

10871091
expected = {
1088-
"message": "Some exception during thumb creation"
1092+
'message': 'Some exception during thumb creation',
1093+
'success': False
10891094
}
10901095
self.assertEqual(response.status_code, 500)
10911096
self.assertEqual(expected, response.json())

geonode/base/api/views.py

+21-7
Original file line numberDiff line numberDiff line change
@@ -553,30 +553,44 @@ def resource_service_permissions(self, request, pk=None):
553553
IsAuthenticated,
554554
])
555555
def set_thumbnail_from_bbox(self, request, resource_id):
556+
import traceback
557+
from django.utils.datastructures import MultiValueDictKeyError
556558
try:
557559
resource = ResourceBase.objects.get(id=ast.literal_eval(resource_id))
558560

559561
if not isinstance(resource.get_real_instance(), (Dataset, Map)):
560562
raise NotImplementedError("Not implemented: Endpoint available only for Dataset and Maps")
561563

562564
request_body = request.data if request.data else json.loads(request.body)
563-
bbox = request_body["bbox"] + [request_body["srid"]]
564-
zoom = request_body.get("zoom", None)
565+
try:
566+
bbox = request_body["bbox"] + [request_body["srid"]]
567+
zoom = request_body.get("zoom", None)
568+
except MultiValueDictKeyError:
569+
for _k, _v in request_body.items():
570+
request_body = json.loads(_k)
571+
break
572+
bbox = request_body["bbox"] + [request_body["srid"]]
573+
zoom = request_body.get("zoom", None)
565574

566575
thumbnail_url = create_thumbnail(resource.get_real_instance(), bbox=bbox, background_zoom=zoom, overwrite=True)
567-
return Response({"thumbnail_url": thumbnail_url}, status=200)
576+
return Response({"message": "Thumbnail correctly created.", "success": True, "thumbnail_url": thumbnail_url}, status=200)
568577
except ResourceBase.DoesNotExist:
578+
traceback.print_exc()
569579
logger.error(f"Resource selected with id {resource_id} does not exists")
570-
return Response(data={"message": f"Resource selected with id {resource_id} does not exists"}, status=404, exception=True)
580+
return Response(
581+
data={"message": f"Resource selected with id {resource_id} does not exists", "success": False}, status=404, exception=True)
571582
except NotImplementedError as e:
583+
traceback.print_exc()
572584
logger.error(e)
573-
return Response(data={"message": e.args[0]}, status=405, exception=True)
585+
return Response(data={"message": e.args[0], "success": False}, status=405, exception=True)
574586
except ThumbnailError as e:
587+
traceback.print_exc()
575588
logger.error(e)
576-
return Response(data={"message": e.args[0]}, status=500, exception=True)
589+
return Response(data={"message": e.args[0], "success": False}, status=500, exception=True)
577590
except Exception as e:
591+
traceback.print_exc()
578592
logger.error(e)
579-
return Response(data={"message": e.args[0]}, status=500, exception=True)
593+
return Response(data={"message": e.args[0], "success": False}, status=500, exception=True)
580594

581595
@extend_schema(
582596
methods=["post"], responses={200}, description="Instructs the Async dispatcher to execute a 'INGEST' operation."

geonode/base/models.py

+19-15
Original file line numberDiff line numberDiff line change
@@ -1022,10 +1022,13 @@ class Meta:
10221022

10231023
def __init__(self, *args, **kwargs):
10241024
# Provide legacy support for bbox fields
1025-
bbox = [kwargs.pop(key, None) for key in ('bbox_x0', 'bbox_y0', 'bbox_x1', 'bbox_y1')]
1026-
if all(bbox):
1027-
kwargs['bbox_polygon'] = Polygon.from_bbox(bbox)
1028-
kwargs['ll_bbox_polygon'] = Polygon.from_bbox(bbox)
1025+
try:
1026+
bbox = [kwargs.pop(key, None) for key in ('bbox_x0', 'bbox_y0', 'bbox_x1', 'bbox_y1')]
1027+
if all(bbox):
1028+
kwargs['bbox_polygon'] = Polygon.from_bbox(bbox)
1029+
kwargs['ll_bbox_polygon'] = Polygon.from_bbox(bbox)
1030+
except Exception as e:
1031+
logger.exception(e)
10291032
super().__init__(*args, **kwargs)
10301033

10311034
def __str__(self):
@@ -1507,17 +1510,18 @@ def set_center_zoom(self):
15071510
"""
15081511
Sets the center coordinates and zoom level in EPSG:4326
15091512
"""
1510-
bbox = self.ll_bbox_polygon
1511-
center_x, center_y = self.ll_bbox_polygon.centroid.coords
1512-
center = Point(center_x, center_y, srid=4326)
1513-
self.center_x, self.center_y = center.coords
1514-
try:
1515-
ext = bbox.extent
1516-
width_zoom = math.log(360 / (ext[2] - ext[0]), 2)
1517-
height_zoom = math.log(360 / (ext[3] - ext[1]), 2)
1518-
self.zoom = math.ceil(min(width_zoom, height_zoom))
1519-
except ZeroDivisionError:
1520-
pass
1513+
if self.ll_bbox_polygon and len(self.ll_bbox_polygon.centroid.coords) > 0:
1514+
bbox = self.ll_bbox_polygon.clone()
1515+
center_x, center_y = bbox.centroid.coords
1516+
center = Point(center_x, center_y, srid=4326)
1517+
self.center_x, self.center_y = center.coords
1518+
try:
1519+
ext = bbox.extent
1520+
width_zoom = math.log(360 / (ext[2] - ext[0]), 2)
1521+
height_zoom = math.log(360 / (ext[3] - ext[1]), 2)
1522+
self.zoom = math.ceil(min(width_zoom, height_zoom))
1523+
except ZeroDivisionError:
1524+
pass
15211525

15221526
def download_links(self):
15231527
"""assemble download links for pycsw"""

geonode/base/templatetags/base_tags.py

+3-11
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,12 @@
2929
from pinax.ratings.models import Rating
3030
from guardian.shortcuts import get_objects_for_user
3131

32-
from geonode.base.models import ResourceBase
33-
from geonode.base.bbox_utils import filter_bbox
34-
from geonode.layers.models import Dataset
3532
from geonode.maps.models import Map
33+
from geonode.layers.models import Dataset
34+
from geonode.base.models import ResourceBase
3635
from geonode.documents.models import Document
3736
from geonode.groups.models import GroupProfile
37+
from geonode.base.bbox_utils import filter_bbox
3838
from geonode.base.models import (
3939
HierarchicalKeyword, Menu, MenuItem
4040
)
@@ -423,11 +423,3 @@ def display_change_perms_button(resource, user, perms):
423423
return True
424424
else:
425425
return not getattr(settings, 'ADMIN_MODERATE_UPLOADS', False)
426-
427-
428-
@register.simple_tag
429-
def get_dataset_count_by_services(service_id, user):
430-
return get_visible_resources(
431-
queryset=Dataset.objects.filter(remote_service=service_id),
432-
user=user
433-
).count()

geonode/geoserver/helpers.py

+16-16
Original file line numberDiff line numberDiff line change
@@ -1483,12 +1483,11 @@ def fetch_gs_resource(instance, values, tries):
14831483
owner=instance.owner))
14841484
else:
14851485
msg = f"There isn't a geoserver resource for this layer: {instance.name}"
1486-
logger.exception(msg)
1486+
logger.debug(msg)
14871487
if tries >= _max_tries:
14881488
# raise GeoNodeException(msg)
14891489
return (values, None)
14901490
gs_resource = None
1491-
time.sleep(5)
14921491
return (values, gs_resource)
14931492

14941493

@@ -1683,20 +1682,6 @@ def sync_instance_with_geoserver(
16831682
except Exception as e:
16841683
logger.warning(e)
16851684

1686-
# Refreshing CSW records
1687-
logger.debug(f"... Updating the Catalogue entries for Dataset {instance.title}")
1688-
try:
1689-
catalogue_post_save(instance=instance, sender=instance.__class__)
1690-
except Exception as e:
1691-
logger.exception(e)
1692-
1693-
# Refreshing dataset links
1694-
logger.debug(f"... Creating Default Resource Links for Dataset {instance.title}")
1695-
try:
1696-
set_resource_default_links(instance, instance, prune=True)
1697-
except Exception as e:
1698-
logger.exception(e)
1699-
17001685
# Save dataset styles
17011686
logger.debug(f"... Refresh Legend links for Dataset {instance.title}")
17021687
try:
@@ -1711,6 +1696,21 @@ def sync_instance_with_geoserver(
17111696
except Exception:
17121697
pass
17131698

1699+
# Refreshing dataset links
1700+
logger.debug(f"... Creating Default Resource Links for Dataset {instance.title}")
1701+
try:
1702+
_prune = (gs_resource is not None)
1703+
set_resource_default_links(instance, instance, prune=_prune)
1704+
except Exception as e:
1705+
logger.exception(e)
1706+
1707+
# Refreshing CSW records
1708+
logger.debug(f"... Updating the Catalogue entries for Dataset {instance.title}")
1709+
try:
1710+
catalogue_post_save(instance=instance, sender=instance.__class__)
1711+
except Exception as e:
1712+
logger.exception(e)
1713+
17141714
return instance
17151715

17161716

geonode/geoserver/manager.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,7 @@ def delete(self, uuid: str, /, instance: ResourceBase = None) -> int:
115115
if instance and getattr(ogc_server_settings, "BACKEND_WRITE_ENABLED", True):
116116
_real_instance = instance.get_real_instance()
117117
if isinstance(_real_instance, Dataset) and hasattr(_real_instance, 'alternate') and _real_instance.alternate:
118-
if _real_instance.remote_service is None or _real_instance.remote_service.method == CASCADED:
118+
if not hasattr(_real_instance, 'remote_service') or _real_instance.remote_service is None or _real_instance.remote_service.method == CASCADED:
119119
geoserver_cascading_delete.apply_async((_real_instance.alternate,))
120120
if "geonode.upload" in settings.INSTALLED_APPS:
121121
from geonode.upload.models import Upload
@@ -555,7 +555,7 @@ def set_workflow_permissions(self, uuid: str, /, instance: ResourceBase = None,
555555
def set_thumbnail(self, uuid: str, /, instance: ResourceBase = None, overwrite: bool = True, check_bbox: bool = True) -> bool:
556556
if instance and not isinstance(instance.get_real_instance(), Document):
557557
if overwrite or instance.thumbnail_url == static(settings.MISSING_THUMBNAIL):
558-
geoserver_create_thumbnail.apply_async((instance.id, overwrite, check_bbox, ))
558+
geoserver_create_thumbnail.apply((instance.id, overwrite, check_bbox, ))
559559
return True
560560
return False
561561

geonode/geoserver/signals.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ def geoserver_pre_delete(instance, sender, **kwargs):
5454
# cascading_delete should only be called if
5555
# ogc_server_settings.BACKEND_WRITE_ENABLED == True
5656
if getattr(ogc_server_settings, "BACKEND_WRITE_ENABLED", True):
57-
if instance.remote_service is None or instance.remote_service.method == CASCADED:
57+
if not hasattr(instance, 'remote_service') or instance.remote_service is None or instance.remote_service.method == CASCADED:
5858
if instance.alternate:
5959
geoserver_cascading_delete.apply_async((instance.alternate,))
6060

geonode/geoserver/tasks.py

+6-1
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
AcquireLock,
3030
FaultTolerantTask)
3131
from geonode.base.models import Link
32+
from geonode.base import enumerations
3233
from geonode.layers.models import Dataset
3334
from geonode.base.models import ResourceBase
3435

@@ -217,7 +218,7 @@ def geoserver_post_save_datasets(
217218
expires=30,
218219
acks_late=False,
219220
autoretry_for=(Exception, ),
220-
retry_kwargs={'max_retries': 3, 'countdown': 10},
221+
retry_kwargs={'max_retries': 1, 'countdown': 10},
221222
retry_backoff=True,
222223
retry_backoff_max=700,
223224
retry_jitter=True)
@@ -235,11 +236,15 @@ def geoserver_create_thumbnail(self, instance_id, overwrite=True, check_bbox=Tru
235236
lock_id = f'{self.request.id}'
236237
with AcquireLock(lock_id) as lock:
237238
if lock.acquire() is True:
239+
instance.set_processing_state(enumerations.STATE_RUNNING)
238240
try:
241+
instance.set_dirty_state()
239242
create_gs_thumbnail(instance, overwrite=overwrite, check_bbox=check_bbox)
240243
logger.debug(f"... Created Thumbnail for Dataset {instance.title}")
241244
except Exception as e:
242245
geoserver_create_thumbnail.retry(exc=e)
246+
finally:
247+
instance.set_processing_state(enumerations.STATE_PROCESSED)
243248

244249

245250
@app.task(

geonode/groups/tests.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -276,7 +276,7 @@ def test_group_permissions_extend_to_user(self):
276276
Ensures that when a user is in a group, the group permissions
277277
extend to the user.
278278
"""
279-
layer = Dataset.objects.all()[0]
279+
layer = Dataset.objects.first()
280280
# Set the default permissions
281281
layer.set_default_permissions()
282282

geonode/harvesting/admin.py

+3
Original file line numberDiff line numberDiff line change
@@ -307,13 +307,15 @@ class HarvestableResourceAdmin(admin.ModelAdmin):
307307
"last_harvested",
308308
"unique_identifier",
309309
"title",
310+
"abstract",
310311
"show_link_to_harvester",
311312
"should_be_harvested",
312313
"remote_resource_type",
313314
)
314315
readonly_fields = (
315316
"unique_identifier",
316317
"title",
318+
"abstract",
317319
"harvester",
318320
"last_updated",
319321
"last_refreshed",
@@ -331,6 +333,7 @@ class HarvestableResourceAdmin(admin.ModelAdmin):
331333
)
332334
search_fields = (
333335
"title",
336+
"abstract",
334337
)
335338
list_editable = (
336339
"should_be_harvested",

0 commit comments

Comments
 (0)