Docker: Cannot upload files with non-ASCII characters in the name

[syldor]

Is your feature request related to a problem? Please describe.
It's currently not possible to upload pdf with non ASCII names. This is the same file uploaded with different names and only "hello.pdf" works.

Describe the solution you'd like
I would like to be able to upload files with non-ASCII characters in the name.

[syldor]

I see in the logs the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/geonode/documents/views.py", line 127, in post
    return super().post(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/views/generic/edit.py", line 172, in post
    return super().post(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/views/generic/edit.py", line 142, in post
    return self.form_valid(form)
  File "/usr/local/lib/python3.10/dist-packages/geonode/documents/views.py", line 168, in form_valid
    filepath = storage_manager.save(f"{dirname}/{file.name}", file)
  File "/usr/local/lib/python3.10/dist-packages/geonode/storage/manager.py", line 164, in save
    return self._concrete_storage_manager.save(name, content, max_length=max_length)
  File "/usr/local/lib/python3.10/dist-packages/geonode/storage/manager.py", line 300, in save
    return self._fsm.save(name, content, max_length=max_length)
  File "/usr/local/lib/python3.10/dist-packages/django/core/files/storage.py", line 54, in save
    name = self._save(name, content)
  File "/usr/local/lib/python3.10/dist-packages/django/core/files/storage.py", line 279, in _save
    fd = os.open(full_path, self.OS_OPEN_FLAGS, 0o666)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 42-48: ordinal not in range(128)

[syldor]

If we replace

documents/views.py L168

            filepath = storage_manager.save(f"{dirname}/{file.name}", file)

by

            filepath = storage_manager.save(f"{dirname}/{file.name}".encode('utf-8'), file)

or maybe (untested yet)

            file.name = file.name.encode('utf-8')

I was able to upload this file:

However it's creating another problem that the extension get a ' added to it.

Manually removing this ' ( with update documents_document set extension = 'jpeg; ) makes the document available (though not the thumbnail) so it should be removed at the saving time, but how to do?

[syldor]

Maybe it's a django thing? Though inside the container it seems to be using utf8

[syldor]

I think I have a solution from this: https://stackoverflow.com/questions/9208802/unicodeencodeerror-when-uploading-files-in-django-admin

In docker we need to add this step:

ENV LANG en_US.UTF-8
RUN apt install -y locales && \
    sed -i -e "s/# $LANG.*/$LANG UTF-8/" /etc/locale.gen && \
    locale-gen --purge &&\
    update-locale LANG=$LANG

With that I could upload a file with non-ascii title and it didn't mess up things like my previous attempts.

I take the liberty to tag you @giohappy as last time you rightfully moved my issue GeoNode/geonode-project#450 from here to geonode-project.

[giohappy]

@syldor interesting. I would investigate a bit more to understand why this solution works. This looks a solution to a very old issue with Django, that I expected it was solved.

@afabiani can you take a look?