Duplicated characters near comma

[gabinguo]

Bug

We are extract text normally with docling with this pdf. We are getting duplicate characters.

Text in PDF

Text Extracted

• (4) Given the sensitivity of personal electronic health data, this Regulation seeks to provide sufficient safeguards at both Union and national level to ensure a high degree of data protection, security, y, confidentiality and ethical use. Such safeguards are necessary to promote trust in safe handling of electronic health data of natural persons for primary use and secondary use as defined in this Regulation.

here we have security, y, instead of security,

Steps to reproduce

ocr_options = TesseractOcrOptions()
device = AcceleratorDevice.AUTO

accelerator_options = AcceleratorOptions(
    num_threads=2,
    device=device,
)

pipeline_options = PdfPipelineOptions(
    do_ocr=False,
    ocr_options=ocr_options,
    table_structure_options=TableStructureOptions(
        mode=TableFormerMode.FAST, do_cell_matching=True
    ),
    do_table_structure=True,
    accelerator_options=accelerator_options,
    artifacts_path=self.docling_artifacts_path or None,
)

converter = CustomDocumentConverter(
    allowed_formats=DOCLING_ALLOWED_FORMATS,
    format_options={
        InputFormat.PDF: PdfFormatOption(
            backend=PyPdfiumDocumentBackend,
            pipeline_options=pipeline_options,
        ),
    },
)

# Then invoked like this:
md_text = converter.convert()

Docling version

docling                                  2.54.0
docling-core                             2.48.4
docling-ibm-models                       3.9.1
docling-parse                            4.5.0

Python version

Python 3.12.7

---

PDF File needed to reproduce this bug

PDF.pdf

[dosubot]

This issue is likely caused by how Docling's PDF backend handles font mapping and glyph conversion, especially with subsetted or complex fonts. Problems like duplicated or garbled characters near punctuation are known and have been reported by other users as well. The root cause often lies in how the PyPdfiumDocumentBackend interprets PDF text objects, font encodings, and layered or complex layouts, and sometimes in the underlying pypdfium2 library itself. There isn't a universal fix yet, but a few workarounds have helped in some cases:

• Try enabling OCR by setting do_ocr=True and ocr_options.force_full_page_ocr=True in your PdfPipelineOptions. This can sometimes bypass problematic text layers and extract cleaner text, though it is more resource-intensive and may not always help if the PDF text layer itself is corrupted or misinterpreted (source, source).
• If possible, flatten the PDF before processing. Layered PDFs can cause text to be "out of view" for the extraction engine, leading to duplication or missing text (source).
• Some users have reported that switching Docling versions or backends, or preprocessing PDFs, can help in specific cases, but results are inconsistent (source).
• The maintainers are aware of these extraction issues, especially with subsetted fonts and complex layouts, but there is no universal fix yet. Providing problematic PDFs for further investigation is encouraged (source).

If you try the OCR workaround and still see issues, or if flattening the PDF isn't feasible, you may need to wait for further improvements in Docling's font/glyph handling logic.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}