bug: stackoverflow error with some repos

[cyclotruc]

For example, this repo is throwing a stackoverflow error in recent versions of main https://github.com/kmaurinjones/Ontario-Building-Code-Chat

[ix-56h]

In local with up-to-date main :

  File "/home/zero/gitingest/src/server/query_processor.py", line 91, in process_query
    summary, tree, content = ingest_query(query)
                             ^^^^^^^^^^^^^^^^^^^
  File "/home/zero/gitingest/src/gitingest/ingestion.py", line 85, in ingest_query
    return format_node(root_node, query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zero/gitingest/src/gitingest/output_formatter.py", line 52, in format_node
    token_estimate = _format_token_count(tree + content)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zero/gitingest/src/gitingest/output_formatter.py", line 188, in _format_token_count
    total_tokens = len(encoding.encode(text, disallowed_special=()))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zero/gitingest/.venv/lib/python3.11/site-packages/tiktoken/core.py", line 124, in encode
    return self._core_bpe.encode(text, allowed_special)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: RuntimeError(StackOverflow)

Tiktoken seems to panic, this is a rust error.

I'll try to investigate this more tomorrow.

[filipchristiansen]

This file https://github.com/kmaurinjones/Ontario-Building-Code-Chat/blob/main/data/vector_db/2c084fba-e16e-4ed4-8a31-f083e22cb767/header.bin

[ix-56h]

I get the issue with o200k_base but not with cl100k_base.

This occurs for a context where len(context) = 9097111 for kmaurinjones/Ontario-Building-Code-Chat but not with fastapi context which's larger : len(context) = 9387998.

Also, i think we should rework the tokenizer, at least, provide the possibility to choose the tokenizer for tiktoken : https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

[ix-56h]

I've found the issue 👍

In filesystem.py we have a check to exclude binary files :

        if not _decodes(chunk, "utf-8"):
            return "[Binary file]"

but this doesn't work for this file : https://github.com/kmaurinjones/Ontario-Building-Code-Chat/blob/main/data/vector_db/2c084fba-e16e-4ed4-8a31-f083e22cb767/data_level0.bin