How to make token generation deterministic?

[ThiloteE]

For testing/benchmarking purposes, I want to have the same response everytime.

I tried with following commands:

• llama-server 8080 --top-k 1 --n-predict 128 --reasoning-budget 0 --threads -1 --jinja --flash-attn auto --cache-type-k q8_0 --cache-type-v q8_0
• llama-server 8080 --temp 0 --n-predict 128 --reasoning-budget 0 --threads -1 --jinja --flash-attn auto --cache-type-k q8_0 --cache-type-v q8_0

I was hoping that forcing top-k to 1 or temp to 0 would do the trick.

Unfortunately, once I press the "regenerate" button in the llama-server GUI, it gives me varying responses.

System info:

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
 Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
build: 6683 (946f71ed) with MSVC 19.41.34120.0 for x64
system info: n_threads = 6, n_threads_batch = 6, total_threads = 12

I found some related (old) issues:

• CUDA non-determinism on identical requests #2838
• Eval bug: Nondeterministic output with ROCm backend despite zero temperature #14727