Can we get one RPC server per node instead of one RPC server per GPU

[segmond]

Right now, we have to spin up as many RPC servers as we have GPUs on a remote note. Can we have just one RPC server per node? Could going that route improve performance? I get the feeling that the overhead of talking to N servers for every token generated will be much higher than talking to 1. It means of course that the remote node will need more logic and data to compute everything across it's GPU units. What do you all think?

[abc-nix]

@segmond, if I am understanding you and the PR correctly, #16276 from two days ago achieves this. Sorry if I am misunderstanding you.

Documentation coming soon #16441.

[rgerganov]

Correct, the RPC server is supporting multiple devices now and the documentation is updated. My benchmarks show large improvements in prompts processing and small improvements in token generation. Any feedback is appreciated as always.

[segmond]

Thank you very much! I'm going to rebuild my clusters and try this$!@#$