[core] Support CLI commands to kill Ray resources like tasks/actors/nodes/placement groups

[kevin85421]

Description

Some issues occur only at large scale. In such cases, most computations have already been completed, and deadlines are sometimes stringent, leaving users with no opportunity to rerun the entire job. Therefore, we should provide CLIs to enable users to terminate specific Ray tasks or actor processes, triggering retries of failed tasks so that the job will not hang forever due to some corner cases.

This is useful when users need to babysit some workloads.

Use case

No response

[kevin85421]

cc @rueian are you interested in taking a look at this issue?

[kevin85421]

cc @jjyao @edoakes

[robertnishihara]

Thanks for sharing! A couple questions.

1. Is the behavior you want the equivalent of manually ssh'ing into the machine and killing the relevant process?
2. Is the assumption that individual tasks can hang in corner cases and that if we kill the process and let Ray retry them, they may succeed the next time around?
3. What are the most common underlying causes?
4. How do you typically identify the right task to kill?
5. Would killing at the node level solve the issue as well?

[kevin85421]

Is the behavior you want the equivalent of manually ssh'ing into the machine and killing the relevant process?

Yes, this is what I do for now.

Is the assumption that individual tasks can hang in corner cases and that if we kill the process and let Ray retry them, they may succeed the next time around?

Yes

What are the most common underlying causes?

Hardware issues.

How do you typically identify the right task to kill?

• Use some methods to find which Ray resources hang.
• Try to find the IPs of the Ray resources.
• Search which K8s Pods have the corresponding IPs.
• Log in to the Pods
• kill $PID

Would killing at the node level solve the issue as well?

I think so.