[data] [streaming] Support/test fault tolerance with ActorPoolStrategy

[ericl]

Implement fault tolerance for the actor pool strategy in MapOperator. This involves enabling actor restarts on failure for MapOperator, and testing this out in practice.

There are two levels of testing:

• Actor faults: can randomly kill x% of actors (e.g., os.exit() in inference call) and see if it can recover.
• Lineage faults: can randomly kill x% of raylets (e.g.. pkill raylet in inference call) and see if it can recover lost objects.

One limitation of our actor fault tolerance right now is it requires the identical original actor to complete reconstruction. This may be a problem.

[anyscalesam]

Revewed with @c21 < this is still relevant but not currently opened. Want to continue to keep on our backlog.

[Superskyyy]

Hi, @ericl @anyscalesam could you elaborate a bit on this? I don't quite get the problem described by One limitation of our actor fault tolerance right now is it requires the identical original actor to complete reconstruction. This may be a problem. Could you kindly point me to the doc/code where it showcases the limitation on why the ray core actor fault tolerance doesn't work for actorpool?