Updated flow diagrams and finished detailing free compute flow.

Author: mariacarmina

.gitbook.yaml

@@ -79,7 +79,7 @@ redirects:
     data-science/ocean.py/publish-flow: data-scientists/ocean.py/publish-flow.md
     data-science/ocean.py/remote-setup: data-scientists/ocean.py/remote-setup.md
     data-science/ocean.py/technical-details: data-scientists/ocean.py/technical-details.md
-    "developers/ocean.py": data-scientists/ocean.py/README.md
+    developers/ocean.py: data-scientists/ocean.py/README.md
     developers/ocean.py/compute-flow: data-scientists/ocean.py/compute-flow.md
     developers/ocean.py/consume-flow: data-scientists/ocean.py/consume-flow.md
     developers/ocean.py/datatoken-interface-tech-details: data-scientists/ocean.py/datatoken-interface-tech-details.md

.gitbook/assets/c2d/Free Flow for Compute to Data - Part 1.png

@@ -51,6 +51,11 @@ group Start compute job
     consumer -> ocean_js: Calls freeStartCompute with resources from env.
     ocean_js -> ocean_node: **POST /freeCompute**
     ocean_node -> ocean_node: Checks nonce, creates job ID.
+    alt Nonce and signature are invalid
+        ocean_node --> ocean_js: Returns 500, error <i>'Invalid nonce or signature, unable to proceed.'</i>
+        ocean_js --> consumer: Returns error
+        consumer --> end_user
+    end
     ocean_node -> ocean_node: Checks if the assets are orderable + <font color=red>have grant access to run compute jobs</font>
     group Credentials check
         alt Policy server configured

.gitbook/assets/c2d/Free Flow for Compute to Data - Part 2.png

@@ -25,7 +25,7 @@ end group
 
 group Retrieve compute job results
     end_user -> consumer: Requests results, provide path.
-    consumer -> ocean_js: Calls computeStatus.
+    consumer -> ocean_js: Calls computeResult.
     ocean_js -> ocean_node: **GET /computeResult**
     ocean_node -> db: Requests job from specific C2D engine.
     db --> ocean_node: Returns job from specific C2D engine.

.gitbook/assets/c2d/Paid Flow for Compute to Data - Part 2.png

@@ -16,6 +16,11 @@ participant "Smart Contracts" as smart_contracts
 group Start compute job
     consumer -> ocean_js: Calls startCompute with resources from env.
     ocean_js -> ocean_node: **POST /compute**
+    alt Nonce and signature are invalid
+        ocean_node --> ocean_js: Returns 500, error <i>'Invalid nonce or signature, unable to proceed.'</i>
+        ocean_js --> consumer: Returns error
+        consumer --> end_user
+    end
     ocean_node -> ocean_node: Checks if the assets are orderable + <font color=red>have grant access to run compute jobs</font>
     group Credentials check
         alt Policy server configured
@@ -89,7 +94,7 @@ end group
 
 group Retrieve compute job results
     end_user -> consumer: Requests results, provide path.
-    consumer -> ocean_js: Calls computeStatus.
+    consumer -> ocean_js: Calls computeResult.
     ocean_js -> ocean_node: **GET /computeResult**
     ocean_node -> db: Requests job from specific C2D engine.
     db --> ocean_node: Returns job from specific C2D engine.

.gitbook/assets/c2d/free-compute-flow-1.puml

@@ -39,8 +39,59 @@ The consumer tool makes a request to the node **GET /computeEnvironments** and t
 The end user selects the free resources and fills in the consumer tool together with job duration that the user considers is needed
 for the algorithm execution.
 
+### Free start compute
+#### Nonce & signature check
 Consumer tool calls through ocean.js `freeStartCompute` which requests in Ocean Node **POST /freeCompute**. Within Ocean Node,
 nonce and signature provided by ocean.js is checked. In case of invalid nonce or signature, node returns __500, 'Invalid nonce or signature, unable to proceed.'__
 
+#### Credentials check
+If the node has configured `POLICY_SERVER_URL` and ddo contains credentials, the credentials check is performed in `Policy Server`, otherwise node performs credentials check for consumer address. In case of failure, node returns back to the consumer tool __403, 'Error: Access to asset ${ddo.id} was denied'__
 
+After these checks are performed and are successful, job id is generated and C2D engine is called for the actual algorithm execution.
+
+#### C2D Docker Engine
+
+The only supported engine for start compute (free and paid) is the one for Docker.
+The following steps executed by C2D Docker engine class are:
+
+1. C2D Engine validates the Docker image by checking manifest operating system and operating system architecture with the ones from environment platform. The manifest from the Docker image is retrieved from the `tag` or `image digest hash` using Docker SDK.
+If validation is failed, Node throws error withtin the engine:
+__Unable to validate docker image__ and creation of the job stops.
+
+2. Creates the folders for datasets, algorithm and result folder of algorithm execution, such as `/data/inputs`, `/data/transformations`, `/data/ddos`, `/data/outputs`.
+
+3. Saves the job structure into `SQLite` database.
+
+4. Starts monitoring the job execution and persists journalizing the lifecycle status of the job in `SQLite` database.
+
+Whenever a job has started, an internal loop which monitors all the new jobs is triggered. The loop determines the lifecycle of a compute job execution.
+**Lifecycle of a job according to statuses:**
+
+`JobStarted` -> `PullImage` or `PullImageFailed` -> `ConfiguringVolumes` or `VolumeCreationFailed` -> `Provisioning` or `ContainerCreationFailed` -> `RunningAlgorithm` or `AlgorithmFailed` -> `PublishingResults` or `ResultsUploadFailed`
+
+**Sequence of steps for internal loop:**
+1. Pulling docker image for the algorithm - if failure -> throws error and returns back to the consumer tool and updates job status `SQLite` database in `PullImageFailed`.
+2. Configuring volumes for the dedicated algorithm container - if failure -> throws error and returns back to the consumer tool and updates job status `SQLite` database in `VolumeCreationFailed`.
+3. Create Docker container for the algorithm - if failure -> throws error and returns back to the consumer tool and updates job status `SQLite` database in `ContainerCreationFailed`.
+4. Triggers algorithm execution on dedicated container - if failure from the algorithm -> throws error and returns back to the consumer tool and updates job status `SQLite` database in `AlgorithmFailed`.
+5. Publish results in `/data/outputs` even if the algorithm execution was successful or not - if failure -> throws error and returns back to the consumer tool and updates job status `SQLite` database in `ResultsUploadFailed`.
+If publishing results step was executed successfully, the container and volumes will be deleted together with the folders
+for datasets, algorithms and results.
+
+**Observation**: If a job exceeds its specified duration, C2DEngine internal loop will terminate the allocated container and volumes which facilitates algorithm's execution and sets the job to `PublishingResults`, meaning will perform a forced cleanup of the job setup.
+
+### Get job status
+
+To display the progress to the end user, the consumer tool requests the node at **GET /compute** with job ID for the job status through ocean.js method `computeStatus`.
+
+In case of request failure from node side, the error is retruned back to the consumer tool and displayed to the end user.
+
+### Retrieve compute job results
+
+If compute job status is `PublishingResults`, consumer tool will
+call ocean.js `computeResult` method which requests from node
+on endpoint `GET /computeResult`. Node returns to ocean.js the results content and ocean.js generates a downloadable URL to pass further to the consumer tools.
+
+In case of request failure from node side, the error is retruned back to the consumer tool and displayed to the end user.
+Consumer tools received the downloadable URL and will fetch the BLOB content from it and store on end user's specified results folder path.