Skip to content

Processing Deployment⚓︎

Overview⚓︎

A deployment wrapper script has been prepared for a ‘processing’ deployment - that is focused on the ADES and the deployment/execution of processing jobs.

The script deploy/processing/processing achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/processing/processing-options.

The processing deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS
    To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • ADES for processing
    • Minio for S3 object storage
  • ADES stage-out to Minio
  • Open ingress are enabled for unauthenticated access to ADES service
  • Other eoepca services not deployed

Initiate Deployment⚓︎

Deployment is initiated by invoking the script…

./deploy/processing/processing

The ADES service is accessed at the endpoint zoo-open.192-168-49-2.nip.io.

Post-deploy Manual Steps⚓︎

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

Example Requests⚓︎

Some sample requests have been prepared in the subdirectory deploy/samples/requests/processing - for example…

  • convert
    Provides a ‘hello world’ processing example that can be used simply to check that the processing capability has been well deployed
  • snuggs
    Provides a packaged EO exploitation algorithm that perform ‘real’ work and, as such, is more resource demanding (10GB RAM, 3 CPU) - and so may not be suitable for execution within a local minikube deployment (depending on resource allocations)

These sample http files have been prepared with sample requests for OGC API Processes operations:

  • List Processes
  • Deploy Process
  • Get Process Details
  • Execute Process
  • Get Job Status
  • Get Job Results

Note

  • The first requests in the file provide optional calls to obtain a user ID token (openidConfiguration / authenticate).
    These are to be used in the case that protected (not ‘open’) endpoints are deployed.
  • The file describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client.
    Install in vscode with ext install humao.rest-client.
  • The variables @hostname and @domain can be configured at the top of the file.

Alternative curl Commands⚓︎

Alternatively the following curl commands can be used…

List Processes 
curl -k \
  --request GET \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \
  --header 'accept: application/json'
Deploy & Execute (convert)
Deploy Process (convert) - By Reference (JSON) 
curl -k \
  --request POST \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --data '{"executionUnit": {"href": "https://raw.githubusercontent.com/EOEPCA/convert/main/convert-url-app.cwl","type": "application/cwl"}}'
Deploy Process (convert) - Inline (CWL) 
curl -k \
  --request POST \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \
  --header 'accept: application/json' \
  --header 'content-type: application/cwl+yaml' \
  --data '< convert-url-app.cwl'
Get Process Details (convert) 
curl -k \
  --request GET \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url \
  --header 'accept: application/json'
Execute Process (convert) 
curl -k -v \
  --request POST \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url/execution \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --header 'prefer: respond-async' \
  --data '{"inputs": {"fn":  "resize","url": "https://eoepca.org/media_portal/images/logo6_med.original.png", "size": "50%"},"response":"raw"}'
Undeploy Process (convert) 
curl -k -v \
  --request DELETE \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url \
  --header 'accept: application/json'
Deploy & Execute (snuggs)
Deploy Process (snuggs) 
curl -k \
  --request POST \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --data '{"executionUnit": {"href": "https://raw.githubusercontent.com/EOEPCA/deployment-guide/eoepca-v1.4/deploy/samples/requests/processing/snuggs.cwl","type": "application/cwl"}}'
Get Process Details (snuggs) 
curl -k \
  --request GET \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs \
  --header 'accept: application/json'
Execute Process (snuggs) 
curl -k -v \
  --request POST \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs/execution \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --header 'prefer: respond-async' \
  --data '{"inputs": {"input_reference":  "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A","s_expression": "ndvi:(/ (- B05 B03) (+ B05 B03))"},"response":"raw"}'
Undeploy Process (snuggs) 
curl -k -v \
  --request DELETE \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs \
  --header 'accept: application/json'
Get Job Status

This request requires the Location header from the response to the execute request. This will be of the form http://zoo-open.192-168-49-2.nip.io/{user}/ogc-api/jobs/{job-id} - e.g. http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e.

curl -k \
  --request GET \
  --url {location-header} \
  --header 'accept: application/json'
Get Job Results

This request uses the same URL as Get Job Status, with the additional URL path /results - i.e. /{user}/ogc-api/jobs/{job-id}/results - e.g. /eric/ogc-api/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e/results

curl -k \
  --request GET \
  --url {location-header}/results \
  --header 'accept: application/json'

The response indicates the location of the results, which should be in the minio object storage. See Processing Results.

The response also provides links to log files regarding each step of the Application Package workflow execution - which may be useful for debugging.

List Jobs 
curl -k \
  --request GET \
  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/jobs \
  --header 'accept: application/json'

Processing Results⚓︎

The outputs are published as a static STAC catalogue to a path that includes the unique job ID.

In the default configuration, the processing results are pushed to the Minio S3 object storage. This can be checked via browser access at the endpoint http://console.minio.192-168-49-2.nip.io/, or using an S3 client such as…

s3cmd -c ./deploy/cluster/s3cfg ls s3://eoepca

For the default credentials to connect to Minio see Minio Object Storage Default Credentials.

Note

If the ADES deployment has been configured to stage-out to the user’s Workspace, then the above s3cmd and credentials would have to be adjusted accordingly - for example the bucket s3://ws-eric.