Cluster Prerequisites⚓︎
The following prerequisite components are assumed to be deployed in the cluster.
Nginx Ingress Controller⚓︎
# Install the Nginx Ingress Controller helm chart
helm upgrade -i --version='<4.5.0' \
--repo https://kubernetes.github.io/ingress-nginx \
ingress-nginx ingress-nginx \
--wait
Note
For Kubernetes version 1.22 and earlier the version of the Nginx Ingress Controller must be before v4.5.0.
To target the Nginx Ingress Controller the kubernetes.io/ingress.class: nginx
annotation must be applied to the Ingress resource…
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
...
Cert Manager⚓︎
# Install the Cert Manager helm chart
helm upgrade -i --namespace cert-manager --create-namespace \
--repo https://charts.jetstack.io \
--set installCRDs=true \
cert-manager cert-manager
Letsencrypt Certificates⚓︎
Once the Certificate Manager is deployed, then we can establish ClusterIssuer
operators in the cluster to support use of TLS with service Ingress
endpoints.
For Letsencrypt we can define two ClusterIssuer
- for production
and for staging
.
NOTE that these require the cluster to be publicly accessible, in order for the http01
acme flow to verify the domain ownership. Local development deployments will typically not have public IP/DNS - in which case the system deployment can proceed, but without TLS support for the service endpoints.
Production⚓︎
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-production
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: eoepca.systemteam@telespazio.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource that will be used to store the account's private key.
name: letsencrypt-production-account-key
# Add a single challenge solver, HTTP01 using nginx
solvers:
- http01:
ingress:
class: nginx
Staging⚓︎
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: eoepca.systemteam@telespazio.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource that will be used to store the account's private key.
name: letsencrypt-staging-account-key
# Add a single challenge solver, HTTP01 using nginx
solvers:
- http01:
ingress:
class: nginx
To exploit the specified ClusterIssuer the cert-manager.io/cluster-issuer
annotation must be applied to the Ingress resource. For example…
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-production
...
Sealed Secrets⚓︎
The EOEPCA development team maintain their deployment configurations in GitHub - for declarative, reproducible cluster deployments.
Various Secret
are relied upon by the system services. Secrets should not be exposed by commit to GitHub.
Instead SealedSecret
are committed to GitHub, which are encrypted, and can only be decrypted by the sealed-secret-controller
that runs within the cluster. The sealed-secret-controller
decrypts the SealedSecret
to a regular Secret
(of the same name) that can then be consumed by the cluster components.
The sealed-secret-controller
is deployed to the cluster using the helm chart…
helm install --version 2.1.8 --create-namespace --namespace infra \
--repo https://bitnami-labs.github.io/sealed-secrets \
eoepca-sealed-secrets sealed-secrets
Once the controller is deployed within the cluster, then the kubeseal
command can be used to create a SealedSecret
from a regular Secret
, as follows…
Create example Secret…
kubectl -n test create secret generic mysecret \
--from-literal=password=changeme \
--dry-run=client -o yaml \
> mysecret.yaml
Create SealedSecret from Secret using kubeseal…
kubeseal -o yaml \
--controller-name eoepca-sealed-secrets \
--controller-namespace infra \
< mysecret.yaml \
> mysecret-sealed.yaml
References⚓︎
MinIO Object Storage⚓︎
Various building blocks require access to an S3-compatible object storage service. In particular the ADES processing service expects to stage-out its processing results to S3 object storage. Ideally the cloud provider for your deployment will make available a suitable object storage service.
As a workaround, in the absence of an existing object storage, it is possible to use MinIO to establish an object storage service within the Kubernetes cluster. We use the minio helm chart provided by the MinIO Project.
# Install the minio helm chart
helm upgrade -i -f minio-values.yaml --namespace rm --create-namespace \
--repo https://charts.min.io/ \
minio minio \
--wait
Note
The Kubernetes namespace rm
is used above as an example, and can be changed according to your deployment preference.
The minio deployment is customised via the values file minio-values.yaml
, for example…
existingSecret: minio-auth
replicas: 2
ingress:
enabled: true
ingressClassName: nginx
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: '600'
path: /
hosts:
- minio.192-168-49-2.nip.io
tls:
- secretName: minio-tls
hosts:
- minio.192-168-49-2.nip.io
consoleIngress:
enabled: true
ingressClassName: nginx
annotations:
cert-manager.io/cluster-issuer: "letsencrypt"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: '600'
path: /
hosts:
- console.minio.192-168-49-2.nip.io
tls:
- secretName: minio-console-tls
hosts:
- console.minio.192-168-49-2.nip.io
resources:
requests:
memory: 1Gi
persistence:
storageClass: standard
buckets:
- name: eoepca
- name: cache-bucket
Note
- The example values assuming a TLS configuration using
letsencrypt
certificate provider - The admin credentials are provided by the Kubernetes secret named
minio-auth
- see below - The annotation
nginx.ingress.kubernetes.io/proxy-body-size
was found to be required to allow transfer of large files (such as data products) through the nginx proxy
Minio Credentials Secret⚓︎
The Minio admin credentials are provided via a Kubernetes secret that is referenced from the Minio helm chart deployment values. For example…
kubectl -n rm create secret generic minio-auth \
--from-literal=rootUser="eoepca" \
--from-literal=rootPassword="changeme"
Note
The secret must be created in the same Kubernetes namespace as the Minio service deployment - e.g. rm
namespce in the example above.
s3cmd Configuration⚓︎
The s3cmd
can be configured for access to the MinIO deployment. The --configure
option can be used to prepare a suitable configuration file for s3cmd
…
In response to the prompts, the following configuration selections are applicable to the above settings…
Access Key: eoepca
Secret Key: changeme
Default Region: us-east-1
S3 Endpoint: minio.192-168-49-2.nip.io
DNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Save the configuration file, and check access to the S3 object store with…
For example, using our sample deployment, the following can be used to interface with the MinIO service deployed in minikube…