Welcome to the systems administrator guide to the ELIXIR Cloud. Whether you would like to onboard your data or compute center, set up your own GA4GH-based cloud or simply play around with our compute and storage solutions, this is the right place to get you off the ground.
General deployment notes
Most of our services (see our GitHub organization for a comprehensive list) come with Helm charts for deployment on Cloud Native infrastructure and Docker Compose configurations for testing/development deployments. If you do not have experience with these technologies, please find some brief primers with references to additional documentation below.
Helm is an IaC tool that is described as the "package manager for Kubernetes". It allows the management of the lifecycle of a Kubernetes application, i.e., its deployment, configuration, upgrade, retiring, etc. Applications ara packaged into "Charts". Using Helm Charts allows us to version control an application and therefore follow its evolution over time, make identical copies (e.g., development, staging, production), make predictable upgrades, and share/publish the application.
Some useful Helm commands to manage a Chart are:
helm create: Create a Helm Chart
helm install: Install an application
helm upgrade: Upgrade an application
helm uninstall: Uninstall an application
Using Docker Compose
Most of our services provide a Docker Compose configuration
file for easy deployment of the software on a local machine. If the Docker
Engine and Docker Compose are already
installed on your system, it is as simple as cloning the service's Git
repository, changing into the folder where the Docker Compose file resides
docker-compose.yml in a repository's root directory) and running
docker-compose up -d
Non-standard name or location of config file
The command will be different if the Docker Compose config file is not in
the current working directory and/or is not called
This will bring the service up. The argument
--detach) starts the
app in daemonized mode, i.e., all launched containers that compose creates run
in the background.
In order to stop the deployment, simply run:
Onboarding your compute center
Follow the instructions below to onboard your compute node with the ELIXIR Cloud. Afterwards, your compute cluster will be accessible through the GA4GH Task Execution Service (TES) API and, optionally, available in the ELIXIR Cloud compute network.
TESK uses the Kubernetes Batch API (Jobs) to schedule execution of TES tasks. This means that it should be possible to deploy TESK in any flavor of Kubernetes, but tests are currently only performed with Kubernetes, OpenShift, and Minikube. Follow these instructions if you wish to deploy a TES endpoint on your Native Cloud cluster, and please let us know if you deploy TESK in any new and interensting platform.
TESK currently does not use any other storage (DB) than Kubernetes itself. Persistent Volume Claims are used as a temporary storage to handle input and output files of a task and pass them over between executors of a task. Note that PVCs are destroyed immediately after task completion! This means your cluster will need to provide a ReadWriteMany StorageClass. Commonly used storage classes are NFS and CephFS.
Here is an overview of TESK's architecture:
Follow these steps:
- Install Helm
Clone the TESK repository:
git clone https://github.com/elixir-cloud-aai/TESK.git
Find the Helm chart at
- Edit file
values.yaml(see notes below)
- Log into the cluster and install TESK with:
helm install -n TESK-NAMESPACE TESK-DEPLOYMENT-NAME . \ -f secrets.yaml \ -f values.yaml
TESK-NAMESPACEwith the name of the namespace where you want to install TESK. If the namespace is not specified, the default namespace will be used.
- The argument provided for
TESK-DEPLOYMENT-NAMEwill be used by Helm to refer to the deployment, for example when upgrading or deleting the deployment. You can choose whichever name you like.
You should now have a working TESK isntance!
Notes for editing chart values
host_name: Will be used to serve the API.
storageClass: Specify the storage class. If left empty, TESK will use the default one configred in the Kubernetes cluster.
auth.mode: Enable (
auth) or disable (
noauth; default) authentication. When enabled, an OIDC client must be in a file
./secrets.yaml, with the following format:
auth: client_id: <client_id> client_secret: <client_secret>
ftp: Which FTP credentials mode to use. Two options are supported:
.classic_ftp_secretfor basic authentication (username and password) or
.netrc_secretfor using a
For the classic approach, you must write in
ftp: classic_ftp_secret: ftp-secret
And in a file
.secrets.yaml write down the username and password as:
ftp: username: <username> password: <password>
.netrc approach, create a
.netrc file in the
ftp folder with
the connections details in the correct format.
clusterType: Type of Kubernetes flavor. Currently supported:
When creating a
.secrets.yaml file, ensure that the file is never shared
or committed to a code repository!
- Make sure the build dependencies
makeand Go 1.11+ are installed,
GOPATHis set and
GOPATH/binis added to
For example, in Ubuntu this can be achieved via:
sudo apt update sudo apt install make golang-go export GOPATH=/your/desired/path export PATH=$GOPATH/bin:$PATH go version
- Clone the repository:
git clone https://github.com/ohsu-comp-bio/funnel.git
- Build Funnel:
cd funnel make
- Test the installation by starting the Funnel server with:
funnel server run
If all works, Funnel should be ready for deployment on your HPC/HTC.
For the use of Funnel with Slurm, make sure the following conditions are met:
funnelbinary must be placed in a server with access to Slurm.
- A config file must be created and placed on the same server. This file can be used as a starting point.
- If we would like to deploy Funnel as a Systemd service,
this file can be used as a template. Set the
correct paths to the
funnelbinary and config file.
If successfull Funnel should be listening on port
More info coming soon...
Follow the instructions below to connect your TES endpoint to one or more ELIXIR Cloud cloud storage solutions. The currently supported solutions are:
Other storage solutions
Other S3 and FTP implementations may work but have not being tested.
Deploying MinIO (Amazon S3)
If you are deploying Minio to OpenShift, you may find this Minio-OpenShift template useful.
- It is required to activate secure FTP support with
- For onboarding with the ELIXIR Cloud, currently the server should have one account with a specific username and password created. Please contact us for details.
Registering your TES service
We are currently working on implementing access control mechanisms and providing a user interface for the ELIXIR Cloud Registry. Once available, we will add registration instructions here. For now, please let us know about your new TES endpoint by email.
Custom cloud deployments
More info coming soon...