Installing on your Kubernetes cluster¶
This page is for installing a workspace on a Kubernetes cluster of your own. You run the same
installer as on a single host, but on a jump host instead of on the server. It fetches the
release with your account, takes your configuration, applies the workspace to the cluster your
kubectl points at, builds the customer layer of your images with the jump host's docker and
runs the smoke test. Upgrades and rollbacks use the same commands as on a single host.
What the cluster must provide¶
A conformant Kubernetes cluster with:
- dynamic provisioning through a default StorageClass (ReadWriteOnce is enough),
- an Ingress controller that terminates TLS for the workspace's host names, or a proxy in front of it that does,
- NetworkPolicy enforcement by the CNI,
- user namespaces for pods (Kubernetes 1.33 or later, containerd 2.0 or later, nodes with
Linux 6.3 or later), because the workspace builds apps in pods with a user namespace of their own.
Inside that namespace a build runs as root with the capabilities
SYS_ADMIN,NET_ADMINandSYS_PTRACEand without a seccomp or AppArmor profile. A cluster that enforces Pod Security Standards or policies of its own must admit such pods in the workspace's namespace, - outbound access to the package repositories named in
velonde.env, or to your mirrors, - outbound access from the nodes to Docker Hub and to
public.ecr.aws. Besides the workspace's own images, which come from your workspace's registry, the cluster pulls ShinyProxy and its operator, Redis, the registry, Caddy, BuildKit, nginx and a few small helper images from there (with the package repository also RDepot and PostgreSQL). In a network that reaches no public registry, these images need a mirror that the nodes are configured for, - a kubeconfig with cluster-admin.
We recommend a limit on the processes per pod (the kubelet's podPidsLimit, 4096 is ample): without
one, a fork bomb in a user's session is held only by its memory limit. The single-host install sets it.
The workspace's namespace must stay out of a service mesh. A mesh proxy in the workspace's pods hides the address that the workspace recognises sessions by. Security and data explains why.
Some controllers pass on request headers whose names contain underscores, others reject them. The workspace does not depend on either: it sets every spelling of the headers that name the user itself for apps, and removes them in front of APIs, so a request cannot claim another user. If your controller rejects such headers, leave that setting on as a second line of defence.
Optionally, the cluster can provide a ReadWriteMany StorageClass for pipeline stores that apps
mount, an S3-compatible bucket for backups and for a stateless registry, and a node pool for
user code (see session_pool).
The jump host needs Ubuntu or Debian with bash, sudo, kubectl (its current context set
to the cluster) and docker with its buildx plugin (docker-buildx with Ubuntu's docker.io). The installer installs curl, jq, rsync, openssl and python3 with
apt when they are missing, and it needs ssh-keygen (the package openssh-client) to check the release's signature. The jump host must be able to reach the cluster, our distribution
registry, your package mirrors and your registry. The installer keeps its releases and state
under /opt/velonde, or under VELONDE_HOME if you set it.
Before the install, DNS for the host names must point at your ingress, and TLS must work there: the installer pushes the customer layer to the registry's host name, and the smoke test opens the workspace's.
Install¶
curl -fsSLo install.sh https://images.hub.linnerprogramming.com/install.sh
bash install.sh generate --cluster --name acme --domain velonde.acme.example --account acme
Check the downloaded installer first, as on a single host.
generate writes the overlay to /opt/velonde/overlay/acme. It takes the same
options as on a single host: features, IDEs, --auth oidc, --monitoring
and --bucket. On a cluster, --monitoring installs Prometheus, Grafana, Loki and
Alertmanager cluster-wide in the namespaces monitoring and loki, and --bucket installs
Velero cluster-wide in the namespace velero. Add --ingress-class <class> when your
controller is not Traefik. Add --ha to keep the workspace up when a node fails (see
High availability). When kubectl reaches the cluster, generate reads the API
server's address and the pod range from it. Otherwise it writes placeholders, and the apply
refuses them until you fill them in. If we maintain the overlay for you, skip this step:
install fetches it instead.
Then review the settings that depend on your cluster, which the generator cannot know:
patches/quota.yamlsets what the workspace may use of the cluster. The generated numbers are those of the example configuration.user_networksinvelonde.envlists the private networks that user code may reach. The workspace lets sessions, apps and builds out to the internet and to no private address, because the cluster's nodes, its other namespaces and its storage have such addresses.generatewrites the addresses of the workspace's own host names when they are private. Add your databases, package mirrors, IdP and git server if they have private addresses, each as a network or a network with ports, such as10.20.0.0/16 10.0.5.10/32:5432. The check tells you when the registry or a package repository cannot be reached under these rules.- The ingress class is set in four places, which
--ingress-classfills in together: thekubernetesIngressPatchesinpatches/shinyproxy.yamlandpatches/shinyproxy-apps.yaml,patches/registry.yaml, andingress_classinvelonde.envfor the Ingress of each API. - The workspace's network policies admit the ingress controller from the namespace it runs in, and
let app builds and the deploy service reach it there. That is
kube-systemunless you setingress_namespaceinvelonde.env. Set it whenever your controller runs in a namespace of its own, as it usually does when installed with Helm. The check tests that builds on every node reach the registry through it. - The controller terminates TLS. The registry's Ingress takes a
tls:block inpatches/registry.yaml, for instance with cert-manager's annotation. The Ingresses of the workspace's host name and of the apps host take the same through theirkubernetesIngressPatches. The Ingress of each deployed API names the TLS Secret thatapi_tls_secretinvelonde.envgives for the apps host. Without that key it names none, and the controller's default certificate must cover the apps host. Leave all of it out when a proxy in front of the cluster terminates TLS. kustomization.yamlselects the Redis and registry components (the platform's own, or yours) and the IDE tiles. If you use a registry of your own, also removepatches/registry.yamland thereplacementsentry for the Ingressregistryfromkustomization.yaml, and setregistry=invelonde.envto your registry's host name. The workspace then needs DNS for two host names of its own.- In
velonde.env, setstorage_class_rwxorsingle_nodeif apps are to mount pipeline stores, andsession_poolif you use a node pool for user code.
Then check that the cluster has what the workspace needs:
The check reads the Kubernetes version, your rights, each node's kernel and container runtime,
the default StorageClass and the IngressClass from the cluster. In a namespace of its own,
which it deletes again, it starts a pod shaped like an app build (its own user namespace),
checks that a NetworkPolicy really blocks traffic, that pods reach the package repositories in
velonde.env and that the cluster's DNS resolves the workspace's host names. From the jump host it checks
DNS, docker with buildx and our distribution registry. Each line says ok, warn or FAIL
and what a failure means. install --cluster runs the same check first and stops on a failure,
unless you add --skip-check.
Then install:
The installer fetches the newest release (or the one you name with --version vX.Y.Z) and
your license, and applies the workspace. It builds the customer layer, pushes it to your registry and restarts the
deploy service, which waited for its image. Finally it runs the smoke test, which deploys the
sample app from inside the cluster and removes it again. At the end it prints the workspace's
address. If we maintain your overlay, leave --overlay out: install syncs the overlay for
your account and asks for the secrets it needs.
High availability¶
generate --ha writes an overlay that keeps the workspace running when a node fails or is drained:
- two ShinyProxy servers for the workspace's host name and two for the apps host, each pair on different nodes, with a disruption budget that keeps one of each running during a drain,
- Redis with Sentinel, three pods that keep the login sessions when one of them goes,
- with
--bucket, the registry on the object store with two replicas instead of one pod on a volume.
It needs at least two nodes, which the check tests. Some parts still run once. The deploy
service, the operator and the package repository each run as one pod. When their node fails,
Kubernetes starts them on another one within a few minutes. Until then, deploys wait and APIs
that check the caller's groups refuse calls, while API keys keep working. An IDE session is one
pod: when its node fails, the user starts a new session, and their home directory survives if
the default StorageClass replicates volumes across nodes (Longhorn, Rook or a cloud's block
storage). Before you drain a node, check which sessions run on it with
kubectl get pods -n <namespace> -o wide.
Upgrade and roll back¶
Run the same commands as on a single host, from the jump host:
bash install.sh upgrade latest # or a version
bash install.sh rollback
bash install.sh status
bash install.sh diagnose
Sessions and apps keep running during an upgrade. Routine tasks describes what each command does and when a release cannot be rolled back.
Change the configuration¶
Edit the overlay under /opt/velonde/overlay/<name>, then run bash install.sh upgrade <the
version you run> from the jump host. It applies the change and rebuilds the images when the
files they are built from changed. The overlay says what
each change costs, and Secrets says what to restart
after a change to secrets.env.
If we maintain the overlay for you, every upgrade replaces it with the version we publish. Any
edit you made on the jump host is kept as a copy under /opt/velonde/state/<name>/, and the
upgrade says so. Send us the change instead of editing the overlay, so that it reaches every
version.
Install without access to our registry¶
If the jump host cannot reach images.hub.linnerprogramming.com, write to us. We send the
release and the platform images as files for each version and agree the steps with you.