Skip to content

Operating a workspace

This page is for the person who runs a workspace, on their own host or on their cluster, without having built it. install.sh is the installer you downloaded during installation. On a cluster, you run it on the jump host, and /opt/velonde is on the jump host too. <ns> stands for the workspace's namespace, as set in the overlay's namespace.yaml, and velonde.<your domain> for the workspace's host name.

Where things are

What Where
Configuration, host names, authentication, package stack /opt/velonde/overlay/<name>/, the overlay
Secrets Only in secrets.env in the overlay directory
The license velonde.license in the overlay directory, and its state in install.sh status
Releases, the one that runs and the ones that ran before /opt/velonde/releases/ and install.sh status
User home directories One volume per user, home-<user id>-<hash>, in <ns>
Deployed apps HubApp objects in <ns>, with their images in the workspace's registry
Sessions https://velonde.<your domain>/admin, the workspace's admin page, which lists every session with a stop button
Dashboards The Grafana tile, or https://velonde.<your domain>/grafana/, for admins only, when the workspace has the monitoring stack: usage, app logs and resources, deploys and pipeline runs
Alerts Mail to the addresses in alerts_to in velonde.env, when the workspace has the monitoring stack
Backups The workspace's object store, with backups=true in velonde.env

Admins are the members of the group that the auth patch names in admin-groups. On a workspace with the built-in users, the admin is the admin user.

Routine tasks

bash install.sh status              # releases here, what each workspace runs
bash install.sh license             # a renewed license
bash install.sh upgrade latest      # or a version
bash install.sh rollback
bash install.sh diagnose            # one archive for us

On a single host and on a cluster the commands are the same. On a cluster, run them on the jump host. Where a host runs several workspaces, name the workspace with --name <name> in upgrade, rollback, license, diagnose and remove.

Status. status lists the releases downloaded to the host and, for each workspace, the release it runs, the one it ran before, its address, and the state of its license with the last valid day.

Renew the license. license fetches the newest license we published for your account and installs it in the workspace. With a file we sent you, run license <file>. The workspace follows within a minute and needs no upgrade. The timer velonde-license-<overlay> runs the same command once a day, so you need this only to install a renewal at once. The license describes what happens when a license runs out.

Upgrade. Read the release notes of the version first. upgrade works through these steps:

  1. It fetches the release. The previous release stays under /opt/velonde/releases.
  2. If we maintain the overlay for you, it replaces the overlay with the version we publish for that release. Your secrets.env and certs/ stay. An edit you made on the host is kept as a copy under /opt/velonde/state/<name>/, and the upgrade says so. Send us such a change, so that it reaches every version.
  3. On a single host, it renders Caddy's configuration again from the host names and the certificate settings.
  4. Where backups are on, it backs up the workspace's namespace.
  5. It builds the customer layer of the images for the new version if the workspace's registry does not have it yet, or if the files the images are built from changed since the last build on this host. Images that are in the registry before the first build on this host count as built from the overlay as it is. After a rebuild it restarts the deploy service.
  6. It applies the release and runs the smoke test, which deploys a sample app from inside the cluster and removes it again.

Sessions and apps keep running. New sessions start on the new images, and apps are rebuilt on their next deploy. The apps server restarts (about a minute). If the smoke test fails, the release is applied nonetheless: run diagnose and send us the archive.

Roll back. rollback applies the release the workspace ran before, with the overlay as it is now, and builds the customer layer for it where needed. An overlay that we manage for you goes back to the version that release ran with. If an upgrade stopped before it was applied, rollback applies the release that still runs, also when the upgrade was to the version that runs (a changed configuration). On a single host, it first renders Caddy's configuration as that release does. A second rollback returns to the release you rolled back from. The commands rollback, remove, fetch and license run with the installer of the newest release on the host, so the install.sh you downloaded once does not have to be current. If a release has a bullet starting No rollback in its release notes, it cannot go back to its predecessor. rollback refuses whenever the way back crosses such a release, and so does upgrade with an older version. You then restore from the backup instead.

Old releases. After every upgrade and rollback, the installer removes the releases under /opt/velonde/releases that no workspace on the host runs or would roll back to, and keeps the newest one. Once a day, the deploy service deletes from the workspace's registry the platform images of releases older than the one a rollback returns to. The images of the release that runs, of newer releases and of the one before it stay, and so do your deployed apps. The weekly registry cleanup then frees the space. We keep the six newest releases available for download, so install --version and upgrade can reach any of them.

Change the configuration. Edit the overlay, then run upgrade with the version you already run. The overlay page says what each change costs. A setting needs only an apply. The package stack or a certificate needs an image build that takes minutes, and the upgrade builds it. A change to secrets.env needs one more step, described under Secrets.

Report a problem you cannot place. diagnose writes one archive, velonde-diagnose-<name>-<time>.tar.gz, into the current directory. It holds the host's details, the overlay without secrets and certificates, the cluster's objects, the platform's logs, and checks of DNS, HTTPS and the certificates for the host names. It contains host names, user ids and log lines, but no secrets. Send it to us with a description of what you saw.

Manage users

  • Add or remove users, change their groups. You do all of this in your identity provider. A user exists in the workspace from their first login, and their groups decide what they see.
  • Grow a full home volume. New home volumes get the size home_size from velonde.env. On a cluster whose storage class allows volume expansion, you grow an existing one by patching its volume claim:
    kubectl patch pvc home-<id> -n <ns> -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}'
    
    On a single host, volumes have no size limit of their own. They all share the host's disk, and the alert VelondeNodeDiskFilling tells you when less than a fifth of it is free. To see which volume grew, run sudo du -xsh /var/lib/rancher/k3s/storage/* | sort -h on the host; each directory is named after its volume claim.
  • Delete a user's data. Stop their sessions, then delete their volume claim. Their apps remain, and you can give each of them to someone else with velonde owner <app> <user id>.

Sessions and apps

  • A session hangs or does not start. The admin page lists every session. Stop the session there, and the user can start a new one. When the cause is in the cluster, kubectl describe pod in <ns> shows it. The event "exceeded quota" means the workspace's quota (patches/quota.yaml) is used up: raise the quota or stop idle sessions.
  • A deploy failed. The user sees the build log, and so does the Velonde deploys dashboard, per app, when the workspace has the monitoring stack. The deploy page lists the common messages and their fixes.
  • A deploy was interrupted. The build continues and is recorded when it ends. velonde history shows it about a minute later.
  • Roll back, delete or restrict an app. The owner, a maintainer or an admin runs velonde rollback from any IDE terminal, and the owner or an admin runs velonde delete. To restrict an app, they set access-groups in the app's velonde.yaml and deploy it again.
  • Give every app the same variables. Put them into apps-env/ in the overlay, as Variables for every app describes. They reach every app, API, MCP server and pipeline run as environment variables, as .Renviron does for one app. An app's own variable of the same name wins. New app sessions and the next pipeline run see a change. An API or MCP server sees it after kubectl rollout restart deployment/api-<name> -n <ns>.
  • A scheduled pipeline run did not happen. kubectl get cronjob -n <ns> pipe-<name> shows the schedule and the last run. A run that starts while another run is in progress does nothing, ends with skipped: another run of this pipeline is in progress in its log, and velonde runs lists it as skipped. On a workspace with the monitoring stack, a failed run, scheduled or manual, sends an alert mail to the addresses in alerts_to. A skipped run sends none. velonde logs <name> shows the reason.

Monitoring and alerts

The monitoring stack is optional. The installer's --monitoring option sets it up on a new workspace. To add it to a workspace that runs:

  1. Copy monitoring/ and patches/monitoring.yaml from the release's example overlay, /opt/velonde/releases/<the version you run>/platform/overlays/example/, into your overlay.
  2. In monitoring/grafana.yaml and monitoring/alertmanager-url.yaml, replace velonde.example.invalid with the workspace's host name.
  3. In kustomization.yaml, uncomment the entry patches/monitoring.yaml and the component ../../base/grafana, the Grafana tile.
  4. Set the mail keys smtp_* and alerts_to in velonde.env (see the reference) and smtp-password in secrets.env.
  5. Run an upgrade to the version you run.

With the stack, Grafana at https://velonde.<your domain>/grafana/ is open to admins, and the landing page shows them the tile Grafana under On this workspace, which opens it. Other users see neither. Grafana has ShinyProxy's usage dashboard, the logs and resources of every app, and the Velonde deploys dashboard with every build and pipeline run. Alerts go by mail to the addresses in alerts_to, and the Alertmanager dashboard shows the ones currently firing. Every alert mail says what to check first. These are the alerts:

Alert Meaning
VelondeShinyProxyDown The workspace's ShinyProxy server is down, so nobody can log in.
VelondeAppsInstanceDown The apps server is down, so deployed apps are unreachable.
VelondeVolumeMountFailing A volume cannot be mounted, so the sessions or apps that need it do not start. The alert names the PersistentVolume, and kubectl get pv <name> shows its claim. For a shared volume, the file server cannot be reached or the password is wrong.
VelondeComponentDown The registry, the deploy service, the auth service or the operator has no replica. Without the registry no session or app can start, without the deploy service no deploys run, without the auth service APIs and MCP servers refuse every call that carries no API key, and without the operator no configuration change is applied.
VelondeRedisDown Sessions and app state are unavailable.
VelondeVolumeAlmostFull A home, a store or the registry is more than 90 % full.
VelondePipelineRunFailed, VelondeGitBuildFailed A pipeline run, or a build from a bound repository, failed. The owner sees why with velonde logs or velonde git status. The alert ends when a later run of the same pipeline or app succeeds, or a day after the failure.
VelondeHousekeepingJobFailed The weekly registry cleanup, a token refresh, or the job that mounts idle homes for the backup failed. The alert ends with the job's next successful run, or a day after the failure.
VelondeBackupFailed, VelondeBackupStale The last backup failed, or no backup has succeeded for more than a day.
VelondeBackupServerDown, VelondeBackupNeverSucceeded Velero is not running, or a backup schedule has tried and never completed a backup.
VelondePackagesDown A part of the package repository has no replica, so installs from it and package deploys fail.
VelondeQuotaAlmostFull More than 90 % of the workspace's quota is used. Once it is full, new sessions, apps and home volumes are refused.
VelondeNodeDiskFilling The disk of a node has less than 20 % free. On a single host every volume, the registry and the cluster live there.
VelondeLicenseExpiring The license expires within 30 days. The mail names the date.
VelondeLicenseExpired The license has expired, or the workspace has none that is valid, so new sessions and deploys are blocked, and after 30 days the open sessions end. Deployed apps, APIs and pipelines keep running.

Know when the monitoring itself stops

If Prometheus, Alertmanager or the mail account stops working, no alert can tell you. A heartbeat service closes that gap. Alertmanager sends it a request every five minutes, and the service mails you, through its own mail, when the requests stop. To set it up with healthchecks.io (other services work the same way):

  1. Create a check with a simple schedule, a period of 5 minutes and a grace time of 10 minutes. Allow only POST requests, and enable the email integration for your address.
  2. Add the check's ping URL to secrets.env as heartbeat-url. Anyone with the URL can report that all is well, so treat it as a secret.
  3. In monitoring/alertmanager.yaml, uncomment the route and the receiver named heartbeat.
  4. Run an upgrade to the version you run.

Within five minutes the check shows its first request.

For a quick health check from a shell, check that all pods in <ns> are running, including the two ShinyProxy pods (sp-workspace-… and sp-apps-…), and that curl -sI https://velonde.<your domain>/login answers 302.

The audit trail

The workspace records every action that changes or reveals something, with the person who did it, the time and the outcome. Refused attempts are recorded too. The trail covers:

Action What it records
deploy, deploy.recorded A deploy that was started or refused, and the deploy the workspace then recorded, with what it changed of the audience and of the lists of who may open, call, maintain or mount the app. A push to a bound repository is recorded with the user git.
delete, rollback, run An app deleted, rolled back, or a pipeline or report run started by hand.
env.list, env.change The names of an app's secret variables listed, and which variables were set or removed. Values are never recorded.
apikey.read, apikey.rotate An API's key shown or replaced.
owner.change An app handed over, with the new owner.
git.bind, git.unbind An app bound to a repository or released, with the repository and branch.
ide-packages.change System packages added to or removed from the IDE images.
packages.token.create, packages.token.revoke A personal token for the package repository made or revoked.
api.call A call of an API or MCP server that the workspace refused: a token it did not accept, or a person who is not admitted. Calls the workspace admits are not recorded.
audit.read Someone read the trail, with the filters they used.

Logins, opening an app and admitted API calls are not part of the trail. Neither is what an administrator does with kubectl or the installer; the cluster's own audit log records that.

The trail needs the monitoring stack, which keeps it for one year, while other logs are kept for 90 days. Without the monitoring stack, velonde audit shows only what the workspace recorded since its services last restarted.

To read the trail, run velonde audit in a session, or open the dashboard "Velonde audit" in Grafana. Both are for workspace admins only.

To remove a person from the trail within the year, for example after a request to erase their data, contact us.

Back up and restore

Backups are optional. With backups=true in velonde.env, the workspace backs up its namespace every day at 02:00 into the S3 bucket configured by object_store_* in velonde.env, with the keys from secrets.env. The installer's --bucket option sets all of it on a new workspace. On a workspace that runs, set the keys and run an upgrade to the version you run. A backup contains the Kubernetes objects and the volume data, and is kept for 30 days. Every upgrade also takes a backup. Volume data is copied only from volumes that a running pod mounts, so the workspace mounts idle homes during the backup window to include them. The commands below need the Velero command-line tool, in the version the workspace ships, on the host or the jump host:

velero backup get
velero backup create now --include-namespaces <ns> --storage-location <ns> --wait

Restore a whole workspace. Recreate the namespace, restore the last backup into it, then apply the release again and run the smoke test:

velero restore create --from-backup <backup> --wait
bash install.sh upgrade <the version you run>

Restore one user's home to look at it without touching the workspace. Velero runs one restore at a time, so wait for each command to finish.

  1. Restore the Secrets, ConfigMaps and ServiceAccounts into a scratch namespace:
    velero restore create --from-backup <backup> --namespace-mappings <ns>:restore-test \
      --include-resources secrets,configmaps,serviceaccounts --wait
    
  2. Restore that user's pods, volume claims and volumes. A volume is restored only together with the pod that mounted it, and without the volumes the claim stays pending:
    velero restore create --from-backup <backup> --namespace-mappings <ns>:restore-test \
      --include-resources pods,persistentvolumeclaims,persistentvolumes \
      --selector hub.linnerprogramming.com/user=<id> --wait
    
    The pod comes up with the files. Look at them with kubectl exec -n restore-test.
  3. Delete the scratch namespace when you are done: kubectl delete namespace restore-test.

Disk, reboots and removing a workspace

  • Disk. On a single host, check the disk with df -h /, and the volumes with kubectl get pvc -A. The registry is cleaned every Sunday at 03:00, keeping the ten builds in each app's history.
  • Reboot. After a reboot everything comes back on its own. Sessions restart empty, and files persist.
  • Remove a workspace. bash install.sh remove asks you to type the namespace. It then deletes the namespace with every session, app, home volume and secret, the workspace's backup schedule, backup location and object-store keys in the velero namespace, on a single host the Caddy configuration for the workspace's names, and the overlay and state under /opt/velonde, with the copies of the overlay and its secrets in each downloaded release. Backups in the object store, the releases themselves and the monitoring stack stay. Do not delete the workspace with kubectl delete -k: that removes the platform's resource definitions from the cluster, and every workspace on it with them.