Operating a workspace¶
This page is for the person who runs a workspace, on their own host or on their cluster, without
having built it. install.sh is the installer you downloaded during
installation. On a cluster, you run it on the jump host, and /opt/velonde is
on the jump host too. <ns> stands for the workspace's namespace, as set in the overlay's
namespace.yaml, and velonde.<your domain> for the workspace's host name.
Where things are¶
| What | Where |
|---|---|
| Configuration, host names, authentication, package stack | /opt/velonde/overlay/<name>/, the overlay |
| Secrets | Only in secrets.env in the overlay directory |
| The license | velonde.license in the overlay directory, and its state in install.sh status |
| Releases, the one that runs and the ones that ran before | /opt/velonde/releases/ and install.sh status |
| User home directories | One volume per user, home-<user id>-<hash>, in <ns> |
| Deployed apps | HubApp objects in <ns>, with their images in the workspace's registry |
| Sessions | https://velonde.<your domain>/admin, the workspace's admin page, which lists every session with a stop button |
| Dashboards | The Grafana tile, or https://velonde.<your domain>/grafana/, for admins only, when the workspace has the monitoring stack: usage, app logs and resources, deploys and pipeline runs |
| Alerts | Mail to the addresses in alerts_to in velonde.env, when the workspace has the monitoring stack |
| Backups | The workspace's object store, with backups=true in velonde.env |
Admins are the members of the group that the auth patch names in admin-groups. On a workspace with
the built-in users, the admin is the admin user.
Routine tasks¶
bash install.sh status # releases here, what each workspace runs
bash install.sh license # a renewed license
bash install.sh upgrade latest # or a version
bash install.sh rollback
bash install.sh diagnose # one archive for us
On a single host and on a cluster the commands are the same. On a cluster, run them on the
jump host. Where a host runs several workspaces, name the workspace with --name <name> in upgrade,
rollback, license, diagnose and remove.
Status. status lists the releases downloaded to the host and, for each workspace, the release
it runs, the one it ran before, its address, and the state of its license with the last valid
day.
Renew the license. license fetches the newest license we published for your account and
installs it in the workspace. With a file we sent you, run license <file>. The workspace follows within
a minute and needs no upgrade. The timer velonde-license-<overlay> runs the same command once a
day, so you need this only to install a renewal at once. The license describes what happens
when a license runs out.
Upgrade. Read the release notes of the version first. upgrade works
through these steps:
- It fetches the release. The previous release stays under
/opt/velonde/releases. - If we maintain the overlay for you, it replaces the overlay with the version we publish for
that release. Your
secrets.envandcerts/stay. An edit you made on the host is kept as a copy under/opt/velonde/state/<name>/, and the upgrade says so. Send us such a change, so that it reaches every version. - On a single host, it renders Caddy's configuration again from the host names and the certificate settings.
- Where backups are on, it backs up the workspace's namespace.
- It builds the customer layer of the images for the new version if the workspace's registry does not have it yet, or if the files the images are built from changed since the last build on this host. Images that are in the registry before the first build on this host count as built from the overlay as it is. After a rebuild it restarts the deploy service.
- It applies the release and runs the smoke test, which deploys a sample app from inside the cluster and removes it again.
Sessions and apps keep running. New sessions start on the new images, and apps are rebuilt on
their next deploy. The apps server restarts (about a
minute). If the smoke test fails, the release is applied nonetheless: run diagnose and send
us the archive.
Roll back. rollback applies the release the workspace ran before, with the overlay as it is
now, and builds the customer layer for it where needed. An overlay that we manage for you
goes back to the version that release ran with. If an upgrade stopped before it was applied,
rollback applies the release that still runs, also when the upgrade was to the version
that runs (a changed configuration). On a single host, it first renders
Caddy's configuration as that release does. A second rollback returns to the
release you rolled back from. The commands rollback, remove, fetch and license run with the
installer of the newest release on the host, so the install.sh you downloaded once does not
have to be current. If a release has a bullet starting No rollback in its
release notes, it cannot go back to its predecessor. rollback refuses whenever the way back
crosses such a release, and so does upgrade with an older version. You then
restore from the backup instead.
Old releases. After every upgrade and rollback, the installer removes the releases under
/opt/velonde/releases that no workspace on the host runs or would roll back to, and keeps the newest
one. Once a day, the deploy service deletes from the workspace's registry the platform images of
releases older than the one a rollback returns to. The images of the release that runs, of
newer releases and of the one before it stay, and so do your deployed apps. The weekly
registry cleanup then frees the space. We keep the six newest releases available for download,
so install --version and upgrade can reach any of them.
Change the configuration. Edit the overlay, then run upgrade with the version you already
run. The overlay page says what each change costs. A setting
needs only an apply. The package stack or a certificate needs an image build that takes
minutes, and the upgrade builds it. A change to secrets.env needs one more step, described
under Secrets.
Report a problem you cannot place. diagnose writes one archive,
velonde-diagnose-<name>-<time>.tar.gz, into the current directory. It holds the host's details,
the overlay without secrets and certificates, the cluster's objects, the platform's logs, and
checks of DNS, HTTPS and the certificates for the host names. It contains host names, user ids
and log lines, but no secrets. Send it to us with a description of what you saw.
Manage users¶
- Add or remove users, change their groups. You do all of this in your identity provider. A user exists in the workspace from their first login, and their groups decide what they see.
- Grow a full home volume. New home volumes get the size
home_sizefromvelonde.env. On a cluster whose storage class allows volume expansion, you grow an existing one by patching its volume claim: On a single host, volumes have no size limit of their own. They all share the host's disk, and the alertVelondeNodeDiskFillingtells you when less than a fifth of it is free. To see which volume grew, runsudo du -xsh /var/lib/rancher/k3s/storage/* | sort -hon the host; each directory is named after its volume claim. - Delete a user's data. Stop their sessions, then delete their volume claim. Their apps
remain, and you can give each of them to someone else with
velonde owner <app> <user id>.
Sessions and apps¶
- A session hangs or does not start. The admin page lists every session. Stop the session
there, and the user can start a new one. When the cause is in the cluster,
kubectl describe podin<ns>shows it. The event "exceeded quota" means the workspace's quota (patches/quota.yaml) is used up: raise the quota or stop idle sessions. - A deploy failed. The user sees the build log, and so does the Velonde deploys dashboard, per app, when the workspace has the monitoring stack. The deploy page lists the common messages and their fixes.
- A deploy was interrupted. The build continues and is recorded when it ends.
velonde historyshows it about a minute later. - Roll back, delete or restrict an app. The owner, a maintainer or an admin runs
velonde rollbackfrom any IDE terminal, and the owner or an admin runsvelonde delete. To restrict an app, they setaccess-groupsin the app'svelonde.yamland deploy it again. - Give every app the same variables. Put them into
apps-env/in the overlay, as Variables for every app describes. They reach every app, API, MCP server and pipeline run as environment variables, as.Renvirondoes for one app. An app's own variable of the same name wins. New app sessions and the next pipeline run see a change. An API or MCP server sees it afterkubectl rollout restart deployment/api-<name> -n <ns>. - A scheduled pipeline run did not happen.
kubectl get cronjob -n <ns> pipe-<name>shows the schedule and the last run. A run that starts while another run is in progress does nothing, ends withskipped: another run of this pipeline is in progressin its log, andvelonde runslists it asskipped. On a workspace with the monitoring stack, a failed run, scheduled or manual, sends an alert mail to the addresses inalerts_to. A skipped run sends none.velonde logs <name>shows the reason.
Monitoring and alerts¶
The monitoring stack is optional. The installer's --monitoring option sets it up on a new
workspace. To add it to a workspace that runs:
- Copy
monitoring/andpatches/monitoring.yamlfrom the release's example overlay,/opt/velonde/releases/<the version you run>/platform/overlays/example/, into your overlay. - In
monitoring/grafana.yamlandmonitoring/alertmanager-url.yaml, replacevelonde.example.invalidwith the workspace's host name. - In
kustomization.yaml, uncomment the entrypatches/monitoring.yamland the component../../base/grafana, the Grafana tile. - Set the mail keys
smtp_*andalerts_toinvelonde.env(see the reference) andsmtp-passwordinsecrets.env. - Run an upgrade to the version you run.
With the stack, Grafana at https://velonde.<your domain>/grafana/ is open to admins, and the
landing page shows them the tile Grafana under On this workspace, which opens it. Other users
see neither. Grafana has
ShinyProxy's usage dashboard, the logs and resources of every app, and the Velonde deploys
dashboard with every build and pipeline run. Alerts go by mail to the addresses in
alerts_to, and the Alertmanager dashboard shows the ones currently firing. Every alert mail
says what to check first. These are the alerts:
| Alert | Meaning |
|---|---|
VelondeShinyProxyDown |
The workspace's ShinyProxy server is down, so nobody can log in. |
VelondeAppsInstanceDown |
The apps server is down, so deployed apps are unreachable. |
VelondeVolumeMountFailing |
A volume cannot be mounted, so the sessions or apps that need it do not start. The alert names the PersistentVolume, and kubectl get pv <name> shows its claim. For a shared volume, the file server cannot be reached or the password is wrong. |
VelondeComponentDown |
The registry, the deploy service, the auth service or the operator has no replica. Without the registry no session or app can start, without the deploy service no deploys run, without the auth service APIs and MCP servers refuse every call that carries no API key, and without the operator no configuration change is applied. |
VelondeRedisDown |
Sessions and app state are unavailable. |
VelondeVolumeAlmostFull |
A home, a store or the registry is more than 90 % full. |
VelondePipelineRunFailed, VelondeGitBuildFailed |
A pipeline run, or a build from a bound repository, failed. The owner sees why with velonde logs or velonde git status. The alert ends when a later run of the same pipeline or app succeeds, or a day after the failure. |
VelondeHousekeepingJobFailed |
The weekly registry cleanup, a token refresh, or the job that mounts idle homes for the backup failed. The alert ends with the job's next successful run, or a day after the failure. |
VelondeBackupFailed, VelondeBackupStale |
The last backup failed, or no backup has succeeded for more than a day. |
VelondeBackupServerDown, VelondeBackupNeverSucceeded |
Velero is not running, or a backup schedule has tried and never completed a backup. |
VelondePackagesDown |
A part of the package repository has no replica, so installs from it and package deploys fail. |
VelondeQuotaAlmostFull |
More than 90 % of the workspace's quota is used. Once it is full, new sessions, apps and home volumes are refused. |
VelondeNodeDiskFilling |
The disk of a node has less than 20 % free. On a single host every volume, the registry and the cluster live there. |
VelondeLicenseExpiring |
The license expires within 30 days. The mail names the date. |
VelondeLicenseExpired |
The license has expired, or the workspace has none that is valid, so new sessions and deploys are blocked, and after 30 days the open sessions end. Deployed apps, APIs and pipelines keep running. |
Know when the monitoring itself stops¶
If Prometheus, Alertmanager or the mail account stops working, no alert can tell you. A heartbeat service closes that gap. Alertmanager sends it a request every five minutes, and the service mails you, through its own mail, when the requests stop. To set it up with healthchecks.io (other services work the same way):
- Create a check with a simple schedule, a period of 5 minutes and a grace time of 10 minutes. Allow only POST requests, and enable the email integration for your address.
- Add the check's ping URL to
secrets.envasheartbeat-url. Anyone with the URL can report that all is well, so treat it as a secret. - In
monitoring/alertmanager.yaml, uncomment the route and the receiver namedheartbeat. - Run an upgrade to the version you run.
Within five minutes the check shows its first request.
For a quick health check from a shell, check that all pods in <ns> are running, including the
two ShinyProxy pods (sp-workspace-… and sp-apps-…), and that
curl -sI https://velonde.<your domain>/login answers 302.
The audit trail¶
The workspace records every action that changes or reveals something, with the person who did it, the time and the outcome. Refused attempts are recorded too. The trail covers:
| Action | What it records |
|---|---|
deploy, deploy.recorded |
A deploy that was started or refused, and the deploy the workspace then recorded, with what it changed of the audience and of the lists of who may open, call, maintain or mount the app. A push to a bound repository is recorded with the user git. |
delete, rollback, run |
An app deleted, rolled back, or a pipeline or report run started by hand. |
env.list, env.change |
The names of an app's secret variables listed, and which variables were set or removed. Values are never recorded. |
apikey.read, apikey.rotate |
An API's key shown or replaced. |
owner.change |
An app handed over, with the new owner. |
git.bind, git.unbind |
An app bound to a repository or released, with the repository and branch. |
ide-packages.change |
System packages added to or removed from the IDE images. |
packages.token.create, packages.token.revoke |
A personal token for the package repository made or revoked. |
api.call |
A call of an API or MCP server that the workspace refused: a token it did not accept, or a person who is not admitted. Calls the workspace admits are not recorded. |
audit.read |
Someone read the trail, with the filters they used. |
Logins, opening an app and admitted API calls are not part of the trail. Neither is what an
administrator does with kubectl or the installer; the cluster's own audit log records that.
The trail needs the monitoring stack, which keeps it for one year,
while other logs are kept for 90 days. Without the monitoring stack, velonde audit shows only
what the workspace recorded since its services last restarted.
To read the trail, run velonde audit in a session, or open the dashboard
"Velonde audit" in Grafana. Both are for workspace admins only.
To remove a person from the trail within the year, for example after a request to erase their data, contact us.
Back up and restore¶
Backups are optional. With backups=true in velonde.env, the workspace backs up its namespace every
day at 02:00 into the S3 bucket configured by object_store_* in velonde.env, with the keys from
secrets.env. The installer's --bucket option sets all of it on a new workspace. On a workspace that
runs, set the keys and run an upgrade to the version you run. A backup contains the Kubernetes
objects and the volume data, and is kept for 30 days. Every upgrade also takes a backup. Volume
data is copied only from volumes that a running pod mounts, so the workspace mounts idle homes during
the backup window to include them. The commands below need the Velero command-line tool, in the
version the workspace ships, on the host or the jump host:
Restore a whole workspace. Recreate the namespace, restore the last backup into it, then apply the release again and run the smoke test:
Restore one user's home to look at it without touching the workspace. Velero runs one restore at a time, so wait for each command to finish.
- Restore the Secrets, ConfigMaps and ServiceAccounts into a scratch namespace:
- Restore that user's pods, volume claims and volumes. A volume is restored only together
with the pod that mounted it, and without the volumes the claim stays pending:
The pod comes up with the files. Look at them with
velero restore create --from-backup <backup> --namespace-mappings <ns>:restore-test \ --include-resources pods,persistentvolumeclaims,persistentvolumes \ --selector hub.linnerprogramming.com/user=<id> --waitkubectl exec -n restore-test. - Delete the scratch namespace when you are done:
kubectl delete namespace restore-test.
Disk, reboots and removing a workspace¶
- Disk. On a single host, check the disk with
df -h /, and the volumes withkubectl get pvc -A. The registry is cleaned every Sunday at 03:00, keeping the ten builds in each app's history. - Reboot. After a reboot everything comes back on its own. Sessions restart empty, and files persist.
- Remove a workspace.
bash install.sh removeasks you to type the namespace. It then deletes the namespace with every session, app, home volume and secret, the workspace's backup schedule, backup location and object-store keys in theveleronamespace, on a single host the Caddy configuration for the workspace's names, and the overlay and state under/opt/velonde, with the copies of the overlay and its secrets in each downloaded release. Backups in the object store, the releases themselves and the monitoring stack stay. Do not delete the workspace withkubectl delete -k: that removes the platform's resource definitions from the cluster, and every workspace on it with them.