Kiosk
Last updated
Last updated
https://github.com/loft-sh/kiosk
Accounts & Account Users to separate tenants in a shared Kubernetes cluster
Self-Service Namespace Provisioning for account users
Account Limits to ensure quality of service and fairness when sharing a cluster
Namespace Templates for secure tenant isolation and self-service namespace initialization
Multi-Cluster Tenant Management for sharing a pool of clusters (coming soon)
Kubernetes is designed as a single-tenant platform, which makes it hard for cluster admins to host multiple tenants in a single Kubernetes cluster. However, sharing a cluster has many advantages, e.g. more efficient resource utilization, less admin/configuration effort or easier sharing of cluster-internal resources among different tenants.
While there are hundreds of ways of setting up multi-tenant Kubernetes clusters and many Kubernetes distributions provide their own tenancy logic, there is no lightweight, pluggable and customizable solution that allows admins to easily add multi-tenancy capabilities to any standard Kubernetes cluster.
kiosk is designed to be:
100% Open-Source: CNCF compatible Apache 2.0 license
Pluggable: easy to install into any existing cluster and suitable for different use cases
Fast: emphasizing automation and self-service for tenants
Secure: offering default configurations for different levels of tenant isolation
Extensible: providing building blocks for higher-level Kubernetes platforms
The core idea of kiosk is to use Kubernetes namespaces as isolated workspaces where tenant applications can run isolated from each other. To minimize admin overhead, cluster admins are supposed to configure kiosk which then becomes a self-service system for provisioning Kubernetes namespaces for tenants.
The following diagram shows the main actors (Cluster Admins and Account Users) as well as the most relevant Kubernetes resources and their relationships.
Click on the following links to view the description for each of the actors and kiosk components:
Details
Cluster Admin
Cluster Admins have the permission to perform CRUD operations for cluster-wide / non-namespaced resources (especially RBAC related resources as well as the custom resources Account, AccountQuota, AccountQuotaSet, and Template). Cluster Admins configure kiosk by creating and managing Accounts, AccountQuotas, AccountQuotaSets, and Templates. They can also see and configure all Spaces owned by all Accounts.
Details
Account
Every tenant is represented by an Account. Cluster Admins define and manage Accounts and assign Account Users (Users, Groups, ServiceAccounts) to Accounts - similar to assigning RBAC Roles to subjects as part of a RoleBinding configuration.
Details
Account User
Account Users perform actions within the Kubernetes cluster via API server requests while using a certain Account. Cluster Admins can assign the same Account User to multiple Accounts. Account Users have access to Spaces that belong to the Accounts they are using. If assigned the default kiosk ClusterRole, every Account User has the permission to list/get/create/delete Spaces for the respective Account, however, this can be changed via RBAC RoleBindings.
Details
Space
A Space is a non-persistent, virtual resource that represents exactly one Kubernetes namespace. Spaces have the following characteristics:
Every space can belong up to one Account which is the owner of this Space. Ownerless Spaces are possible.
If a user has rights to access the underlying Namespace, the user can access the Space in the same way. Hence besides Account Users, other actors (User, Group, ServiceAccount) can also access the Space if someone grants this access via additional Kubernetes RBAC.
Every User only sees the Spaces the User has access to. This is in contrast to regular namespaces, where Users can only list all namespaces or none
Space ownership can be changed, by changing the ownership annotation on the namespace
During Space creation (or Space ownership changes) a RoleBinding for the owning Account is created in the corresponding Space namespace. The referenced RBAC ClusterRole can be configured in the account
A Space can be prepopulated during creation with a predefined set of resources by configuring default Templates in the Account. Kiosk will make sure that these resources will be correctly deployed before the user gets access to the namespace.
Details
Namespace
A Namespace is a regular Kubernetes Namespace that can be accessed by anyone who has the appropriate RBAC rules to do so. Namespaces are provisioned and managed by kiosk and have a 1-to-1 relationship to the resource Space which is a custom resource of kiosk. By default, Account Users have the permission to operate within all Namespaces that are represented by Spaces which belong to one of their Accounts.
Details
Template
Templates are defined and managed by Cluster Admins. Templates are used to initialize Spaces/Namespaces with a set of Kubernetes resources (defined as manifests or as part of a Helm chart). Templates can be created using a different ClusterRole than the Account User uses, so they can be used to create resources that are not allowed to be created by actors of the Space/Namespace, e.g. to set up certain isolation resources (e.g. Network Policies, Pod Security Policies etc.). Cluster Admins can define default Templates within the Account configuration which automatically applies these templates to each Space that is created using the respective Account. Additionally, Account Users can state other non-mandatory Templates that should also be applied when creating a Space.
Details
TemplateInstance
When a Template is applied to a Space, kiosk creates a TemplateInstance to keep track of which Templates have been applied to the Space. A TemplateInstance contains information about the Template as well as about the parameters used to instantiate it. Additionally, TemplateInstances can be configured to sync with Templates, i.e. the TemplateInstance will update the resources whenever the Template changes that has been used to create these resources.
Details
AccountQuota
AccountQuotas are defined and managed by Cluster Admins. AccountQuotas define cluster-wide aggregated limits for Accounts. The resources of all Spaces/Namespaces that belong to an Account count towards the aggregated limits defined in the AccountQuota. Similar to Namespaces which can be limited by multiple ResourceQuotas, an Account can be limited by multiple AccountQuotas. If the same limit (e.g. total CPU per Account) is defined by multiple AccountQuotas, the Account will be limited according to the lowest value.
When installing kiosk in a Kubernetes cluster, these components will be added to the cluster:
CRDs for Account, AccountQuota, AccountQuotaSet, Template, TemplateInstance
Controller for kiosk Custom Resources (runs inside the cluster)
API Server Extension (runs inside the cluster similar to the Controller)
kiosk adds two groups of resources to extend the Standard API Groups of Kubernetes:
Details Show List of Custom Resources
Custom Resources: config.kiosk.sh
Custom Resource Definitions (CRDs) for configuring kiosk. These resources are persisted in etcd just like any other Kubernetes resources and are managed by an operator which runs inside the cluster.
config.kiosk.sh/Account
config.kiosk.sh/AccountQuota
config.kiosk.sh/AccountQuotaSet (soon)
config.kiosk.sh/Template
config.kiosk.sh/TemplateInstance
Details Show List of API Extension Resources
API Extension: tenancy.kiosk.sh
Virtual resources which are accessible via an API Server Extension and will not be persisted in etcd. These resources are similar to views in a relational database. The benefit of providing these resources instead of only using CRDs is that we can calculate access permissions dynamically for every request. That means that it does not only allow to list, edit and manage Spaces (which map 1-to-1 to Namespaces), it also allows to show a different set of Spaces for different Account Users depending on the Accounts they are associated with or in other words: this circumvents the current limitation of Kubernetes to show filtered lists of cluster-scoped resources based on access rights.
tenancy.kiosk.sh/Account
tenancy.kiosk.sh/AccountQuota
tenancy.kiosk.sh/Space
tenancy.kiosk.sh/TemplateInstance
kubectl
: Follow this guide to install it.
helm
version 3: Follow this guide to install it.
kiosk supports Kubernetes version: v1.14 and higher. Use kubectl version
to determine the Server Version
of your cluster. While this getting started guide should work with most Kubernetes clusters out-of-the-box, there are certain things to consider for the following types of clusters:
Details
Docker Desktop Kubernetes
All ServiceAccounts have cluster-admin role by default, which means that emulating users with ServiceAccounts is not a good idea. Use impersonation instead.
Details
Digital Ocean Kubernetes (DOKS)
All users in DOKS have cluster-admin role by default which means that when using impersonation, every user will have admin access. To see kiosk-based multi-tenancy in action, create ServiceAccounts to emulate different users.
Details
Google Kubernetes Engine (GKE)
Your kube-context will by default not have cluster-admin role. Run the following command to get your google email address and to make your user cluster admin:
You need a kube-context with admin rights.
If running all the following commands returns yes
, you are most likely admin:
To verify the installation make sure the kiosk pod is running:
In the following steps, we will use Kubernetes user impersonation to allow you to quickly switch between cluster admin and simple account user roles. If you are cluster admin and you want to run a kubectl
command as a different user, you can impersonate this user by adding the kubectl
flags --as=[USER]
and/or --as-group=[GROUP]
.
In this getting started guide, we assume two user roles:
Cluster Admin: use your admin-context as current context (kubectl
commands without -as
flag)
Account User john
: use your admin-context to impersonate a user (kubectl
commands with -as=john
)
If you are using Digital Ocean Kubernetes (DOKS), follow this guide to simulate a user using a Service Account.
To allow a user to create and manage namespaces, they need a kiosk account. Run the following command to create such an account for our example user john
:
Details
View: account.yaml
As you can see in this example, every account defines subjects
which are able to use this account. In this example, there is only one subject which is a User
with name john
. However, Accounts can also have multiple subjects.
Subjects for kiosk Accounts are defined in the exact same way as subjects in RoleBindings. Subjects can be a combination of:
Users
Groups
ServiceAccounts (see example below: account-sa.yaml
)
Details
View: account-sa.yaml (alternative for ServiceAccounts, e.g. Digital Ocean Kubernetes)
If you want to assign an Account to a ServiceAccount (e.g. when using Digital Ocean Kubernetes / DOKS), please use the following alternative:
Learn more about User Management and Accounts in kiosk.
All Account Users are able to view their Account through their generated ClusterRole. Let's try this by impersonating john
:
Spaces are the virtual representation of namespaces. Each Space represents exactly one namespace. The reason why we use Spaces is that by introducing this virtual resource, we can allow users to only operate on a subset of namespaces they have access to and hide other namespaces they shouldn't see.
By default, Account Users cannot create Spaces themselves. They can only use the Spaces/Namespaces that belong to their Accounts. That means a cluster admin would need to create the Spaces for an Account and then the Account Users could work with these Spaces/Namespaces.
To allow all Account Users to create Spaces for their own Accounts, create the following RBAC ClusterRoleBinding:
Details
View: rbac-creator.yaml
Of course, you can also adjust this ClusterRoleBinding in a way that only certain subjects/users can create Spaces for their Accounts. Just modify the
subjects
section.
After granting Account Users the right to create Spaces for their Accounts (see ClusterRoleBinding in 3.1.), all Account Users are able to create Spaces. Let's try this by impersonating john
:
Details
View: space.yaml
As you can see in this example, every Space belongs to exactly one Account which is referenced by
spec.account
.
Let's take a look at the Spaces of the Accounts that User john
owns by impersonating this user:
Every Space is the virtual representation of a regular Kubernetes Namespace. That means we can use the associated Namespace of our Spaces just like any other Namespace.
Let's impersonate john
again and create an nginx deployment inside johns-space
:
That's great, right? A user that did not have any access to the Kubernetes cluster, is now able to create Namespaces on-demand and gets restricted access to these Namespaces automatically.
To allow Account Users to delete all Spaces/Namespace that they create, you need to set the spec.space.clusterRole
field in the Account to kiosk-space-admin
.
When creating a Space, kiosk creates the according Namespace for the Space and then creates a RoleBinding within this Namespace which binds the standard Kubernetes ClusterRole
admin
to every Account User (i.e. allsubjects
listed in the Account). While this ClusterRole allows full access to this Namespace, it does not allow to delete the Space/Namespace. (The verbdelete
is missing in the default admin clusterrole)
As john
can be User of multiple Accounts, let's create a second Account which allows john
to delete Spaces/Namespaces that belong to this Account:
Details
View: account-deletable-spaces.yaml
If you are using ServiceAccounts instead of impersonation, adjust the
subjects
section of this Account similar toaccount-sa.yaml
in 2.1.
Now, let's create a Space for this Account:
Details
View: space-deletable.yaml
If a Space belongs to an Account that allows Account Users to delete such Spaces, an Account User can simply delete the Space using kubectl:
Deleting a Space also deletes the underlying Namespace.
kiosk provides the spec.space.spaceTemplate
option for Accounts which lets admins define defaults for new Spaces of an Account. The following example creates the Account account-default-space-metadata
which defines default labels and annotations for all Spaces created with this Account:
Details
View: account-default-space-metadata.yaml
With kiosk, you have two options to limit Accounts:
Limit Number of Spaces for an Account
Set Account Quotas = maximum values for resources aggregated across all Spaces of an Account (e.g. total number of Pods combined in all Spaces)
By setting the spec.space.limit
in an Account, Cluster Admins can limit the number of Spaces that Account Users can create for a certain Account.
Let's run the following command to update the existing Account johns-account
and specify spec.space.limit: 2
:
Details
View: account-space-limit.yaml
Now, let's try to create more than 2 Spaces (note that you may have already created a Space for this Account during earlier steps of this getting started guide):
AccountQuotas allow you to define limits for an Account which are aggregated across all Spaces of this Account.
Let's create an AccountQuota for johns-account
which will set the aggregated number of Pods across all Spaces to 2 and the aggregated maximum of limits.cpu
across all Pods in all Spaces to 4 CPU Cores (see Kubernetes resource limits):
Details
View: accountquota.yaml
AccountQuotas allow you to restrict the same resources as Kubernetes ResourceQuotas but unlike ResourceQuotas, AccountQuotas are not restricted to a single Namespace. Instead, AccountQuotas add up all used resources across all Spaces of an Account to generate an aggregated value which is then compared to the max value defined in the AccountQuota.
If there are multiple AccountQuotas referencing the same Account via spec.account
, kiosk merges the Quotas. In case multiple AccountQuotas define different limits for the same resource type, kiosk uses the lowest value.
Templates in kiosk are used to initialize Namespaces and share common resources across namespaces (e.g. secrets). When creating a Space, kiosk will use these Templates to populate the newly created Namespace for this Space. Templates:
can either contain one or more Kubernetes manifests or alternatively a Helm chart
are being tracked by TemplateInstances in each Namespace they are applied to
can contain parameters such as ${NAMESPACE}
or ${MY_PARAMETER}
that can be specified within an TemplateInstance
The easiest option to define a Template is by specifying an array of Kubernetes manifests which should be applied when the Template is being instantiated.
The following command will create a Template called space-restrictions
which defined 2 manifests, a Network Policy which will make sure that the users of this Space/Namespace cannot create privileged containers and a LimitRange for default CPU limits of containers in this Namespace:
Details
View: template-manifests.yaml
Instead of manifests, a Template can specify a Helm chart that will be installed (using helm template
) when the Template is being instantiated. Let's create a Template called redis
which installs the stable/redis
Helm chart:
Details
View: template-helm.yaml
By default, only admins can list Templates. To allow users to view templates, you need to set up RBAC accordingly. Run the following code to allow every cluster user to list and view all Templates:
Details
View: rbac-template-viewer.yaml