Tech Recipe Book
My Services
  • Book
    • About the author
    • Architect
      • Algorithms
        • DB index algorithms
          • How does database indexing work
        • Neural network optimization
          • Neural Network Optimization
        • Route search
          • Road network in a database to build a route
          • Traveling Salesman Problem (TSP)
      • Architecture Frameworks
        • DODAF
        • TOGAF
        • Enterprise Architecture (EA) Tools Reviews 2023 | Gartner
      • Zero Trust
      • Billing
        • SHM billing system
      • Bots
        • Discord
        • Telegram
          • Chat GPT Telegram bot
          • Получаем статистику Telegram-канала при помощи api и python или свой tgstat с регистрацией и смс
          • Как хостить телеграм-бота (и другие скрипты на Python) на Repl.it бесплатно 24/7
          • Создание Telegram бота на PHP #1: основные понятия для работы с API
          • Создание Telegram бота на PHP #2: создание первого бота для Telegram
          • Создание Telegram бота на PHP #3: примеры отправки сообщений с кнопками в Telegram
          • Создание Telegram бота на PHP #4: отправка файлов и изображений в Telegram
          • Создание Telegram бота на PHP #5: работа с хуками
      • Business intelligence
      • Cloud Storage
        • Ceph
        • Virtual Distributed File System
      • Cryptography
        • Open Source PKI Software
        • OpenPGP
          • Email Encryption
          • Kleopatra
          • Miscellaneous Tools
          • Server side applications
      • Message broker
        • Kafka
          • Kafka UI-tools
          • Kafka streams ksqlDb
        • RabbitMQ
      • DB
        • MySQL
          • Auto sharding
          • MariaDB Zabbix monitoring
          • MySQL and MariaDB replication with Zabbix monitoring
        • Postgres
          • HA PostgreSQL with Patroni, Haproxy, Keepalived
          • Mass parallel requests - Greenplum
          • PostgreSQL cluster for development and testing
        • Vitess - Scalable. Reliable. MySQL-compatible. Cloud-native. Database.
      • Identity and Access Management (IDM)
        • FreeIPA - Identity, Policy, Audit
        • FreeIPA as an Enterprise solution
        • Keycloak
          • Keycloak HA cluster
        • Open Identity Platform
        • SSO
          • Keycloak for Java app
          • OpenAM
          • OpenIG
      • Firewall
        • nftables
      • Infrastructure As a Code
        • Ansible
        • IaC Packer Ansible Teraform
        • Installing Jenkins using terraform in Kubernetes in Yandex Cloud with letsencypt
        • Teraform Crosplan Pulumi
        • Yandex IaC solutions
      • Kubernetes
        • Installation
          • Install Kubernetes cluster
          • Deploying a Kubespray cluster to OpenStack using Terraform
          • Kube deploy in Yandex cloud
        • Frameworks
          • Deckhouse
            • LDAP authentification
            • On premise Install
            • Yandex Cloud Install
          • K3S
          • OpenShift OKD
          • RKE2
          • Rancher
            • Rancher Install
        • Auth
          • Keycloak in k8s
          • LDAP
        • GUI management Lens
        • Monitoring
          • Monitoring with Falco
          • Network monitoring
          • Nginx ingress
          • Prometheus Graphana for sample Nodejs app
          • Rsource monitoring Avito
        • Exposing services
          • Exposing Kubernetes Services
          • Cilium BGP
        • CNCF
        • Helm
          • Repositories
            • Artifact Hub | official
            • Bitnami | vmware
          • Awesome helm charts and resources
          • Essential Services for Modern Organizations
          • Security and Compliance
          • Additional charts
        • Isolation
          • vcluster - Virtual Kubernetes Clusters
          • Kiosk
          • KubeArmor
          • Control Plane Hardening
          • Hierarchical namespaces
        • Security Center
          • Minesweeper
          • NeuVector by SUSE
          • SOAR in Kubernetes
          • Security Сenter for Kubernetes
        • Terraform CI security
          • Terraform plan analysis with Checkov and Bridgecrew
          • Yandex Terraform scan
        • Vulnerability management
          • Aqua
          • Sysdig
          • Kyverno
          • GitLab
          • NeuVector by SUSE
        • Image scanning
          • Snyk
          • Sysdig
          • Harbor
          • Trivy
        • Signature verification
          • Sigstore
        • Control plane security
          • Gatekeeper
            • Applying OPA Gatekeeper
          • Kyverno
            • Policy as a code. Kyverno
        • Runtime Security
          • Osquery
          • Falco
          • ClamAV
        • Network security
          • Cilium
          • Control Plane Hardening (API restriction)
          • Network policy recipes
          • Service mesh
            • Istio HA, LoadBalance, Rate limit
          • mTLS Autocert
        • Honeypot
          • Building honeypot using vcluster and Falco
        • Backup
          • Kasten K10
        • Secrets
          • Vault CSI Driver
      • Load Balance
        • Nginx
        • HAProxy
          • Proxy methods
          • HAProxy for RDP
          • Payment gateway A/B test with HAProxy
          • HAPRoxy for Percona or Galera
      • Monitoring
        • Zabbix
          • Apache Zabbix
          • Disc Quota
          • Nginx Zabbix
          • SSL certificates Zabix
          • Zabbix notifications
        • Nagios
          • Datacenter monitoring
        • Prometheus and Grafana
      • Windows
        • Sysmon enhanced Windows audit
        • Sysmon to Block Unwanted File
      • Linux
        • Rsync
        • Debian based
          • Apt-Cacher NG
          • Unattended Upgrades in Debian / Ubuntu
        • RedHat basede
          • RPM Server
        • Logs analysis
        • Build armhf qemu
      • NGFW
      • CI/CD
        • DevSecOps
          • DAST
            • Burp
              • Dastardly
            • StackHawk
            • ZAP and GitHub Actions
          • SAST
            • Checkmarx
            • OSV by Google
            • Snyk
            • SonarQube
        • GitLab Runner in Yandex Cloud
        • Dynamic Gitlab Runners in Yandex Cloud
        • GitLab runner in Kubernetes with Werf
        • Kubernetes deploy strategies
        • Kubernetes highload deploy. part 1
        • Kubernetes highload deploy. part 2
        • Kubernetes Argo Rollouts
        • Jenkins in Kubernetes
        • Ansible Semaphore
        • Image storage, scaning and signing
        • Install WireGuard with Gitlab and Terraform
        • CI/CD example fror small web app
        • Threat matrix for CI CD Pipeline
      • SIEM / SOC
        • Datadog
        • Splunk
          • Splunk — general description
        • MaxPatrol
          • MaxPatrol 8 and RedCheck Enterprise
        • QRadar IBM
        • Cloud Native Security Platform (CNAPP) - Aqua
        • OSSIM | AT&T
          • AlienVault (OSSIM) install
        • Wazuh
        • EDR
          • Cortex XDR | Palo Alto Networks
          • Cynet
          • FortiEDR | Fortinet
          • Elastic
        • Elastic
          • Install Elasticsearch, Logstash, and Kibana (Elastic Stack) on Ubuntu 22.04
          • Setting Up Elastic 8 with Kibana, Fleet, Endpoint Security, and Windows Log Collection
        • Threat Intelligence
          • MISP
          • msticpy Microsoft
          • X-Force | IBM
          • Elastic
      • VPN
        • Full-Mesh VPN fastd, tinc, VpnCloud
        • Wireguard
          • WireGuard for Internet access
          • WireGuard on MikroTik and Keenetic
          • WireGuard site to site
        • SoftEther VPN Project
        • Cisco AnyConnect client
        • OpenConnect
        • SSTP python server
      • OS hardening
        • CIS Benchmarks
      • Cloud Providers
      • OpenNebula
        • OpenNebula Edge Cloud - Open Source Cloud & Edge Computing
        • Discover OpenNebula – Open Source Cloud & Edge Computing Platform
        • OpenNebula Multi-Cloud
        • Kubernetes on OpenNebula
        • The Open Source Alternative to Nutanix
        • The Simple Alternative to OpenStack
        • OpenNebula Partner Ecosystem
      • OpenStack
        • Install manual
        • Install with DevStack
      • VM
        • Create a VHD file from a Linux disk
        • Backup / Migration
          • Coriolis
          • Proxmox Backup Server
        • oVirt
        • VMware vCenter
        • Proxmox
      • Docker
        • Container optimization
        • Ubuntu RDP container
      • LXC
        • LXD on Ubuntu 18.04
        • Install, Create and Manage LXC in Ubuntu/Debian
    • Big Data
      • OLAP data qubes
      • Storage and autoscale in Lerua
    • Machine Learning
      • Yandex YaLM 100B. GPT model
      • Kaggle Community Datasts Models
      • AI in video production
      • Image search
      • Chat bots
        • You.com
        • Chat GPT
          • Implementing GPT in NumPy
        • Jailbreak Chat
      • Coding plugins CodeWhisperer
    • Malware
      • Isiaon/Pitraix: Modern Cross-Platform Peer-to-Peer Botnet over TOR
      • theZoo A repository of LIVE malwares
    • Pentest
      • Red Team
        • MITRE ATT&CK matrix
        • C2 Frameworks
          • Brute Ratel C4
          • Cobalt Strike
          • Covenant
          • Havoc Framework
          • Merlin
          • Metasploit
          • Sillenttrinity
          • Sliver
        • Manage and report
          • Dradis Framework
          • Hexway
        • Underground
      • Social engineering
        • Social Engineer Toolkit setoolkit
      • OSINT
        • OSINT for comapny
        • Instagram fishing
      • Forensics
        • Forensics tools
      • Pentesting Methodology
      • Web
      • CI/CD Methodology
      • Cloud Methodology
        • Hacking The Cloud
      • Kubernetes Pentesting
      • Android
        • SSL Unpinning for Android applications
      • iOS
        • SSL unpinning iOS and macOS applications
      • HackBar tool
      • CyberChef Tools
      • Python virtualenv
      • IppSec - YouTube
      • Hacktricks.xyz
    • Compliance
      • 152 ФЗ. Personal data
      • PCI DSS and ГОСТ Р 57580.1-2017
      • Cloud compliance
      • ГОСТ Р 57580.1-2017 для Kubernetes
      • Kubernets as DevSecOps and NIST compliance
      • NIST SP 800-61 cyberincidece control
      • CIS Kubernetes Benchmark v1.6 - RKE2 v1.20
      • CIS Kubernetes Benchmark v1.23 - RKE2
      • Requirements for Russian Banks
      • Tools
        • Chef InSpec
        • Elastic SIEM
    • Asset management
      • CMDBuild
    • Project management
    • Incident management SRE
    • Risk management
      • IT risk management
      • BSI-Standard 200-3
    • Web Dev
      • Cookie security
      • OWASP Top 10 2021
      • Docker nginx php mysql
      • Docker tor hiddenservice nginx
      • Docker Compose wp nginx php mariadb
      • Dependency Checking
        • Nexus Analyzer
        • OWASP dependency-check
      • Yii skeeks cms
      • YiiStudio
    • Art
      • GTK Themes
      • Themes for Xfce Desktop
      • XFCE / Xubuntu Windows 95
      • Moscow events
      • Photo goods
      • Russian style gifts
    • Cryptocurrency
      • News
      • Arbitrage
      • Stocks
      • Exchange aggregators
      • Where to use
      • Prepaid cards
        • BitFree
        • Pyypl Your Money at Your Fingertips
    • IT magazines
      • WIKI and Writeups tools
        • BookStack
        • GitBook
        • MkDocs
        • Wiki.js
        • DokuWiki
    • Languages
    • Learning
      • (ISC)2
        • CISSP
      • Offensive Security
        • OSCP
        • OSEP
        • OSED
      • DevSecOps
        • Certified DevSecOps Professional (CDP)
        • Certified DevSecOps Expert (CDE)
      • Web Security Academy: PortSwigger
    • Relocation
      • London experience
      • IT visas in 2022
      • Remote work
      • Running business in UAE
    • Freenet
      • Independent online services: the philosophy of a free Internet
      • Tor Project Anonymity Online
      • I2P Anonymous Network
    • Services
      • SMS Registration
        • Registering ChatGPT in Russia
      • Local and regional eSIMs for travellers - Airalo
      • Digital busines cards
      • No KYC services and exchanges
Powered by GitBook
On this page
  • Kiosk
  • Multi-Tenancy Extension For Kubernetes
  • Contents
  • Why kiosk?
  • Architecture
  • Getting Started
  • Upgrade kiosk
  • Uninstall kiosk
  • Extra: User Management & Authentication
  • Contributing
  • About kiosk

Was this helpful?

  1. Book
  2. Architect
  3. Kubernetes
  4. Isolation

Kiosk

Last updated 1 year ago

Was this helpful?

Kiosk

  • Accounts & Account Users to separate tenants in a shared Kubernetes cluster

  • Self-Service Namespace Provisioning for account users

  • Account Limits to ensure quality of service and fairness when sharing a cluster

  • Namespace Templates for secure tenant isolation and self-service namespace initialization

Kubernetes is designed as a single-tenant platform, which makes it hard for cluster admins to host multiple tenants in a single Kubernetes cluster. However, sharing a cluster has many advantages, e.g. more efficient resource utilization, less admin/configuration effort or easier sharing of cluster-internal resources among different tenants.

While there are hundreds of ways of setting up multi-tenant Kubernetes clusters and many Kubernetes distributions provide their own tenancy logic, there is no lightweight, pluggable and customizable solution that allows admins to easily add multi-tenancy capabilities to any standard Kubernetes cluster.

kiosk is designed to be:

  • 100% Open-Source: CNCF compatible Apache 2.0 license

  • Pluggable: easy to install into any existing cluster and suitable for different use cases

  • Fast: emphasizing automation and self-service for tenants

  • Secure: offering default configurations for different levels of tenant isolation

  • Extensible: providing building blocks for higher-level Kubernetes platforms

The core idea of kiosk is to use Kubernetes namespaces as isolated workspaces where tenant applications can run isolated from each other. To minimize admin overhead, cluster admins are supposed to configure kiosk which then becomes a self-service system for provisioning Kubernetes namespaces for tenants.

The following diagram shows the main actors (Cluster Admins and Account Users) as well as the most relevant Kubernetes resources and their relationships.

Click on the following links to view the description for each of the actors and kiosk components:

Details

Cluster Admin

Cluster Admins have the permission to perform CRUD operations for cluster-wide / non-namespaced resources (especially RBAC related resources as well as the custom resources Account, AccountQuota, AccountQuotaSet, and Template). Cluster Admins configure kiosk by creating and managing Accounts, AccountQuotas, AccountQuotaSets, and Templates. They can also see and configure all Spaces owned by all Accounts.

Details

Account

Every tenant is represented by an Account. Cluster Admins define and manage Accounts and assign Account Users (Users, Groups, ServiceAccounts) to Accounts - similar to assigning RBAC Roles to subjects as part of a RoleBinding configuration.

Details

Account User

Account Users perform actions within the Kubernetes cluster via API server requests while using a certain Account. Cluster Admins can assign the same Account User to multiple Accounts. Account Users have access to Spaces that belong to the Accounts they are using. If assigned the default kiosk ClusterRole, every Account User has the permission to list/get/create/delete Spaces for the respective Account, however, this can be changed via RBAC RoleBindings.

Details

Space

A Space is a non-persistent, virtual resource that represents exactly one Kubernetes namespace. Spaces have the following characteristics:

  • Every space can belong up to one Account which is the owner of this Space. Ownerless Spaces are possible.

  • If a user has rights to access the underlying Namespace, the user can access the Space in the same way. Hence besides Account Users, other actors (User, Group, ServiceAccount) can also access the Space if someone grants this access via additional Kubernetes RBAC.

  • Every User only sees the Spaces the User has access to. This is in contrast to regular namespaces, where Users can only list all namespaces or none

  • Space ownership can be changed, by changing the ownership annotation on the namespace

  • During Space creation (or Space ownership changes) a RoleBinding for the owning Account is created in the corresponding Space namespace. The referenced RBAC ClusterRole can be configured in the account

  • A Space can be prepopulated during creation with a predefined set of resources by configuring default Templates in the Account. Kiosk will make sure that these resources will be correctly deployed before the user gets access to the namespace.

Details

Namespace

A Namespace is a regular Kubernetes Namespace that can be accessed by anyone who has the appropriate RBAC rules to do so. Namespaces are provisioned and managed by kiosk and have a 1-to-1 relationship to the resource Space which is a custom resource of kiosk. By default, Account Users have the permission to operate within all Namespaces that are represented by Spaces which belong to one of their Accounts.

Details

Template

Templates are defined and managed by Cluster Admins. Templates are used to initialize Spaces/Namespaces with a set of Kubernetes resources (defined as manifests or as part of a Helm chart). Templates can be created using a different ClusterRole than the Account User uses, so they can be used to create resources that are not allowed to be created by actors of the Space/Namespace, e.g. to set up certain isolation resources (e.g. Network Policies, Pod Security Policies etc.). Cluster Admins can define default Templates within the Account configuration which automatically applies these templates to each Space that is created using the respective Account. Additionally, Account Users can state other non-mandatory Templates that should also be applied when creating a Space.

Details

TemplateInstance

When a Template is applied to a Space, kiosk creates a TemplateInstance to keep track of which Templates have been applied to the Space. A TemplateInstance contains information about the Template as well as about the parameters used to instantiate it. Additionally, TemplateInstances can be configured to sync with Templates, i.e. the TemplateInstance will update the resources whenever the Template changes that has been used to create these resources.

Details

AccountQuota

AccountQuotas are defined and managed by Cluster Admins. AccountQuotas define cluster-wide aggregated limits for Accounts. The resources of all Spaces/Namespaces that belong to an Account count towards the aggregated limits defined in the AccountQuota. Similar to Namespaces which can be limited by multiple ResourceQuotas, an Account can be limited by multiple AccountQuotas. If the same limit (e.g. total CPU per Account) is defined by multiple AccountQuotas, the Account will be limited according to the lowest value.

When installing kiosk in a Kubernetes cluster, these components will be added to the cluster:

  • CRDs for Account, AccountQuota, AccountQuotaSet, Template, TemplateInstance

  • Controller for kiosk Custom Resources (runs inside the cluster)

  • API Server Extension (runs inside the cluster similar to the Controller)

kiosk adds two groups of resources to extend the Standard API Groups of Kubernetes:

  1. Details Show List of Custom Resources

    Custom Resources: config.kiosk.sh

    Custom Resource Definitions (CRDs) for configuring kiosk. These resources are persisted in etcd just like any other Kubernetes resources and are managed by an operator which runs inside the cluster.

    • config.kiosk.sh/Account

    • config.kiosk.sh/AccountQuota

    • config.kiosk.sh/AccountQuotaSet (soon)

    • config.kiosk.sh/Template

    • config.kiosk.sh/TemplateInstance

  2. Details Show List of API Extension Resources

    API Extension: tenancy.kiosk.sh

    Virtual resources which are accessible via an API Server Extension and will not be persisted in etcd. These resources are similar to views in a relational database. The benefit of providing these resources instead of only using CRDs is that we can calculate access permissions dynamically for every request. That means that it does not only allow to list, edit and manage Spaces (which map 1-to-1 to Namespaces), it also allows to show a different set of Spaces for different Account Users depending on the Accounts they are associated with or in other words: this circumvents the current limitation of Kubernetes to show filtered lists of cluster-scoped resources based on access rights.

    • tenancy.kiosk.sh/Account

    • tenancy.kiosk.sh/AccountQuota

    • tenancy.kiosk.sh/Space

    • tenancy.kiosk.sh/TemplateInstance

kiosk supports Kubernetes version: v1.14 and higher. Use kubectl version to determine the Server Version of your cluster. While this getting started guide should work with most Kubernetes clusters out-of-the-box, there are certain things to consider for the following types of clusters:

Details

Docker Desktop Kubernetes

Details

Digital Ocean Kubernetes (DOKS)

Details

Google Kubernetes Engine (GKE)

Your kube-context will by default not have cluster-admin role. Run the following command to get your google email address and to make your user cluster admin:

# GKE: make yourself admin
GKE_USER=$(gcloud config get-value account)
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $GKE_USER

You need a kube-context with admin rights.

If running all the following commands returns yes, you are most likely admin:

kubectl auth can-i "*" "*" --all-namespaces
kubectl auth can-i "*" namespace
kubectl auth can-i "*" clusterrole
kubectl auth can-i "*" crd
# Install kiosk with helm v3
kubectl create namespace kiosk
helm install kiosk --repo https://charts.devspace.sh/ kiosk --namespace kiosk --atomic

To verify the installation make sure the kiosk pod is running:

$ kubectl get pod -n kiosk

NAME                     READY   STATUS    RESTARTS   AGE
kiosk-58887d6cf6-nm4qc   2/2     Running   0          1h

In this getting started guide, we assume two user roles:

  • Cluster Admin: use your admin-context as current context (kubectl commands without -as flag)

  • Account User john: use your admin-context to impersonate a user (kubectl commands with -as=john)

To allow a user to create and manage namespaces, they need a kiosk account. Run the following command to create such an account for our example user john:

# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/account.yaml

# Alternative: ServiceAccount as Account User (see explanation for account-sa.yaml below)
# kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/account-sa.yaml

Details

View: account.yaml

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Account
metadata:
  name: johns-account
spec:
  subjects:
  - kind: User
    name: john
    apiGroup: rbac.authorization.k8s.io

As you can see in this example, every account defines subjects which are able to use this account. In this example, there is only one subject which is a User with name john. However, Accounts can also have multiple subjects.

Subjects for kiosk Accounts are defined in the exact same way as subjects in RoleBindings. Subjects can be a combination of:

  • Users

  • Groups

  • ServiceAccounts (see example below: account-sa.yaml)

Details

View: account-sa.yaml (alternative for ServiceAccounts, e.g. Digital Ocean Kubernetes)

If you want to assign an Account to a ServiceAccount (e.g. when using Digital Ocean Kubernetes / DOKS), please use the following alternative:

# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/account-sa.yaml
apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Account
metadata:
  name: johns-account
spec:
  subjects:
  - kind: ServiceAccount
    name: john
    namespace: kiosk

All Account Users are able to view their Account through their generated ClusterRole. Let's try this by impersonating john:

# View your own accounts as regular account user
kubectl get accounts --as=john

# View the details of one of your accounts as regular account user
kubectl get account johns-account -o yaml --as=john

Spaces are the virtual representation of namespaces. Each Space represents exactly one namespace. The reason why we use Spaces is that by introducing this virtual resource, we can allow users to only operate on a subset of namespaces they have access to and hide other namespaces they shouldn't see.

By default, Account Users cannot create Spaces themselves. They can only use the Spaces/Namespaces that belong to their Accounts. That means a cluster admin would need to create the Spaces for an Account and then the Account Users could work with these Spaces/Namespaces.

To allow all Account Users to create Spaces for their own Accounts, create the following RBAC ClusterRoleBinding:

# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/rbac-creator.yaml

Details

View: rbac-creator.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kiosk-creator
subjects:
- kind: Group
  name: system:authenticated
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: kiosk-edit
  apiGroup: rbac.authorization.k8s.io

Of course, you can also adjust this ClusterRoleBinding in a way that only certain subjects/users can create Spaces for their Accounts. Just modify the subjects section.

After granting Account Users the right to create Spaces for their Accounts (see ClusterRoleBinding in 3.1.), all Account Users are able to create Spaces. Let's try this by impersonating john:

kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/space.yaml --as=john

Details

View: space.yaml

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Space
metadata:
  name: johns-space
spec:
  # spec.account can be omitted if the current user only belongs to a single account
  account: johns-account

As you can see in this example, every Space belongs to exactly one Account which is referenced by spec.account.

Let's take a look at the Spaces of the Accounts that User john owns by impersonating this user:

# List all Spaces as john:
kubectl get spaces --as=john

# Get the defails of one of john's Spaces:
kubectl get space johns-space -o yaml --as=john

Every Space is the virtual representation of a regular Kubernetes Namespace. That means we can use the associated Namespace of our Spaces just like any other Namespace.

Let's impersonate john again and create an nginx deployment inside johns-space:

kubectl apply -n johns-space --as=john -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/application/deployment.yaml

That's great, right? A user that did not have any access to the Kubernetes cluster, is now able to create Namespaces on-demand and gets restricted access to these Namespaces automatically.

To allow Account Users to delete all Spaces/Namespace that they create, you need to set the spec.space.clusterRole field in the Account to kiosk-space-admin.

When creating a Space, kiosk creates the according Namespace for the Space and then creates a RoleBinding within this Namespace which binds the standard Kubernetes ClusterRole admin to every Account User (i.e. all subjects listed in the Account). While this ClusterRole allows full access to this Namespace, it does not allow to delete the Space/Namespace. (The verb delete is missing in the default admin clusterrole)

As john can be User of multiple Accounts, let's create a second Account which allows john to delete Spaces/Namespaces that belong to this Account:

# Run this as cluster admin:
# Create Account johns-account-deletable-spaces
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/account-deletable-spaces.yaml

Details

View: account-deletable-spaces.yaml

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Account
metadata:
  name: johns-account-deletable-spaces
spec:
  space:
    clusterRole: kiosk-space-admin
  subjects:
  - kind: User
    name: john
    apiGroup: rbac.authorization.k8s.io

If you are using ServiceAccounts instead of impersonation, adjust the subjects section of this Account similar to account-sa.yaml in 2.1.

Now, let's create a Space for this Account:

# Run this as john:
# Create Space johns-space-deletable
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/space-deletable.yaml --as=john

Details

View: space-deletable.yaml

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Space
metadata:
  name: johns-space-deletable
spec:
  account: johns-account-deletable-spaces

If a Space belongs to an Account that allows Account Users to delete such Spaces, an Account User can simply delete the Space using kubectl:

kubectl get spaces --as=john
kubectl delete space johns-space-deletable --as=john
kubectl get spaces --as=john

Deleting a Space also deletes the underlying Namespace.

kiosk provides the spec.space.spaceTemplate option for Accounts which lets admins define defaults for new Spaces of an Account. The following example creates the Account account-default-space-metadata which defines default labels and annotations for all Spaces created with this Account:

# Run this as cluster admin:
# Create Account johns-account-default-space-metadata
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/account-default-space-metadata.yaml

Details

View: account-default-space-metadata.yaml

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Account
metadata:
  name: johns-account-default-space-metadata
spec:
  space:
    clusterRole: kiosk-space-admin
    spaceTemplate:
      metadata:
        labels:
          some-label: "label-value"
          some--other-label: "other-label-value"
        annotations:
          "space-annotation-1": "annotation-value-1"
          "space-annotation-2": "annotation-value-2"
  subjects:
  - kind: User
    name: john
    apiGroup: rbac.authorization.k8s.io

With kiosk, you have two options to limit Accounts:

By setting the spec.space.limit in an Account, Cluster Admins can limit the number of Spaces that Account Users can create for a certain Account.

Let's run the following command to update the existing Account johns-account and specify spec.space.limit: 2:

# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/account-space-limit.yaml

Details

View: account-space-limit.yaml

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Account
metadata:
  name: johns-account
spec:
  space:
    limit: 2
  subjects:
  - kind: User
    name: john
    apiGroup: rbac.authorization.k8s.io

Now, let's try to create more than 2 Spaces (note that you may have already created a Space for this Account during earlier steps of this getting started guide):

# List existing spaces:
kubectl get spaces --as=john

# Create space-2 => should work if you had only one Space for this Account so far
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/space-2.yaml --as=john

# Create space-3 => should result in an error
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/space-3.yaml --as=john

AccountQuotas allow you to define limits for an Account which are aggregated across all Spaces of this Account.

# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/accountquota.yaml

Details

View: accountquota.yaml

apiVersion: config.kiosk.sh/v1alpha1
kind: AccountQuota
metadata:
  name: default-user-limits
spec:
  account: johns-account
  quota:
    hard:
      pods: "2"
      limits.cpu: "4"

If there are multiple AccountQuotas referencing the same Account via spec.account, kiosk merges the Quotas. In case multiple AccountQuotas define different limits for the same resource type, kiosk uses the lowest value.

Templates in kiosk are used to initialize Namespaces and share common resources across namespaces (e.g. secrets). When creating a Space, kiosk will use these Templates to populate the newly created Namespace for this Space. Templates:

  • can contain parameters such as ${NAMESPACE} or ${MY_PARAMETER} that can be specified within an TemplateInstance

The easiest option to define a Template is by specifying an array of Kubernetes manifests which should be applied when the Template is being instantiated.

# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/template-manifests.yaml

Details

View: template-manifests.yaml

apiVersion: config.kiosk.sh/v1alpha1
kind: Template
metadata:
  name: space-restrictions
# This section defines parameters that can be used for this template
# Can be used in resources.manifests and resources.helm.values
parameters:
# Name of the parameter
- name: DEFAULT_CPU_LIMIT
  # The default value of the parameter
  value: "1"
- name: DEFAULT_CPU_REQUESTS
  value: "0.5"
  # If a parameter is required the template instance will need to set it
  # required: true
  # Make sure only values are entered for this parameter
  validation: "^[0-9]*\\.?[0-9]+$"
resources:
  manifests:
  - kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: deny-cross-ns-traffic
    spec:
      podSelector:
        matchLabels:
      ingress:
      - from:
        - podSelector: {}
  - apiVersion: v1
    kind: LimitRange
    metadata:
      name: space-limit-range
    spec:
      limits:
      - default:
          # Use the DEFAULT_CPU_LIMIT parameter here and
          # parse it as json, which renders the "1" as 1.
          cpu: "${{DEFAULT_CPU_LIMIT}}"
        defaultRequest:
          cpu: "${{DEFAULT_CPU_REQUESTS}}"
        type: Container

Instead of manifests, a Template can specify a Helm chart that will be installed (using helm template) when the Template is being instantiated. Let's create a Template called redis which installs the stable/redis Helm chart:

# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/template-helm.yaml

Details

View: template-helm.yaml

apiVersion: config.kiosk.sh/v1alpha1
kind: Template
metadata:
  name: redis
resources:
  helm:
    releaseName: redis
    chart:
      repository:
        name: redis
        repoUrl: https://kubernetes-charts.storage.googleapis.com
    values: |
      redisPort: 6379
      # Use a predefined parameter here
      myOtherValue: ${NAMESPACE}

By default, only admins can list Templates. To allow users to view templates, you need to set up RBAC accordingly. Run the following code to allow every cluster user to list and view all Templates:

# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/rbac-template-viewer.yaml

Details

View: rbac-template-viewer.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kiosk-template-view
rules:
- apiGroups:
  - config.kiosk.sh
  resources:
  - templates
  verbs:
  - get
  - list
  - watch
  - create
  - delete
  - deletecollection
  - patch
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kiosk-template-viewer
subjects:
- kind: Group
  name: system:authenticated
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: kiosk-template-view
  apiGroup: rbac.authorization.k8s.io

To view a list of available Templates, run the following command:

kubectl get templates --as=john
# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/rbac-template-instance-admin.yaml

Note: Creating a TemplateInstance in a Space is only possible if a RoleBinding exists that binds the Role kiosk-template-admin to the user. Because kiosk-template-admin has the label rbac.kiosk.sh/aggregate-to-space-admin: "true" (see rbac-instance-admin.yaml below), it is also possible to create a RoleBinding for the Role kiosk-space-admin (which automatically includes kiosk-template-admin).

Details

View: rbac-instance-admin.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kiosk-template-admin
  labels:
    rbac.kiosk.sh/aggregate-to-space-admin: "true"
rules:
- apiGroups:
  - config.kiosk.sh
  resources:
  - templateinstances
  verbs:
  - get
  - list
  - watch
  - create
  - delete
  - deletecollection
  - patch
  - update
kubectl apply --as=john -n space-2 -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/template-instance.yaml

Note: In the above example, we are using space-2 which belongs to Account johns-account-deletable-spaces. This Account defines space.clusterRole: kiosk-space-admin which automatically creates a RoleBinding for the Role kiosk-space-admin when creating a new Space for this Account.

Details

View: template-instance.yaml

apiVersion: config.kiosk.sh/v1alpha1
kind: TemplateInstance
metadata:
  name: redis-instance
spec:
  template: redis
  # You can also specify a parameter value here
  # parameters:
  # - name: DEFAULT_CPU_REQUESTS
  #   value: "1"

Templates can either be mandatory or optional. By default, all Templates are optional. Cluster Admins can make Templates mandatory by adding them to the spec.space.templateInstances array within the Account configuration. All Templates listed in spec.space.templateInstances will always be instantiated within every Space/Namespace that is created for the respective Account.

Let's see this in action by updating the Account johns-account and referencing our space-restrictions Template from 5.1. in spec.space.templateInstances:

# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/account-default-template.yaml

Details

View: account-default-template.yaml

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Account
metadata:
  name: johns-account-deletable-spaces
spec:
  space:
    clusterRole: kiosk-space-admin
    templateInstances:
    - spec:
        template: space-restrictions
        # Specifying parameter values here is also possible
        parameters:
          - name: DEFAULT_CPU_REQUESTS
            value: "2"
  subjects:
  - kind: User
    name: john
    apiGroup: rbac.authorization.k8s.io

If you are using ServiceAccounts instead of impersonation, adjust the subjects section of this Account similar to account-sa.yaml in 2.1.

Now, let's create a Space without specifying any templates and see how this Template will automatically be instantiated:

kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/space-template-mandatory.yaml --as=john

Details

View: space-template-mandatory.yaml

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Space
metadata:
  name: johns-space-template-mandatory
spec:
  account: johns-account # can be omitted if the user only has 1 account

Now, we can run the following command to see that the two resources (PodSecurityPolicy and LimitRange) defined in our Template space-restrictions have been created inside the Space/Namespace:

# Run this as cluster admin:
kubectl get podsecuritypolicy,limitrange -n johns-space-template-mandatory

Mandatory Templates are generally used to enforce security restrictions and isolate namespaces from each other while Optional Templates often provide a set of default applications that a user might want to choose from when creating a Space/Namespace (see example in 5.2).

To keep track of resources created from Templates, kiosk creates a so-called TemplateInstance for each Template that is being instantiated inside a Space/Namespace.

To view the TemplateInstances of the namespace johns-space-template-mandatory, run the following command:

# Run this as cluster admin:
kubectl get templateinstances -n johns-space-template-mandatory

Generally, a TemplateInstance is created from a Template and then, the TemplateInstances will not be updated when the Template changes later on. To change this behavior, it is possible to set spec.sync: true in a TemplateInstance. Setting this option, tells kiosk to keep this TemplateInstance in sync with the underlying template using a 3-way merge (similar to helm upgrade).

kubectl apply --as=john -n space-2 -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/template-instance-sync.yaml

Details

View: template-instance-sync.yaml

apiVersion: config.kiosk.sh/v1alpha1
kind: TemplateInstance
metadata:
  name: redis-instance-sync
spec:
  template: redis
  sync: true
  # You can specify parameters here as well
  # parameters:
  # - name: DEFAULT_CPU_REQUESTS
  #   value: "1"
helm upgrade kiosk --repo https://charts.devspace.sh/ kiosk -n kiosk --atomic --reuse-values

Do not skip releases with release notes containing upgrade instructions!

helm delete kiosk -n kiosk

kiosk does not provide a built-in user management system.

If you like to use ServiceAccounts for a small and easy to set up authentication and user management, you can use the following instructions to create new users / kube-configs.

Use bash to run the following commands.

USER_NAME="john"
kubectl -n kiosk create serviceaccount $USER_NAME
# If not already set, then:
USER_NAME="john"

KUBECONFIG_PATH="$HOME/.kube/config-kiosk"

kubectl config view --minify --raw >$KUBECONFIG_PATH
export KUBECONFIG=$KUBECONFIG_PATH

CURRENT_CONTEXT=$(kubectl config current-context)
kubectl config rename-context $CURRENT_CONTEXT kiosk-admin

CLUSTER_NAME=$(kubectl config view -o jsonpath="{.clusters[].name}")
ADMIN_USER=$(kubectl config view -o jsonpath="{.users[].name}")

SA_NAME=$(kubectl -n kiosk get serviceaccount $USER_NAME -o jsonpath="{.secrets[0].name}")
SA_TOKEN=$(kubectl -n kiosk get secret $SA_NAME -o jsonpath="{.data.token}" | base64 -d)

kubectl config set-credentials $USER_NAME --token=$SA_TOKEN
kubectl config set-context kiosk-user --cluster=$CLUSTER_NAME --user=$USER_NAME
kubectl config use-context kiosk-user

# Optional: delete admin context and user
kubectl config unset contexts.kiosk-admin
kubectl config unset users.$ADMIN_USER

export KUBECONFIG=""
# If not already set, then:
KUBECONFIG_PATH="$HOME/.kube/config-kiosk"

export KUBECONFIG=$KUBECONFIG_PATH

kubectl ...
export KUBECONFIG=""

kubectl ...

There are many ways to get involved:

  • Open an issue for questions, to report bugs or to suggest new features

  • Open a pull request to contribute improvements to the code base or documentation

This is a very new project, so we are actively looking for contributors and maintainers. Reach out if you are interested.

Multi-Cluster Tenant Management for sharing a pool of clusters ()

kubectl: Follow to install it.

helm version 3: Follow to install it.

All ServiceAccounts have cluster-admin role by default, which means that emulating users with ServiceAccounts is not a good idea. Use instead.

All users in DOKS have cluster-admin role by default which means that when using impersonation, every user will have admin access. To see kiosk-based multi-tenancy in action, .

In the following steps, we will use to allow you to quickly switch between cluster admin and simple account user roles. If you are cluster admin and you want to run a kubectl command as a different user, you can impersonate this user by adding the kubectl flags --as=[USER] and/or --as-group=[GROUP].

If you are using Digital Ocean Kubernetes (DOKS), .

for an Account

= maximum values for resources aggregated across all Spaces of an Account (e.g. total number of Pods combined in all Spaces)

Let's create an AccountQuota for johns-account which will set the aggregated number of Pods across all Spaces to 2 and the aggregated maximum of limits.cpu across all Pods in all Spaces to 4 CPU Cores (see ):

AccountQuotas allow you to restrict the same resources as but unlike ResourceQuotas, AccountQuotas are not restricted to a single Namespace. Instead, AccountQuotas add up all used resources across all Spaces of an Account to generate an aggregated value which is then compared to the max value defined in the AccountQuota.

can either contain one or more or

are being tracked by in each Namespace they are applied to

The following command will create a Template called space-restrictions which defined 2 manifests, a which will make sure that the users of this Space/Namespace cannot create privileged containers and a of containers in this Namespace:

To instantiate a Template, users need to have permission to create within their Namespaces. You can grant this permission by running this command:

After creating the ClusterRole kiosk-template-admin as shown above, users can instantiate templates inside their Namespaces by creating so-called . The following example creates an instance of the Helm Chart :

TemplateInstances allow admins and user to see which Templates are being used within a Space/Namespace and they make it possible to upgrade the resources created by a Template if there is a newer version of the Template ().

The following example creates an instance of the Helm Chart and defines that this TemplateInstance should be kept in sync with the underlying Template:

Check the for details on how to upgrade to a specific release.

To manage users in your cluster, you can either use vendor-neutral solutions such as or or alternatively, if you are in a public cloud, you may be able to use provider-specific solutions such as or .

Email one of the maintainers (, ) to find out more about the project and how to get involved

For more detailed information, see our .

kiosk is an open-source project licensed under Apache-2.0 license. The project will be contributed to once it reaches the required level of popularity and maturity. The first version of kiosk was developed by as core component for their DevSpace Cloud on-premise edition.

coming soon
Contents
Why kiosk?
Architecture
Workflow & Interactions
Custom Resources & Resource Groups
Getting Started
0. Requirements
1. Install kiosk
2. Configure Accounts
3. Working with Spaces
4. Setting Account limits
5. Working with Templates
Upgrade kiosk
Uninstall kiosk
Extra: User Management & Authentication
Contributing
Why kiosk?
The Missing Multi-Tenancy Extension for Kubernetes
Kiosk/68747470733a2f2f696d672e736869656c64732e696f2f747769747465722f75726c3f7374796c653d736f6369616c2675726c3d68747470732533412532462532466769746875622e636f6d2532466b696f736b2d73682532466b696f736b
Architecture
Workflow & Interactions
Custom Resources & Resource Groups
Getting Started
0. Requirements
0.1. CLI Tools
this guide
this guide
0.2. Kubernetes Cluster
impersonation
create ServiceAccounts to emulate different users
0.3. Admin Context
1. Install kiosk
2. Configure Accounts
Kubernetes user impersonation
follow this guide to simulate a user using a Service Account
2.1. Create Account
Learn more about User Management and Accounts in kiosk.
2.2. View Accounts
3. Working with Spaces
3.1. Allow Users To Create Spaces
3.2. Create Spaces
3.3. View Spaces
3.4. Use Spaces
3.5. Create Deletable Spaces
3.6. Delete Spaces
3.7. Defaults for Spaces
4. Setting Account Limits
Limit Number of Spaces
Set Account Quotas
4.1. Limit Number of Spaces
4.2. AccountQuotas
Kubernetes resource limits
Kubernetes ResourceQuotas
5. Working with Templates
Kubernetes manifests
alternatively a Helm chart
TemplateInstances
5.1. Manifest Templates
Network Policy
LimitRange for default CPU limits
5.2. Helm Chart Templates
5.3. Using Templates
TemplateInstances
TemplateInstances
Template redis which has been created above
5.4. Mandatory vs. Optional Templates
5.5. TemplateInstances
coming soon
5.6. Template Sync
Template redis which has been created above
Upgrade kiosk
release notes
Uninstall kiosk
Extra: User Management & Authentication
dex
DevSpace Cloud
AWS IAM for EKS
GCP IAM for GKE
Using ServiceAccounts For Authentication
1. Create a ServiceAccount
2. Create Kube-Config For ServiceAccount
3. Use ServiceAccount Kube-Config
4. Reset Kube-Config
Contributing
Fabian
Lukas
Contributing Guide
About kiosk
CNCF
DevSpace Technologies
https://github.com/loft-sh/kiosk
Multi-Tenancy Extension For Kubernetes