Stop Making Kubernetes Auth Hard

September 19, 2024 by Thomas Rampelberg

I’ve spent most of my time working with Kubernetes being afraid of auth. I understood how RBAC works and I knew that .kube/config is what’s required to talk to an API server, but that’s pretty much where my understanding stopped. Configuring the API server to use an auth plugin, getting tokens or certificates and setting up plugins on my (and all my user’s) laptops made me think that it was all a monumental task. Just getting my environment setup correctly was a monumental task. Well, as part of implementing kty’s oauth support, I’ve been forced to figure out how it all actually works. And, as turns out, it doesn’t need to be nearly as complex as I thought it was.

TL;DR

Use OpenID and grant groups or users the correct permissions in your cluster. Your organization already has an OpenID provider in place. Google, GitHub, Okta (and many more) can all be used. That’s it, that’s all you need. Don’t bother with IAM, service accounts or any of that other stuff. Those are all reasonable for machines - not for users.

If you’d like to see how easy it is to get running, check out the getting started guide. kty uses OpenID to verify your identity over SSH and then takes care of the rest for you.

Give it a shot with kubectl as well. Check out kubelogin, a kubectl plugin that will do the OIDC dance for you. Note that if you can’t make the modifications required for the API server, you’ll want to use an oidc-proxy. Luckily, most Kubernetes solutions support OIDC out of the box like EKS or GKE.

You can hear a little bit about Robinhoood’s journey to this setup from their Kubecon talk. Now, if you’re interested in how it all works and would like to understand the details, keep reading.

Authentication

Let’s start out by splitting “auth” into two parts: authentication and authorization. Authentication is how you prove who you are. The result of the authentication process is an identity that can be used to see what you are, or aren’t authorized to do. If we didn’t actually care about verifying your identity, authentication could be nothing more than sending the username in cleartext to the API server. Obviously, we’d like a solution that is a little bit more secure than that.

Kubernetes has a whole bunch of ways to authenticate. Because it is the easiest to understand, let’s start with the static token file. This is equivalent to having a username as password. You put the token AKA “password” into the file and then associate it with a username. If this sounds like /etc/passwd, that’s because it is! Each request sent to the API server contains your token as a header. The API server looks up the token in its file and maps that to a user or set of groups. Very similar to sending the username to the API server, but now we’ve got a piece of shared data, the token, that verifies the identity.

Open ID Connect (OIDC) gets rid of the pre-shared secret and instead uses some cryptography magic to do the same thing. This allows for identity to be created in a central location and subsequently verified by anyone. When you authenticate with an OIDC provider, the end result of the process is an ID token.

The ID token is a JSON web token (JWT) that contains a bunch of information about your identity. This is signed by the provider and can be verified by anyone with the public key. Most importantly, OIDC providers publish their configuration so that anyone can verify the token. If you’re interested in what’s in that configuration, check out [kty’s][odic-config].

With an ID token and the way to verify it in hand, the API server can extract an identity from the token and use that as part of RBAC to understand what you’re allowed to do. The association between the token and either groups or users happens as part of claims. If you’ve got a JWT, you can see the claims in your token by going to jwt.io and pasting it in. Here’s a token that I’ve gotten for kty.

{
  "iss": "https://kty.us.auth0.com/",
  "aud": "P3g7SKU42Wi4Z86FnNDqfiRtQRYgWsqx",
  "iat": 1726784050,
  "exp": 1726820050,
  "sub": "github|123456",
  "email": "me@my-domain.com"
}

For this token, we could configure the API server to map the email claim to a user. This is just like the token file from above! Instead of using the pre-shared secret as the mapping, we’ve used the public key from the OIDC provider.

Authorization

Here’s where it gets interesting. Now that we have a verified identity, authorization can take place. We’ll check a list of rules (or roles) and test whether the identity can do the action requested. Kubernetes’ role based access control system doesn’t care about how you authenticated. If the API says you’re a user - then you are that user. All it cares about is your identity and what roles that identity is bound to. Let’s look at a simple role:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: view
rules:
  - apiGroups:
      - ''
    resources:
      - pods
    verbs:
      - get
      - list
      - watch

Any identity that is bound to this role can get, list or watch pods in any namespace. How does an identity get associated with this role? That’s where the ClusterRoleBinding comes into play.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: view
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: me@my-domain.com

Assuming that we’re still talking about the token from above, this role binding associates all the permissions in the view role with the user me@my-domain.com. That’s it! We’ve authenticated the identity and then verified that it can do some actions on the cluster. As RBAC is opt-in, you start off with no permissions and need to be granted them to do anything. There are some policies that come by default, in fact the view cluster role is one that comes out of the box (but simplified in this example). To see what can be granted, make sure to check out the documentation.

For extra credit, you can also bind roles to groups. We can configure a claim from the JWT to be a group in addition to the email address. Imagine granting permissions on a cluster based on which teams a user is a part of. In fact, you can map almost anything from someone’s GitHub profile directly over to a group. This way, you can setup permissions once and manage membership entirely through your OIDC provider. When using groups, the role binding ends up looking a little different:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: view
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: my-team

Bringing it Together

So, what does this all mean? Well, it means that we’ve now got a central location to manage access to our cluster. If you’re using groups, membership when the token is granted is mapped to a role binding that grants exactly what someone needs to work with your cluster. The IDs can be user friendly, so you can read through the RoleBinding YAML to understand what’s allowed or not. If you’re using kty, you don’t even need any plugins or configuration! Your users can use ssh and immediately get access to the cluster.

Please don’t be afraid of auth! Don’t continue to use incredibly complex systems consisting of multiple plugins, webhooks, tokens and certificates. They’re all hard to setup and/or easy to break. After all, security everyone can follow is the best security. Say no to services that require blanket permissions like the Kubernetes dashboard. Use OIDC and make sure that users have exactly the permissions they need.