👇🏿 🏕️ 🌏 Launching Keycloak in HA mode on Kubernetes 💤 ⛹🏻 🎒

TL; DR : There will be a description of Keycloak, an open source access control system, an analysis of the internal device, setup details.

Introduction and main ideas

In this article, we'll see the basic ideas to keep in mind when deploying a Keycloak cluster on top of Kubernetes.

Keycloak — . , — , ( , , . ).

Keycloak — , Java Wildfly. , framework , SSO (single sign-on).

Keycloak

Keycloak :

, ,
Datagrid cache, , , . Infinispan, . Infinispan — - .

Keycloak :

— , standalone.xml
( ) — , . standalone-ha.xml, .
— , . . domain.xml
— , Keycloak , . Keycloak .

, , , Kubernetes. Kubernetes ( Keycloak), .

, Keycloak, , Kubernetes.

Keycloak

Keycloak :

ip multicast

, . , - — . .

, Keycloak (HA) , , Wildfly .

Wildfly , , — . , . :

mod_cluster: Apache HTTP, TCP multicast . .
infinispan: , JGroups . HotRod Infinispan .
jgroups: JGroups. , , , , .

ingress Kubernetes :

Keycloak , , HTTP , ip- . ingress HTTP X-Forwarded-For X-Forwarded-Proto, HOST. ingress-nginx (> 0.22.0)

proxy-address-forwarding PROXY_ADDRESS_FORWARDING true Keycloak , proxy.

sticky sessions ingress. Keycloak Infinispan , . , , , .

cookie AUTH_SESSION_ID. Keycloak , cookie sticky session.

Keycloak , , AUTH_SESSION_ID, , . JAVA_OPTS jboss.node.name jboss.tx.node.id — . — 23 jboss, StatefulSet, Deployment.

— , . , . — Wildfly , /opt/jboss/startup-scripts :

embed-server --server-config=standalone-ha.xml --std-out=echo
batch

echo * Setting CACHE_OWNERS to "${env.CACHE_OWNERS}" in all cache-containers

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:write-attribute(name=owners, value=${env.CACHE_OWNERS:1})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions:write-attribute(name=owners, value=${env.CACHE_OWNERS:1})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=actionTokens:write-attribute(name=owners, value=${env.CACHE_OWNERS:1})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions:write-attribute(name=owners, value=${env.CACHE_OWNERS:1})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=clientSessions:write-attribute(name=owners, value=${env.CACHE_OWNERS:1})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineClientSessions:write-attribute(name=owners, value=${env.CACHE_OWNERS:1})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures:write-attribute(name=owners, value=${env.CACHE_OWNERS:1})

run-batch
stop-embedded-server

CACHE_OWNERS .

ip multicast

Weavenet CNI, multicast — Keycloak , .

ip multicast Kubernetes, JGroups .

— KUBE_DNS, headless service Keycloak, JGroups , .

— KUBE_PING, API ( serviceAccount list get, serviceAccount).

JGroups JGROUPS_DISCOVERY_PROTOCOL JGROUPS_DISCOVERY_PROPERTIES. KUBE_PING namespace labels.

️ multicast Keycloak Kubernetes ( namespace production, — staging) — Keycloak . multicast jboss.default.multicast.address jboss.modcluster.multicast.address JAVA_OPTS.

Keycloak Infinispan , Keycloack, Keycloak. Keycloak .

Keycloak Java Data Grid ( Infinispan) . Infinispan HotRod.

Infinispan remoteStore, , ( , . ) . infinispan JDG , , JDG1 site1 JDG2 site2.

, JDG Keycloak , HotRod. Keycloak site2 Infinispan, Keycloak site2.

Infinispan. remote-store Infinispan ( standalone-ha.xml), replicated-cache Infinispan .

Keycloak:

. , , . realm, , . , — Keycloak. — , . . work , .
. , offline , . — , , .

Infinispan

— Keycloak, , authenticationSessions, . Keycloak, . sticky sessions, , Active-Active .

. , , , , - . , forget password actionTokens — , . .

, . , . Keycloak , , . Keycloak realms, users authorization .

work, . - , . , , Keycloak , . .

. sessions, clientSessions, offlineSessions offlineClientSessions, , . , HTTP , sticky sessions .

. loginFailures , . — . . , , — .

Infinispan :

<replicated-cache-configuration name="keycloak-sessions" mode="ASYNC" start="EAGER" batching="false">
</replicated-cache-configuration>

<replicated-cache name="work" configuration="keycloak-sessions" />
<replicated-cache name="sessions" configuration="keycloak-sessions" />
<replicated-cache name="offlineSessions" configuration="keycloak-sessions" />
<replicated-cache name="actionTokens" configuration="keycloak-sessions" />
<replicated-cache name="loginFailures" configuration="keycloak-sessions" />
<replicated-cache name="clientSessions" configuration="keycloak-sessions" />
<replicated-cache name="offlineClientSessions" configuration="keycloak-sessions" />

Infinispan Keycloak

remoteStore Keycloak . , , CACHE_OWNERS, /opt/jboss/startup-scripts:

embed-server --server-config=standalone-ha.xml --std-out=echo
batch

echo *** Update infinispan subsystem ***
/subsystem=infinispan/cache-container=keycloak:write-attribute(name=module, value=org.keycloak.keycloak-model-infinispan)

echo ** Add remote socket binding to infinispan server **
/socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-cache:add(host=${remote.cache.host:localhost}, port=${remote.cache.port:11222})

echo ** Update replicated-cache work element **
/subsystem=infinispan/cache-container=keycloak/replicated-cache=work/store=remote:add( \
    passivation=false, \
    fetch-state=false, \
    purge=false, \
    preload=false, \
    shared=true, \
    remote-servers=["remote-cache"], \
    cache=work, \
    properties={ \
        rawValues=true, \
        marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \
        protocolVersion=${keycloak.connectionsInfinispan.hotrodProtocolVersion} \
    } \
)

/subsystem=infinispan/cache-container=keycloak/replicated-cache=work:write-attribute(name=statistics-enabled,value=true)

echo ** Update distributed-cache sessions element **
/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions/store=remote:add( \
    passivation=false, \
    fetch-state=false, \
    purge=false, \
    preload=false, \
    shared=true, \
    remote-servers=["remote-cache"], \
    cache=sessions, \
    properties={ \
        rawValues=true, \
        marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \
        protocolVersion=${keycloak.connectionsInfinispan.hotrodProtocolVersion} \
    } \
)
/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:write-attribute(name=statistics-enabled,value=true)

echo ** Update distributed-cache offlineSessions element **
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions/store=remote:add( \
    passivation=false, \
    fetch-state=false, \
    purge=false, \
    preload=false, \
    shared=true, \
    remote-servers=["remote-cache"], \
    cache=offlineSessions, \
    properties={ \
        rawValues=true, \
        marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \
        protocolVersion=${keycloak.connectionsInfinispan.hotrodProtocolVersion} \
    } \
)
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions:write-attribute(name=statistics-enabled,value=true)

echo ** Update distributed-cache clientSessions element **
/subsystem=infinispan/cache-container=keycloak/distributed-cache=clientSessions/store=remote:add( \
    passivation=false, \
    fetch-state=false, \
    purge=false, \
    preload=false, \
    shared=true, \
    remote-servers=["remote-cache"], \
    cache=clientSessions, \
    properties={ \
        rawValues=true, \
        marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \
        protocolVersion=${keycloak.connectionsInfinispan.hotrodProtocolVersion} \
    } \
)
/subsystem=infinispan/cache-container=keycloak/distributed-cache=clientSessions:write-attribute(name=statistics-enabled,value=true)

echo ** Update distributed-cache offlineClientSessions element **
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineClientSessions/store=remote:add( \
    passivation=false, \
    fetch-state=false, \
    purge=false, \
    preload=false, \
    shared=true, \
    remote-servers=["remote-cache"], \
    cache=offlineClientSessions, \
    properties={ \
        rawValues=true, \
        marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \
        protocolVersion=${keycloak.connectionsInfinispan.hotrodProtocolVersion} \
    } \
)
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineClientSessions:write-attribute(name=statistics-enabled,value=true)

echo ** Update distributed-cache loginFailures element **
/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures/store=remote:add( \
    passivation=false, \
    fetch-state=false, \
    purge=false, \
    preload=false, \
    shared=true, \
    remote-servers=["remote-cache"], \
    cache=loginFailures, \
    properties={ \
        rawValues=true, \
        marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \
        protocolVersion=${keycloak.connectionsInfinispan.hotrodProtocolVersion} \
    } \
)
/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures:write-attribute(name=statistics-enabled,value=true)

echo ** Update distributed-cache actionTokens element **
/subsystem=infinispan/cache-container=keycloak/distributed-cache=actionTokens/store=remote:add( \
    passivation=false, \
    fetch-state=false, \
    purge=false, \
    preload=false, \
    shared=true, \
    cache=actionTokens, \
    remote-servers=["remote-cache"], \
    properties={ \
        rawValues=true, \
        marshaller=org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory, \
        protocolVersion=${keycloak.connectionsInfinispan.hotrodProtocolVersion} \
    } \
)
/subsystem=infinispan/cache-container=keycloak/distributed-cache=actionTokens:write-attribute(name=statistics-enabled,value=true)

echo ** Update distributed-cache authenticationSessions element **
/subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions:write-attribute(name=statistics-enabled,value=true)

echo *** Update undertow subsystem ***
/subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=proxy-address-forwarding,value=true)

run-batch
stop-embedded-server

JAVA_OPTS Keycloak HotRod: remote.cache.host, remote.cache.port jboss.site.name.

— , (Kubernetes, DevOps, Docker, Ansible, Ceph, SRE)

Launching Keycloak in HA mode on Kubernetes

Introduction and main ideas

Keycloak

Keycloak

ip multicast

Infinispan

More articles: