Hosted Control Plane and tenant networking¶

Challenge: running a hosted cluster in a different tenant network segment or VLAN without wide-open access from the tenant segment to the management segment.

An hosted cluster can devide into two parts: control plane and data plan aka worker nodes. For there parts there different technics to place it into a tenant network:

Exposing hosted control plane into tenant network¶

... it is fairly hard. The following components must be reachable from workers and clients in/from tenant network and beyond:

• API Server
• OAuth
• Konnectivity
• Ignition

Here is a summary of common publishing options for these components:

For this proof of concept, endpoints are exposed as follows:

• API Server: LoadBalancer (fronted by external api-lb in the tenant segment; see below)
• OAuth, Konnectivity, Ignition: Route via a dedicated ingress controller shard on the hub, fronted by external ingress-shared-lb with VIPs/DNS in the tenant segment

Exposing components via a dedicated router shard¶

Use a dedicated OpenShift Ingress Controller shard on the hub so only the hosted-cluster control-plane Routes are served by that shard. Tenant clients resolve OAuth, Konnectivity, and Ignition hostnames to ingress-shared-lb, which forwards to the shard’s NodePorts on the management network.

Place an external load balancer in front of that shard (for example F5 BIG-IP or NetScaler) that can reach the hub’s management network and present stable tenant-facing VIPs or addresses.

Exposing hosted worker nodes into netant network¶

Worker nodes (VM's) of the hosted cluster are straightforward: attach them to the tenant network segment (DHCP or equivalent addressing is required).

Proof of concept environment overview¶

Router between Mgmt and Tenant-A¶

VyOS acts as router and firewall between the management and the Tenant-A network. Restrict lateral traffic between the two segments (no full mesh); allow only what you need (for example DNS to resolvers, default route or NAT for internet egress). Hosted-cluster control-plane traffic from tenant nodes should flow to the external load balancer VIPs in the tenant segment (not directly into arbitrary management subnets).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

configure

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth1 address '192.168.203.1/24'

set service ssh
set system host-name 'router-2003'
set system name-server '10.32.96.1'
set system name-server '10.32.96.31'

# Recommend to commit,save and reboot at this point
commit
save
exit
reboot

set firewall group address-group ALLOWED-IPS address '10.32.96.1'
set firewall group address-group ALLOWED-IPS address '10.32.96.31'
set firewall group address-group ALLOWED-IPS address '10.32.111.254'
set firewall ipv4 forward filter rule 49 action 'accept'
set firewall ipv4 forward filter rule 49 description 'Allow IPs'
set firewall ipv4 forward filter rule 49 destination group address-group 'ALLOWED-IPS'
set firewall ipv4 forward filter rule 50 action 'drop'
set firewall ipv4 forward filter rule 50 description 'Drop enire coe lab'
set firewall ipv4 forward filter rule 50 destination address '10.32.96.0/20'

set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 source address '192.168.203.0/24'
set nat source rule 100 translation address 'masquerade'
set service dhcp-server listen-interface 'eth1'
set service dhcp-server shared-network-name coe-2003 authoritative
set service dhcp-server shared-network-name coe-2003 subnet 192.168.203.0/24 option default-router '192.168.203.1'
set service dhcp-server shared-network-name coe-2003 subnet 192.168.203.0/24 option name-server '10.32.96.1'
set service dhcp-server shared-network-name coe-2003 subnet 192.168.203.0/24 range 1 start '192.168.203.100'
set service dhcp-server shared-network-name coe-2003 subnet 192.168.203.0/24 range 1 stop '192.168.203.200'
set service dhcp-server shared-network-name coe-2003 subnet 192.168.203.0/24 subnet-id '1'

# Don't forget to commit and write
commit
save

1

sshuttle -r vyos@<IP of VyOS VM> 192.168.203.0/24

Deployment sequence (reference)¶

Three external load balancers appear in this write-up; keep their roles distinct:

Suggested order:

1. Hub ingress shard + ingress-shared-lb + DNS for the three control-plane hostnames: OAuth, Konnectivity, and Ignition
2. Apply HostedCluster and NodePool.
3. Deploy external load balancer for the Hosted-Cluster API api-lb + API DNS. Based on the NodePorts for the api kubernetes Service, located in hub cluster
4. Deploy external load balancer for the Hosted-Cluster Ingress ingress-lb + wildcard apps DNS. Based on the NodePorts of the ingress kubernetes service, located in hosted cluster.

Hub ingress shard + ingress-shared-lb¶

• 2.3.4. Ingress sharding in OpenShift Container Platform
• 3.1.3.8.1. Example load balancer configuration for user-provisioned clusters

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: tenant-a
  namespace: openshift-ingress-operator
spec:
  domain: tenant-a.coe.muc.redhat.com

  endpointPublishingStrategy:
    type: NodePortService
  namespaceSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values:
          - ingress-test
          - clusters-tenant-a

1
2
3

% oc get svc -n openshift-ingress router-nodeport-tenant-a
NAME                       TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                                     AGE
router-nodeport-tenant-a   NodePort   172.30.141.209   <none>        80:32460/TCP,443:32488/TCP,1936:32095/TCP   106s

The ingress shard load balancer is an RHEL 9 host running HAProxy (external load balancer ingress-shared-lb).

• Install HAProxy: dnf install haproxy
• Configure SELinux: setsebool -P haproxy_connect_any 1
• Apply the example haproxy configuration (update ports to match your NodePort service)
• Enable and start HAProxy: systemctl enable --now haproxy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

global
  log         127.0.0.1 local2
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  daemon
defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000

listen ingress-router-443
  bind *:443
  mode tcp
  balance source
  server ucs-blade-server-5 10.32.96.105:32488 check inter 1s
  server ucs-blade-server-6 10.32.96.106:32488 check inter 1s        
  server ucs-blade-server-7 10.32.96.107:32488 check inter 1s
  server ucs-blade-server-8 10.32.96.108:32488 check inter 1s

listen ingress-router-80
  bind *:80
  mode tcp
  balance source
  server ucs-blade-server-5 10.32.96.105:32460 check inter 1s
  server ucs-blade-server-6 10.32.96.106:32460 check inter 1s        
  server ucs-blade-server-7 10.32.96.107:32460 check inter 1s
  server ucs-blade-server-8 10.32.96.108:32460 check inter 1s

Add DNS records

1
2
3

konnectivity.tenant-a.coe.muc.redhat.com.       IN A 192.168.203.111
oauth.tenant-a.coe.muc.redhat.com.              IN A 192.168.203.111
ignition.tenant-a.coe.muc.redhat.com.           IN A 192.168.203.111

Apply PullSecret, SSH-Key¶

1
2
3

oc create secret docker-registry pullsecret-cluster-tenant-a \
  -n clusters \
  --from-file=${HOME}/redhat-pull-secret.json

1
2
3

oc create secret generic sshkey-cluster-tenant-a \
  -n clusters \
  --from-file=id_rsa.pub=${HOME}/.ssh/id_ed25519.pub

Apply HostedCluster and NodePool¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: 'tenant-a'
  namespace: 'clusters'
  labels:
    "cluster.open-cluster-management.io/clusterset": 'default'
spec:
  configuration:
    ingress:
      appsDomain: apps.tenant-a.coe.muc.redhat.com  # (1)
      domain: ''
      loadBalancer:
        platform:
          type: ''
  channel: fast-4.21
  etcd:
    managed:
      storage:
        persistentVolume:
          size: 8Gi
        type: PersistentVolume
    managementType: Managed
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.21.11-multi
  pullSecret:
    name: pullsecret-cluster-tenant-a
  sshKey:
    name: sshkey-cluster-tenant-a
  networking:
    clusterNetwork:
      - cidr: 10.132.0.0/14
    serviceNetwork:
      - cidr: 172.31.0.0/16
    networkType: OVNKubernetes
  controllerAvailabilityPolicy: SingleReplica
  infrastructureAvailabilityPolicy: SingleReplica
  platform:
    type: KubeVirt
    kubevirt:
      baseDomainPassthrough: false
  infraID: 'tenant-a'
  services:
    - service: APIServer
      servicePublishingStrategy:
        type: LoadBalancer
        loadBalancer:
          hostname: api.tenant-a.coe.muc.redhat.com  # (2)
    - service: OAuthServer
      servicePublishingStrategy:
        type: Route
        route:
          hostname: oauth.tenant-a.coe.muc.redhat.com  # (3)
    - service: OIDC
      servicePublishingStrategy:
        type: Route
    - service: Konnectivity
      servicePublishingStrategy:
        type: Route
        route:
          hostname: konnectivity.tenant-a.coe.muc.redhat.com  # (4)
    - service: Ignition
      servicePublishingStrategy:
        type: Route
        route:
          hostname: ignition.tenant-a.coe.muc.redhat.com  # (5)

  operatorConfiguration:  # (6)
    ingressOperator:
      endpointPublishingStrategy:
        type: HostNetwork
        hostNetwork:
          httpPort: 80
          httpsPort: 443
          protocol: TCP

1. appsDomain: resolve names under apps.tenant-a.coe.muc.redhat.com to ingress-lb (hosted cluster ingress), not the hub shard.
2. API server loadBalancer.hostname: resolve to api-lb, which forwards to the APIServer publishing target on the hub.
3. OAuth route.hostname: resolve to ingress-shared-lb (hub dedicated shard).
4. Konnectivity route.hostname: resolve to ingress-shared-lb.
5. Ignition route.hostname: resolve to ingress-shared-lb.
6. Switch from Kubernetes Type LoadBalancer to just HostNetwork as endpointPublishingStrategy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  name: 'tenant-a'
  namespace: 'clusters'
spec:
  arch: amd64
  clusterName: 'tenant-a'
  replicas: 2
  management:
    autoRepair: false
    upgradeType: Replace
  platform:
    type: KubeVirt
    kubevirt:
      compute:
        cores: 2
        memory: 8Gi
      rootVolume:
        type: Persistent
        persistent:
          size: 32Gi
      additionalNetworks:
        - name: cudn-localnet1-2003  # (1)
      attachDefaultNetwork: false
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.21.11-multi

1. Attach NodePool VMs to the tenant segment using a user-defined network (UDN) localnet attachment (cudn-localnet1-2003 in this lab).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  name: cudn-localnet1-2003
spec:
  namespaceSelector:
    matchExpressions:
      - key: hypershift.openshift.io/hosted-control-plane
        operator: Exists
  network:
    localnet:
      ipam:
        mode: Disabled
      physicalNetworkName: localnet1
      role: Secondary
      vlan:
        access:
          id: 2003
        mode: Access
    topology: Localnet

Deploy external load balancer for the Hosted-Cluster API (api-lb)¶

Use an RHEL 9 virtual machine with HAProxy.

• Install HAProxy: dnf install haproxy
• Configure SELinux: setsebool -P haproxy_connect_any 1
• Apply the example haproxy configuration (update ports to match your environment)
• Enable and start HAProxy: systemctl enable --now haproxy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

global
  log         127.0.0.1 local2
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  daemon
defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000

listen api
  bind *:6443
  mode tcp
  balance source
  server ucs-blade-server-5 10.32.96.105:30918 check inter 1s
  server ucs-blade-server-6 10.32.96.106:30918 check inter 1s        
  server ucs-blade-server-7 10.32.96.107:30918 check inter 1s
  server ucs-blade-server-8 10.32.96.108:30918 check inter 1s

Add DNS record:

1

api.tenant-a.coe.muc.redhat.com.       IN A 192.168.203.<IP of VM>

Deploy external load balancer for Hosted-Cluster ingress (ingress-lb)¶

Use an RHEL 9 virtual machine with HAProxy.

• Install HAProxy: dnf install haproxy
• Configure SELinux: setsebool -P haproxy_connect_any 1
• Apply the example haproxy configuration (update ports to match your environment)
• Enable and start HAProxy: systemctl enable --now haproxy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

global
  log         127.0.0.1 local2
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  daemon
defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000

listen ingress-router-443
  bind *:443
  mode tcp
  balance source
  server tenant-a-gngj5-mfwp6 192.168.203.101:30190 check inter 1s
  server tenant-a-gngj5-rrbmv 192.168.203.102:30190 check inter 1s        

listen ingress-router-80
  bind *:80
  mode tcp
  balance source
  server tenant-a-gngj5-mfwp6 192.168.203.101:30282 check inter 1s
  server tenant-a-gngj5-rrbmv 192.168.203.102:30282 check inter 1s        

Add DNS record:

1

*.apps.tenant-a.coe.muc.redhat.com.       IN A 192.168.203.<IP of VM>

LoadBalancer Services in hosted clusters get IPs from the hub¶

When the hub cluster supports Kubernetes Services of type LoadBalancer (e.g. via MetalLB, a cloud provider CCM, or any other implementation), any such Service created inside the hosted cluster is fulfilled by the hub cluster's load-balancer implementation. The hosted cluster's control plane runs on the hub and inherits its configuration.

This means a tenant workload can unintentionally allocate an IP address from the hub's infrastructure network simply by creating a LoadBalancer Service.

There is currently no supported way to disable this behaviour. Tracking: RFE-7742

Reproducing the problem¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

% oc project service-type-loadbalancer
% oc apply -k 'https://github.com/openshift-examples/kustomize.git/components/simple-https?ref=2026-07-20'
% oc apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: simple-https-lb
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 8443
    protocol: TCP
    targetPort: 8443
  selector:
    app: simple-https
    deployment: simple-https
  sessionAffinity: None
  type: LoadBalancer
EOF

NodePort chicken-and-egg problem¶

The external HAProxy load balancers (api-lb, ingress-shared-lb, ingress-lb) in this setup forward traffic to Kubernetes Services of type NodePort. The problem: NodePort allocates a random high port (default range 30000–32767) that is only known after the Service is created. Every time the Service is recreated or the port changes, the HAProxy configuration must be updated manually.

This creates a chicken-and-egg situation especially for the API load balancer: you need the NodePort to configure HAProxy, but the hosted cluster needs a working API endpoint to become healthy.

Why an external LoadBalancer controller would solve this¶

An external load balancer that integrates with Kubernetes (i.e. watches Service objects and configures itself automatically) eliminates the manual NodePort tracking entirely. Such a controller:

• Reacts to Service creation/updates and provisions the correct backend configuration automatically
• Can expose Services into separate network segments (tenant networks) with its own VIP pool
• Removes the need for static HAProxy configs that break when ports change

Potential solutions¶

Any of these can watch the hub cluster for the relevant Services and automatically configure load balancing into the tenant network—without manual NodePort tracking or HAProxy reconfiguration.

ACM console URL bug¶

ACM displays an incorrect console URL for the hosted cluster. It shows https://console-openshift-console.apps.tenant-a.apps.ocp5.stormshift.coe.muc.redhat.com/ instead of the correct https://console-openshift-console.apps.tenant-a.coe.muc.redhat.com/. The extra .apps.ocp5.stormshift segment comes from the hub's ingress domain being appended.

Tracking: OCPBUGS-105612

Verions¶

Tested with:

2026-08-14 2026-05-12 Contributors: