Integrate Cloud Foundry with Kubernetes using the cf-operator and kubecf

kubecf Feb 04, 2020

If you work in the software industry, you have more then likely come across the terms IaaS, PaaS and SaaS. IaaS stands for infrastructure as a Service. Good examples of IaaS consist of the different cloud providers such as AWS and on-premise cloud such as Openstack. SaaS is a common way of providing software these days and basically means software as a service. PaaS as you can probably guess is platform as a service. Each of these "as a Service" models aim to simplify the process of delivering software for development teams by providing different levels of abstraction from the hardware as you can see below.

Cloud Foundry is a very popular PaaS offering in the enterprise market especially with companies who have on-premise workloads. In a previous life, I used to travel around to different companies designing and deploying on-premise Openstack cloud environments and during this time I noticed a common requirement emerging from the companies. They all wanted their cloud environments to integrate wih PCF (Pivotal Cloud Foundry). The move to Cloud Foundry is driven by the belief that it will speed up software delivery and improve the frequency of releases. According to Pivotal's website they have been able to help a customer increase their release frequency by 340% which is huge. They also advertise that T-Mobile experienced a 2/3 reduction in operations cost due to moving over to Pivotal Cloud Foundry. You can see why a lot of companies would see Cloud Foundry as the solution to all their problems with these kind of impressive numbers out there. (And I presume the VMware sales team are very persuasive)

Admittedly I haven't had much exposure to Cloud Foundry so far apart from rolling out Openstack clouds for PCF to sit on. I was interested in learning more about Cloud Foundry and in comparing the CF development flow versus a traditional development flow and a development flow based around Kubernetes. If you haven't noticed from the rest of my posts here, I am a bit of a fan of Kubernetes and the advantages that it brings. I don't agree however that Kubernetes is the right solution for everything. For a developer working on a simple UI a PaaS platform such as Cloud Foundry could greatly simplify his/her workflow. It would save them from having to learn all the inner workings of a Kubernetes cluster.

I started looking into the Cloud Foundry Developer certification as a way of gaining more familiarity with Cloud Foundry. In a lot of the courses for this certification they recommend using Pivotal Web Services to play around and get practice with Cloud Foundry. There is a free trial available at the moment up to $87 worth of credit. I am awkward though and thought to myself - "Cloud Foundry is open source so I should be able to run it at home". I don't have enough hardware at home for a full Openstack deployment but I do have a small Kubernetes cluster so I started looking into the possibility of running Cloud Foundry on top of Kubernetes. I came across a project on Github called kubecf which is developed by my old friends at SUSE. This project uses the cf-operator which is part of the Cloud Foundry Quarks project to deploy and integrate Cloud Foundry with Kubernetes. Quarks is a Cloud Foundry Foundation project focused on integrating CF with Kubernetes. kubecf is currently pre-release software so your mileage might vary but I have to say it worked well for me. It uses helm to deploy the CF services to the cluster so the deployment process is fairly easy.

At the time of writing this the latest release is v.0.2.0 so that is the one I went with. Make sure to check if their are any newer releases if you are going to try this yourself.

There are a couple of prerequisites before you can deploy this to you cluster

  • You have to be able to assign LoadBalancer service IPs. This should be OK if you are using some public cloud Kubernetes service such as EKS or GKE. If you are on a bare metal cluster you might have to look at something like MetalLB
  • You also need some persisted storage provider such as the nfs-client-provisioner

The first step of the deployment is to add the cf-operator to your cluster again using helm:

helm install cf-operator \
  --namespace cf-operator \
  --set "global.operator.watchNamespace=kubecf" \

Next you need to pull the values.yaml file from the kubecf helm chart and edit the values to match the configuration required for you environment such as updating the system domain. Once you have updated the values.yaml you can deploy kubecf:

helm install kubecf \
  --namespace kubecf \
  --values values.yaml \

This can take some time before all the CF pods are up and running especially if you have a small cluster. There are a good few pods deployed from this so I'd recommend giving them their own namespace.      

[brian@pinebookpro ~]$ kubectl get pods -n kubecf
NAME                                      READY   STATUS    RESTARTS   AGE
cf-operator-744cf8bb7d-tkchw              1/1     Running   0          25h
cf-operator-quarks-job-869889b886-859xs   1/1     Running   0          25h
kubecf-adapter-0                          4/4     Running   0          24h
kubecf-api-0                              15/15   Running   1          24h
kubecf-auctioneer-0                       4/4     Running   1          24h
kubecf-bosh-dns-59cd464989-lmh78          1/1     Running   0          24h
kubecf-bosh-dns-59cd464989-lvvn6          1/1     Running   0          24h
kubecf-cc-worker-0                        4/4     Running   0          24h
kubecf-credhub-0                          5/5     Running   0          24h
kubecf-database-0                         2/2     Running   0          24h
kubecf-diego-api-0                        6/6     Running   2          24h
kubecf-diego-cell-0                       7/7     Running   2          22h
kubecf-doppler-0                          9/9     Running   0          24h
kubecf-log-api-0                          7/7     Running   0          24h
kubecf-nats-0                             4/4     Running   0          24h
kubecf-router-0                           5/5     Running   0          24h
kubecf-routing-api-0                      4/4     Running   0          24h
kubecf-scheduler-0                        10/10   Running   1          24h
kubecf-singleton-blobstore-0              6/6     Running   0          24h
kubecf-tcp-router-0                       5/5     Running   0          24h
kubecf-uaa-0                              6/6     Running   0          24h

In the meantime you can check the IP address assigned to the kubecf-router-public LoadBalancer service. A number of DNS records have to be added to your DNS server for this kubecf-router-public IP address in order for users to connect to Cloud Foundry using the CLI. These include:

  • api.system-domain
  • login.system-domain
  • uaa.system-domain
  • doppler.system-domain

Once the CF pods are all running and the DNS records have been added, you should be able to log into Cloud Foundry using the cf CLI.

[brian ~]$ cf api --skip-ssl-validation
Setting api endpoint to

api endpoint:
api version:    2.144.0
Not logged in. Use 'cf login' or 'cf login --sso' to log in.

[brian ~]$ cf auth admin "${admin_pass}"
API endpoint:

Use 'cf target' to view or set your target org and space.

I ran a quick sanity check to make sure that the basic functionality was working. Firstly, I targeted the default system organisation and created a new space called "test". I then pulled a hello-world node-js app from Github and simply ran a "cf push" from within that repo.

[brian content-hello-node]$ cf target -o system
api endpoint:
api version:    2.144.0
user:           admin
org:            system
No space targeted, use 'cf target -s SPACE'

[brian@ content-hello-node]$ cf create-space test
Creating space test in org system as admin...

Assigning role SpaceManager to user admin in org system / space test as admin...

Assigning role SpaceDeveloper to user admin in org system / space test as admin...

TIP: Use 'cf target -o "system" -s "test"' to target new space

[brian content-hello-node]$ cf push test-node-app 
Pushing app test-node-app to org system / space test as admin...
Getting app info...
name:              test-node-app
requested state:   started
last uploaded:     Sat 01 Feb 01:11:17 GMT 2020
stack:             cflinuxfs3
buildpacks:        nodejs

type:            web
instances:       1/1
memory usage:    1024M
start command:   npm start
     state     since                  cpu    memory    disk      details
#0   running   2020-02-01T01:11:28Z   0.0%   0 of 1G   0 of 1G   

I find this project very interesting as it gives you the option of providing a more complete PaaS type offering to development teams while still having Kubernetes deployments for more complex applications. It would allow development teams to decide which of these best fit their service or application. Teams with basic applications could deploy by simply running a "cf push" which would inevitably lead to more frequent releases.

I haven't noticed any significant limitations of this setup so far. I have been able to carry out most of the tasks outlined in a CFCD certification course from Linux Academy so it has satisfied my requirement for a CFCD playground at home. Again your mileage may vary...

Brian Carey

Burrito Enthusiast | Coffee Drinker | Linux Nerd | Photographer | DevOps Engineer