Build Event-Driven Applications With Knative and Crossplane

5 min readApr 14, 2021

By Paolo Dettori, Aleksander Slominski, Ansu Varghese, Lionel Villard

In recent years, Kubernetes has emerged as the de facto standard for building cloud native applications. Although most use cases for Kubernetes have been around container management and orchestration, more recently, projects such as Crossplane have been extending the Kubernetes control plane for managing infrastructure services and providing developer-oriented abstractions to consume those services.

Knative provides a great set of core abstractions for serverless and event driven computing. Knative includes two main components: serving and eventing (each can be setup and run independently of the other). All of the custom objects to support these components are defined as Custom Resource Definitions (CRDs) to extend and build upon Kubernetes APIs, as we will see in the Kafka example below. Knative Serving is important for deploying serverless applications with additional capabilities to autoscale your deployments up and down, to manage network traffic routes to your serverless workloads, and to maintain history of your configuration changes, making updates as necessary. Knative Eventing provides the building blocks to easily support your event driven services (both producers and consumers of events), such as channels, subscriptions, brokers, triggers etc. Together, these Knative components provide a framework to simplify the user development experience on the cloud.

In this article, we will illustrate how it is possible to complement Knative Eventing with Crossplane to easily provision a cloud-managed Kafka event source for your application. We will be using the Crossplane provider for IBM Cloud and the managed Kafka service on IBM Cloud (Event Streams).

Getting Started

To get started with Crossplane, follow the installation instructions for Self-Hosted Crossplane on a kind cluster. Make sure the kind cluster is a Kubernetes version supported by Knative; e.g. Knative v0.21.0 requires a Kubernetes cluster v1.17 or newer. Instead of provisioning the Kubernetes version suggested in the above instructions we recommend using a more recent Kubernetes distribution:

kind create cluster --image kindest/node:v1.20.2 --name kncross --wait 5m

Once Crossplane and the Crossplane CLI are installed, install the latest stable version of the Crossplane provider for IBM Cloud following these instructions. To install Crossplane and the IBM Cloud provider in OpenShift, follow these instructions. Note that in order to use the provider, you will need to sign up for a free IBM Cloud account.

To install Knative Eventing for the scope of this blog, we recommend installing Knative Eventing using YAML files and the KafkaSource component. You will not need to install the optional default channel and broker.

Before moving to the next steps we need to create a new namespace for the app:

kubectl create ns knapp

Creating the Event Streams Service on IBM Cloud

The Crossplane provider for IBM Cloud project supports the provisioning of managed services on IBM public cloud. The provider templates for this blog can be accessed by cloning the following project in a directory of your choice:

git clone https://github.com/pdettori/crossplane-knative-examples.gitcd crossplane-knative-examples

Crossplane brings the power of Infrastructure as Code (IaC) to Kubernetes. The example file eventing1/resource-instance-eventstreams.yaml contains the declarative definition for provisioning a Kafka instance on IBM Cloud:

The declaration above provisions a “lite” (free) instance of Kafka on the “us-south” region in IBM Cloud. Apply the resource definition with the command:

kubectl apply -f eventing1/resource-instance-eventstreams.yaml

we can check on the status of the provisioning with the commands:

kubectl describe resourceinstance myes
kubectl get resourceinstance myes

after the instance is provisioned, you should see an output similar to the following:

NAME STATUS STATE CLASS AGE
myes active 45s

The Kafka instance has been created and is available on IBM Cloud, but you still need a secret with credentials to connect to it. We will be following similar steps to create a set of credentials for the instance:

kubectl apply -f eventing1/resource-key-eventstream.yaml
kubectl describe resourcekey myes-creds
kubectl get resourcekey myes-creds

Once the resource key gets created, a secret containing the credentials for connecting to Kafka is available:

kubectl get secrets -n knapp myes
NAME TYPE DATA AGE
myes connection.crossplane.io/v1alpha1 17 1m

We can then extract from the secret the connection info we need to connect to the Event Streams service:

BROKER=$(kubectl get secret myes -n knapp -o=jsonpath="{.data['kafka_brokers_sasl\.0']}" | base64 -D)
USER=$(kubectl get secrets myes -n knapp -o=jsonpath="{.data.user}" | base64 -D)
PASSWORD=$(kubectl get secrets myes -n knapp -o=jsonpath="{.data.password}" | base64 -D)

Create Kafka Topic

To create a Kafka topic, we will first create a ConfigMap with the security configuration info required to connect to the Event Streams service:

We can now create a container with all the Kafka console tools:

Make sure that the pod is started:

kubectl get pods -n knapp
NAME          READY   STATUS    RESTARTS   AGE
kafka-tools   1/1     Running   0          64s

Then, to create the topic fruits for this app, we can run the command:

kubectl exec -it kafka-tools -n knapp -- bin/kafka-topics.sh --bootstrap-server=$BROKER --command-config /etc/config/producer.config --create --topic fruits

Create a KafkaSource

Run the following command to create a KafkaSource:

You can check it has been successfully created with the command:

kubectl get kafkasources.sources.knative.dev -n knapp

It should provide an output similar to the following:

kubectl get kafkasources.sources.knative.dev -n knapp
NAME     TOPICS       BOOTSTRAPSERVERS                                                                     READY   REASON   AGE
fruits   ["fruits"]   ["broker-0-0r9l4j279lqrv08z.kafka.svc08.us-south.eventstreams.cloud.ibm.com:9093"]   True             23s

Run Sample Application

It is time now to deploy our Knative Eventing sample application. The purpose of this application is to display data stored in the Kafka topic named fruits:

kubectl apply -f eventing1/event-display.yaml

Make sure the deployment is up and running:

kubectl get deployments.apps event-display -n knapp
NAME READY UP-TO-DATE AVAILABLE AGE
event-display 1/1 1 1 47s

Finally, let’s send some data to the fruits topic. We can use the Kafka console tool container we created previously to send some data:

kubectl exec -it kafka-tools -n knapp -- bin/kafka-console-producer.sh --broker-list $BROKER --producer.config /etc/config/producer.config --topic fruits

type Hello and then open another terminal and the run the following command to look at the event-display log:

kubectl logs -n knapp -l app=event-display --tail=-1

the output should resemble something like the following:

☁️  cloudevents.Event
Validation: valid
Context Attributes,
  specversion: 1.0
  type: dev.knative.kafka.event
  source: /apis/v1/namespaces/knapp/kafkasources/fruits#fruits
  subject: partition:0#0
  id: partition:0/offset:0
  time: 2021-04-13T01:00:00.283Z
Extensions,
  traceparent: 00-fae5d19baa348ff6a2462da683677672-dc70bd0cf67c67c7-00
Data,
  Hello

That’s it! We provisioned a managed Kafka instance on IBM Cloud with Crossplane, bound it to a Knative Eventing KafkaSource, and emitted events to a Kafka topic that have then been delivered to our sample Knative event-display app.

Cleaning Up

Delete your Event Streams key and instance on IBM Cloud by running the following commands:

kubectl delete resourcekey myes-creds
kubectl delete resourceinstance myes

Then, you can delete your kind cluster with the command:

kind delete cluster --name kncross

Conclusion

In this post, we have shown how to combine the power of Knative Eventing and Crossplane to build a simple Knative application backed by a Kafka instance running on IBM Cloud. Crossplane and Knative provide powerful abstraction that will help you to turbocharge your event driven application.

Acknowledgements

We would like to thank Paul Castro for his valuable help with this post and testing the eventing app.