Connecting to Redpanda's Apache Kafka®-compatible streaming platform

In this tutorial

Nussknacker allows creating decision scenarios on real-time data. As one way to ingest your data, you can connect directly to your Redpanda's streaming platform to read events from and write to your topics.

In this tutorial, we will explain how to connect to the Redpanda Cloud from the Nussknacker Cloud and use a managed streaming platform as the input and output from Nussknacker.

Prerequisites

We assume that:

• You are already a Redpanda customer with a Cloud environment and you have created a cluster.
• You have already created a Nussknacker Cloud account.

Setting up Redpanda Cloud

1. Go to your Redpanda Cloud home page
2. Select a namespace or create a new one and open it
3. You will see a list of clusters for this namespace
4. Select your cluster or create a new one and open it
5. Click Security from menu on the left
6. Your principal list will appear
7. If you don't have any principal you should create a new one with a Create User action
8. Open your user and edit ACL for that user accordingly

Setting up Nussknacker Cloud

1. Go to Nussknacker Cloud
2. Click Create instance on the left panel
3. Choose your instance name or use autogenerated value
4. Choose Existing Kafka cluster
5. Please copy and edit the following JSON snippet and paste it into Properties field:

   {
      "bootstrap.servers": "<Redpanda Kafka Bootstrap server URL>",
      "security.protocol": "SASL_SSL",
      "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"<username>\" password=\"<password>\";",
      "sasl.mechanism": "SCRAM-SHA-256",
      "schema.registry.url": "<Redpanda Schema Registry HTTP endoint URL>",
      "basic.auth.credentials.source": "USER_INFO",
      "basic.auth.user.info": "<username>:<password>>"
   }
   
6. Click the Submit button
7. Your Nussknacker instance connected to Redpanda streaming platform will be created within a couple of minutes

Creating a topic

1. In the Redpanda Cloud cluster overview go to Topics
2. Click the Create topic button
3. Type your topic name, e.g. "sensor".
   1. Set the number of partitions
4. Click Create
5. Your topic is created, now let`s define a data contract - schema.

Creating a schema

To use topic as a source or a sink in Nussknacker scenarios you must create a schema for it. By convention schema for a topic called source is named sensor-value. To create it, you must provide a username and a password with a Basic Authentication. Below you can find an example cURL command to create a schema for sensor-value:

curl -X POST --location "https://schema-registry-aaddd3cb.cglu57gsjns3qo0hrepg.fmc.prd.cloud.redpanda.com:30081/subjects/sensor-value/versions" \
    --basic --user "username:password" \
    -H "Content-Type: application/vnd.schemaregistry.v1+json" \
    -d '{"schema": \"{\"type\":\"record\",\"name\":\"sensor_sample\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},{\"name\":\"identifier\",\"type\":\"string\",\"logicalType\":\"uuid\"},{\"name\":\"value\",\"type\":\"double\"}]}"}'

If your request is successful it should return a schema identifier:

{
  "id": 1
}

Creating a scenario

1. Go to Nussknacker Cloud management
2. Open your running Nussknacker instance with the Open instance button
3. Create your first scenario with the Create scenario button
   1. Choose the preferred scenario name
4. Drag and drop the Kafka source from the left panel
5. You should see your topic on the Topic drop-down list
6. Add custom logic like filtering, sending to some other Kafka sink
7. Press deploy button on the left
8. Wait a while until scenario will be deployed

Sending messages to and receiving messages from Nussknacker scenario

You can read about messaging with Nussknacker in the documentation.