Skip to main content

Connecting to Apache Kafka® in Aiven

In this tutorial​

Nussknacker allows creating decision scenarios on real-time data. As one way to ingest your data, you can connect directly to your Kafka and Schema Registry to read events from and write to your topics.

In this tutorial, we will explain how to connect to the Aiven from the Nussknacker Cloud and use a managed Apache Kafka as the input and output from Nussknacker.

Prerequisites​

We assume that:

  • You have already created the Aiven Kafka cluster
  • You have already created a Nussknacker Cloud account

Setting up Aiven​

  1. Go to Aiven console and navigate to Kafka cluster console.
  2. Navigate to 'Connection information' and choose SASL as authentication method (while it's possible to authenticate using client certificates, it's a bit more complex)
  3. You'll need data from Apache Kafka and Schema Registry sections, as shown below:

 

Setting up Nussknacker Cloud​

  1. Go to Nussknacker Cloud

  2. Click Create instance on the left panel

  3. Choose your instance name or use autogenerated value

  4. Choose Existing Kafka cluster

  5. Fill in the following fields

    • Properties:
    {
      "bootstrap.servers": "xxx-project.aivencloud.com:18234",

      "security.protocol": "SASL_SSL",
      "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"avnadmin\" password=\"xxxxx\";",
      "sasl.mechanism": "PLAIN",
      "ssl.truststore.type": "PEM",
      "ssl.truststore.location": "/certs/client.truststore",

      "schema.registry.url": "https://xxx-project.aivencloud.com:18234",
      "basic.auth.credentials.source": "USER_INFO",
      "basic.auth.user.info": "avnadmin:xxxx"
    }

    Where:

    • bootstrap.servers should be Service URI from Apache Kafka tab
    • username should be User (avnadmin by default) from Apache Kafka tab
    • password should be Password from Apache Kafka tab
    • schema.registry.url should contain Service URI from Schema Registry tab, without user and password
    • basic.auth.user.info should contain User and Password from Schema Registry tab, separated with :
    • Truststore: copy CA Certificate from Apache Kafka tab

  6. Click submit button

  7. Your Nussknacker instance connected to Aiven Kafka will be created within a couple of minutes

Aiven configuration

Creating a topic and a schema​

  1. In the Aiven Kafka service go to Topics tab
  2. Click the Add topic button and fill the needed fields
  3. Your topic is created, now lets define a data contract. Go to Schemas` tab
  4. Click the Create schema button, remember that Subject should be equal to created topic name plus -value suffix
  5. Choose your schema type, for now, Avro and JSON Schema are supported in Nussknacker Cloud
  6. Enter your schema
  7. Click the Create button

 

Creating a scenario​

  1. Go to Nussknacker Cloud management
  2. Open your running Nussknacker instance with the Open instance button
  3. Create your first scenario with the Create scenario button
    1. Choose the preferred scenario name
  4. Drag and drop the Kafka source from the left panel
  5. You should see your topic on the Topic drop-down list
  6. Add custom logic like filtering, sending to some other Kafka sink
  7. Press deploy button on the left
  8. Wait a while until scenario will be deployed

nussknackerScenario.png

Sending messages to and receiving messages from Nussknacker scenario​

You can read about messaging with Nussknacker in the documentation.