Connecting to Azure Event Hubs®

In this tutorial

Nussknacker allows creating decision scenarios on real-time data. As one way to ingest your data, you can connect directly to your event broker and Schema Registry to read events from and write to your topics.

In this tutorial, we will explain how to connect to the Azure Event Hubs from the Nussknacker Cloud and use Event Hubs as the input and output from Nussknacker.

Prerequisites

We assume that:

• You have already created the Azure account
• You have created Event Hubs namespace. To do it, you can follow the instruction available at Azure site
• You have enabled at least "Standard" subscription offer option for Event Hubs - it is required to have "Apache Kafka" and "Schema Registry" features enabled. See Azure's pricing option comparison for details
• You have read Apache Kafka and Azure Event Hubs - the most important thing to know is that Azure's "Event Hubs" are similar concepts to Kafka's "Topics"

Authenticating Nussknacker Cloud to access Azure Schema Registry

1. Go to Azure portal
2. Register application (you can call it e.g. "nussknacker-cloud") in Azure's Active Director. To do that, follow the instruction available on Azure site
3. Add a client secret into registered application. To do that, follow the instruction available on Azure site
4. Assign Schema Registry Reader role to your application. To do that follow the instruction available on Azure site
5. Copy information from application: Application (client) ID, Directory (tenant) ID and Client secret value:

Authenticating Nussknacker Cloud to access Azure Event Hubs

1. Go to Azure portal
2. Navigate to your Event Hubs namespace
3. Create Shared Access Policy rule (you can call it e.g. "NussknackerCloud"). It should have "Send" and "Listen" grant. Optionally add "Manage" grant if you want Nussknacker to create "nussknacker-errors" event hub by itself (this event hub will be used for Exception handling purpose). You can read more about SAS Polices at Azure site.
4. Copy the Connection string–primary key:

Creating Schema group in Azure Schema Registry

1. Go to Azure portal
2. Navigate to your Event Hubs namespace
3. Navigate to your Schema Registry Entity
4. Click + Schema Group button
5. Fill the form
   • Name should be the name of your group - Nussknacker will have access only to schemas inside this group
   • Serialization Type should be Avro - Nussknacker currently handle only Avro schemas for Azure Event Hubs. But still you can use JSON messages with it. See the instruction below.
   • Compatibility you can leave the default value (None) but for production usage it is good to ensure at least Backward compatibility

Setting up Nussknacker Cloud

1. Go to Nussknacker Cloud

2. Click Create instance on the left panel

3. Choose your instance name or use autogenerated value

4. Choose Existing Kafka cluster

5. Fill in the following fields

   • Properties:

     {
         "bootstrap.servers": "xxx.servicebus.windows.net:9093",
         "security.protocol": "SASL_SSL",
         "sasl.mechanism": "PLAIN",
         "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"Endpoint=sb://xxx.servicebus.windows.net/;SharedAccessKeyName=xxx;SharedAccessKey=xxx\";",
         "request.timeout.ms": "60000",
     
         "schema.registry.url": "https://xxx.servicebus.windows.net",
         "schema.group": "xxx",
         "azure.client.id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
         "azure.tenant.id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
         "azure.client.secret": "xxx"
     }

     Where:

     • bootstrap.servers should be in format xxx.servicebus.windows.net:9093 where xxx is your Event Hubs namespace
     • sasl.jaas.config should be in format as above
       • username should be $ConnectionString
       • password should be the Connection string–primary key from the SAS Policy created before
     • schema.registry.url should be in format https://xxx.servicebus.windows.net:9093 where xxx is your Event Hubs namespace
     • schema.group should be the name of the Schema Group crated before
     • azure.client.id should be the Application (client) ID of the registered App
     • azure.tenant.id should be the Directory (tenant) ID copied from the registered App's overview
     • azure.client.secret should be the Client secret value copied from the registered App's overview

   • Truststore and Keystore should be left empty

   • Format of messages on Kafka topics if you use the default Depending on schema type option, you will be able to communicate with Nussknacker sources/sinks only using binary, Avro format, because JSON schemas for Azure are currently not supported yet. If you want to communicate using JSON, you can switch this option to Always JSON format which will cause that even for Avro schemas, JSON messages format will be used

6. Click submit button

7. Your Nussknacker instance connected to Azure Event Hubs will be created within a couple of minutes

Creating an Event Hub and schema

1. Go to Azure portal
2. Navigate to your Event Hubs namespace
3. Navigate to your Event Hubs Entity
4. Click + Event Hub button
5. Set a name and optionally additional parameters like Partition count and Retention policy
6. Click Review + create
7. Click Create
8. Navigate to your Schema Registry Entity
9. Enter into previously created Schema group
10. Click + Schema button
11. Set a name and a schema content. Remember that the name should be in format CamelCasedTopicNameValue. Read more about how schemas are associated with even hubs in the documentation

 

Creating and deploying scenario

1. Go to Nussknacker Cloud management
2. Open your running Nussknacker instance with the Open instance button
3. Create your first scenario with the Create scenario button
   1. Choose the preferred scenario name
4. Drag and drop the Kafka source from the left panel
5. You should see your topic on the Topic drop-down list
6. Add custom logic like filtering, sending to some other Kafka sink
7. Press deploy button on the left
8. Wait a while until scenario will be deployed

Sending messages to and receiving messages from Nussknacker scenario

You can read about messaging with Nussknacker in the documentation. One of the easiest options is to use kafka-console-consumer.sh and kafka-console-producer.sh scripts. Remember that this method will work with plain text communication which will be possible after switching Format of messages on Kafka topics into Always JSON format in instance parameters.

To communicate using Avro binary format, check the Azure Schema Registry sample