Introduction to integrating Google Firebase with Kafka Connect and Nussknacker

This blog post will describe how to integrate Google Firebase with Nussknacker with the help of Kafka Connect.

Introduction

Google Firebase is a popular backend-as-a-service platform that provides a wide range of tools for developing and deploying mobile and web applications. By integrating Firebase with Kafka Connect and Nussknacker, you can easily stream data from Firebase to Nussknacker for real-time processing and analysis.

Benefits of using Firebase as a data source

Using Firebase as a data source for Nussknacker has many benefits. Firebase provides a flexible and scalable NoSQL database that can handle large amounts of data. It also provides easy-to-use APIs for accessing and manipulating data, as well as built-in security and authentication features.

Setting up Kafka Connect to connect to Firebase

To set up Kafka Connect, you will need to install and configure a Kafka Connect cluster. This process involves setting up a Zookeeper cluster and a Kafka cluster, as well as configuring the Kafka Connect workers and connectors. Once you have set up your Kafka Connect cluster, you can start configuring your Firebase connector. We will use the Confluent Firebase Connector available here https://docs.confluent.io/kafka-connectors/firebase/current/source/overview.html#google-firebase-source-connector-for-cp

*for production deployments you will need to add you Confluent licence to the below configuration

The basic configuration looks very straightforward.

After creating a connector with the above configuration all the integration should be working.

We go to the Google Firebase console to create some initial records.

For convenience, we can import a file with our data in JSON format.

After that, we see that our Connector has done the job.

There are schemas and topics created in Kafka and Schema Registry.

Now we go to the Nussknacker Designer. We create a sample scenario.

After we drag-and-drop Kafka source on the scenario we can see that there are new topics available

We can add some business logic in the filter expression

Real-world use cases

Integrating Firebase with Nussknacker and Kafka Connect can be used in various real-world applications such as real-time analytics, event-driven architectures, and data pipelines. For example, you can use Firebase to collect sensor data from IoT devices and stream it to Nussknacker for real-time analysis and visualisation. 

Another use case for Firebase integration is handling business rules for incoming Firebase events. Based on the conditions defined inside a Nussknacker scenario users can be notified or not. 

Conclusion

In the example, I showed how Kafka Connect can be used to integrate Nussknacker with Google Firebase. Thanks to Kafka Connect’s schemas autoregistration feature Nussknacker is capable of providing autocompletion and validations on the data. Nussknacker can be used to inject additional business logic for the Firebase events and is easily integrated with Firebase subscriptions. You can run the above example by yourself, the repo with the source code is available here https://github.com/TouK/nussknacker-firebase-example