Introduction of New Data Transfer Method --- Apache Kafka

Author: Spencer

2022-04-06 13:47

GoodWe provides a new data transfer method using Apache Kafka system. This document will introduce you the pros and cons of this methods and how to get access to it. 

 

What is Apache Kafka?

Apache Kafka is an open source event steaming distributed system that working in a mode of message publish and subscribe. The main feature that distinguishes it from  traditional Open API is that user’s data acquisition changes from passive acquisition to active subscription to messages. Moreover, messages does not disappear after being transferred, but remains persistent for a period of time.

 

The main difference between Open API and Apache Kafka

The data forwarding method of Open API interface is that GoodWe takes the initiative to send the data to the Http address provided by the client, which is one-to-one and point-to-point. GoodWe is only responsible for forwarding message. In some cases, the customer has large amount of devices (e.g. more than 2,000 inverters). Due to the customer’s limited ability to receive the excessive amount of data, packet loss is prone to happen.

Figure 1 Open API

To solve this issue, GoodWe developed a new data transfer method based on Apache Kafka. This new publish and subscribe methods provide customer a new choice to avoid data packet loss because Kafka system can keep data for a certain time.

Figure 2 Apache Kafka

The operation logic of Kafka subscription is that Goode will publish the message to the Topic, and the user will consume (pull) this information by subscribing to the Topic. The message published to the Topic will be consumed by all users subscribed to that Topic. However, the data retention is limited, depending on the storage of server. GoodWe will take the initiative to send message to consumers, and customers can choose to consume (pull) message if they find packet loss.

Figure 3 Operation Logic of Kafka

 

How to apply for Kafka Data Transfer?

1. Distributors should provide their distributor code (The devices they purchased should be shipped under this

distributor code. )

2. Based on the application, GoodWe create a Topic named after their distributor code.

3. GoodWe will provide the Kafka subscription address including topic, account, and password.

 

Why choosing Kafka?

1. Shorter development time

2. Simpler method--- customers only need to provide the distributor code

3. Effectively solve the problem of packet loss.

However, the data transfer method of Apache Kafka requires customers to have a certain level of data processing capability that they are able to use the platform of Kafka.

Download