We attended Cloud Field Day 11 (CFD11) last week and among the vendors at the show was Kasten by Veeam talking about their extensive Kubernetes data protection/DR/migration capabilities. All that was worthy of its own blog post but somewhere near the end of their session, they started discussing how they plan to backup Apache Kafka message traffic
We have discussed Kafka before (see our Data in motion post). For those who have read that post or are familiar with Kafka, you can ignore the Kafka primer section below and just for the record, I’m no Kafka expert.
Kafka is a massively scalable message bus and real time stream processing system. Messages or records come in from Kafka producers and are sent to Kafka consumers or subscribers for processing. Kafka supplies a number of guarantees one of which is that messages are processed once and only once.
Messages or records are key-value pairs that are generated continuously and are processed by Kafka apps. Message streams are split into topics.
Kafka connectors allow message traffic to be extracted/generated from external databases and other systems . Kafka stream processors use Kafka streaming primitives and stream flow graphs to construct applications that process unbounded, continuously updated data sets.
Topics in Kafka can have multi-producers and multi-subscribers. Each topic can be further split across partitions which can be processed in different servers or brokers in a Kafka cluster. It’s this partitioning that allows Kafka to scale from processing 1 message to millions of messages per second.
Consumers/subscribers can also be producers, so a message that comes in and is processed by a subscriber could create other messages on topics that need to be processed. Messages are placed on topic partitions in the order they are received.
Messages can be batched and added to a partition or not. Recently Kafka added support for sticky partitions. Normally messages are assigned to partitions based on key hashing but sometimes messages have null keys and in this case Kafka assigned them to partitions in a round robin fashion. Sticky partitioning strategy amends that approach to use batches of messages rather than single messages when adding null key messages to partitions.
Kafka essentially provides a realtime streaming message/event processing system that scales very well.
Data protection for Kafka streams without Kasten
This is our best guess as to what this looks like, so bear with us.
Sometime after messages are processed by a subscriber they can be sent to a logging system. If that system records those messages to external storage, they would be available to be copied off-line and backed up.
How far behind real time, Kafka logs happen to be can vary significantly based on the incoming/outgoing message traffic, resources available for logging and the overall throughput of your logging process/storage. Could it be an hour or two behind, possibly but I doubt it, could it be 1 second behind, also unlikely. Somewhere in between those two extremes seems reasonable
Log processing messages just like any topic could be partitioned. So any single log would only have messages on that partition. So there would need to be some post process to stitch all these log partitions together to get a consistent message stream.
In any case, there is potentially a wide gulf between the message data that is being processed and the logs which hold them. But then they need to be backed up as well so there’s another gap introduced between the backed up data what is happening in real time This is especially true the more messages that are being processed in your Kafka cluster.
(Possible) data protection for Kafka streams with Kasten
For starters, it wasn’t clear what Kasten presented was a current product offering, beta offering or just a germ of an idea. So bear that in mind. See the videos of their CFD11 session here for their take on this.
Kafka supports topic replication. What this means is that any topic can be replicated to other servers in the Kafka cluster. Topics can have 0 to N replications and one server is always designated leader (or primary) for a topic and other replicas are secondary. The way it’s supposed to work is that the primary topic partition server will not acknowledge or formally accept any message until it is replicated to all other topic replicas.
What Kasten is proposing is to use topic replicas and take hot snapshots of replicated topic partitions at the secondary server. That way there should be a minimal impact on primary topic message processing. Once snapped, topic partition message data can be backed up and sent offsite for DR.
However, even though Kasten plans to issue hot snapshots, one after another, for each partition, there is still a small time difference between each snap request. As a result, the overall state of the topic partitions when snapped may be slightly inconsistent. In fact, there is a small possibility that when you stitch all the partition snapshots together for a topic, some messages may be missing.
For example, say a topic is replicated and when looking at the replicated topic partition some message say Msg was delayed (due to server load) in being replicated for partition 0, while Kasten was hot snapping replica partition 15 which already held Msg. In this case the snapshots missed Msg. Thus hot snapshots in aggregate, can be missing one or (potentially) more messages.
Kasten called this a crash consistent backup but it’s not the term we would use (not sure what the term would be but it’s worse than crash consistent). But to our knowledge, this was the first approach that Kasten (or anyone else) has described that comes close to providing data protection for Kafka messaging.
As another alternative, Kasten suggested one could make the backed up set of partitions better would by post processing all the snapshots to find the last message point where all prior messages were available and jettisoning any followon messages. In this way, after post processing, the set of data derived from the hot snaps would be crash consistent up to that message point..
It seems to me, the next step to create an application consistent Kafka messaging backup would require Kafka to provide some way to quiesce a topic message stream. Once quiesced and after some delay to accept all in flight messages, hot snaps could be taken. The resultant snapshots in aggregate would have all the messages to the last one accepted.
Unclear whether quiescing a Kafka topic stream, even for a matter of seconds to minutes, is feasible for a system that processes 1000s to millions of messages per second.
4 thoughts on “Kasten, Kafka & the quest to protect data #CFD11”
Hi Ray, thanks for all this comment, fact is that I had too little time to explain myself and I probably create confusions. You’ll find here the github project https://github.com/michaelcourcy/kafka-kasten corresponding to this project.
I agree with this point and that’s exactly what our solution do :
“It seems to me, the next step to create an application consistent Kafka messaging backup would require Kafka to provide some way to quiesce a topic message stream. Once quiesced and after some delay to accept all in flight messages, hot snaps could be taken. The resultant snapshots in aggregate would have all the messages to the last one accepted.”
When we take the backup at the mirror we stop the only producers (mirror-maker) and we even stop the kafka cluster. Hence the snap is not anymore a hot snap. And if you check the result on the github project the analysis of the structure of the head of the topics show that we are crash consistent.
Thanks for the clarification. I was trying to keep up with the discussion, write notes and take screen shots. Sometimes I miss an important aspect about it. If I had known there was a Github Repo on the project I would have spent more time reading the documentation there.
If your solution quiesces the mirror-maker producer then the messages queue up inside kafka until the mirror-maker can re-convene. It still seems like there could be a hellacious amount of messages could be waiting in Kafka queues for a quiesced topic, especially for 8K messages/second. I don’t know how fast your storage snapshots are but they typically take seconds to longer to complete on high end storage systems and longer for software defined storage. With lots of messages being queued up waiting for un-quiescing, at some point Kafka either over-runs (with random messages being jettisoned) or stops accepting new messages (with all messages lost from some point until un-quiesced).
In either case, stopping the topic should allow you to create an application consistent backup. If Kasten can do that you have a winer in my book.
And about :
“Unclear whether quiescing a Kafka topic stream, even for a matter of seconds to minutes, is feasible for a system that processes 1000s to millions of messages per second.”
The answer is obviously no for the main cluster but yes for the mirror. You only have to put enough capacity on mirror maker pod to catch up the messages in late. Which is easy because mirror maker proceed by default a batch of 16 000 message at each replication operation.
Thanks for your comments.
I talked about this in my reply to your prior comment. But if you can stop the mirror and leave the leader topic running at some point the mirror must be broken (because the secondary can’t be re-synched anymore) or the leader has the be halted. Or at least this is my view. But again I’m no Kafka expert.
Comments are closed.