How can I delete all the messages from a Kafka topic?

Question

I created a Kafka topic and pushed large messages to that Kafka topic. Now I am getting an error saying:

kafka.common.InvalidMessageSizeException: invalid message size

How do I want to purge the topic so, that I can set the fetch.size for the topic?

Answer 1

To delete all the messages from a Kafka topic, you need to change the retention time of that topic the default retention time of Kafka topic is 168 hours, i.e. 7 days. In this case, you have to change the retention time to 1 second, after which the messages from the topic will be deleted. After changing the retention time, you can go ahead and change the retention time of the topic back to 168 hours.

You can change the retention of the topic by using the following command:

kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic1 --config retention.ms=1000

If you wish to know about Kafka visit this Kafka Tutorial and Kafka Interview Questions.

Answer 2

 If you want to clear all the messages from a Kafka topic, you may either choose to drop and create the topic again or you can change the retention settings. Below are the steps concerned in each case.Approach 1: Drop the Topic and Recreate it

• Delete the Topic:

bin/kafka-topics.sh --delete --topic your_topic_name --bootstrap-server your_broker_address

• Create the Topic again:

bin/kafka-topics.sh --create –topic your_topic_name –bootstrap-server your_broker_address –partitions 1 –replication-factor 1Approach 2: Change Retention Settings

• Reduce Retention Period to 1 Minute (60000 ms):

bin/kafka-configs.sh --alter --entity-type topics --entity-name your_topic_name --add-config retention.ms=60000 –bootstrap-server your_broker_addressGive some time after which, if needed reset the retention:  bin/kafka-configs.sh --alter --entity-type topics --entity-name your_topic_name    --add-config retention.ms=604800000 –bootstrap-server your_broker_addressNoteOnce the topic has ben cleared, you will be able to increase your fetch.size after that as per your needs.

Answer 3

 If you want to clear all the messages from a Kafka topic, you may either choose to drop and create the topic again or you can change the retention settings. Below are the steps concerned in each case.Approach 1: Drop the Topic and Recreate it

• Delete the Topic:

bin/kafka-topics.sh --delete --topic your_topic_name --bootstrap-server your_broker_address

• Create the Topic again:

bin/kafka-topics.sh --create –topic your_topic_name –bootstrap-server your_broker_address –partitions 1 –replication-factor 1Approach 2: Change Retention Settings

• Reduce Retention Period to 1 Minute (60000 ms):

bin/kafka-configs.sh --alter --entity-type topics --entity-name your_topic_name --add-config retention.ms=60000 –bootstrap-server your_broker_address. Give some time after which, if needed reset the retention:bin/kafka-configs.sh --alter --entity-type topics --entity-name your_topic_name    --add-config retention.ms=604800000 –bootstrap-server your_broker_addressNote

Once the topic has ben cleared, you will be able to increase your fetch.size after that as per your needs.