Skip to content

Instantly share code, notes, and snippets.

@gphilipp

gphilipp/kafka_releases_history_since_2_0.md

Last active August 8, 2024 15:04
Show Gist options
  • Save gphilipp/d1c01ccb6b179d5ac7cd38b4129cc107 to your computer and use it in GitHub Desktop.
Save gphilipp/d1c01ccb6b179d5ac7cd38b4129cc107 to your computer and use it in GitHub Desktop.
Download ZIP
Kafka releases history since 2.0
Raw
kafka_releases_history_since_2_0.md
Release Blog Post What you’re missing out if you're using this version Notable Changes Date New Features Improvements Bug Fixes
3.8
(29 Jul 2024)		 What's new in Apache Kafka 3.8 0 new features, 0 improvements, 0 bug fixes
  • KIP-858: JBOD in KRaft is now GA.
  • KIP-390: Adds support for specifying compression levels.
  • KIP-848: The new Consumer Rebalance Protocol, available in preview.
  • KIP-993: Restricts file access by File and Directory ConfigProviders.
  • KIP-974: Introduces Docker image for GraalVM-based native Kafka broker (great for faster test runs 🚀).
  • KIP-1018: Introduces remote.fetch.max.wait.ms for DelayedRemoteFetch requests.
  • KIP-1028: Docker official image for Apache Kafka.
  • Kafka Streams: Shareable State Stores, customizable task assignment, and improved StateStore Iterator metrics.
 2024-07-29 5 83 162
3.7.1
(28 Jun 2024)		 5 new features, 81 improvements, 116 bug fixes 2024-06-28 2 46
3.7
(27 Feb 2024)		 What's New in Apache 3.7 5 new features, 83 improvements, 162 bug fixes
  • KIP-858: Adds JBOD support for KRaft clusters (Early Access).
  • KIP-714: Improved client metrics and observability.
  • KIP-848 (Early Access): New Consumer Rebalance Protocol.
  • KIP-975: Docker image for Apache Kafka.
  • KIP-963: Additional metrics in Tiered Storage.
  • KIP-919: AdminClient communication with KRaft Controller Quorum.
  • Kafka Streams: Rack-aware task assignment and enhanced IQ support, Expand default DSL store configuration to custom types, IQ support for Versioned State Stores.
  • Kafka Connect: Cluster-wide dynamic log adjustment, Allow creating connectors in a stopped state.
 2024-02-27 6 47 79
3.6.2
(04 Apr 2024)		 11 new features, 125 improvements, 198 bug fixes 2024-04-04 3 22
3.6.1
(7 Dec 2023)		 11 new features, 128 improvements, 220 bug fixes 2023-12-07 2 21
3.6
(10 Oct 2023)		 What's New in Apache 3.6 11 new features, 130 improvements, 241 bug fixes
  • KIP-405: Kafka Tiered Storage (Early Access) - Enables storage extension to remote objects for separating compute and storage.
  • KIP-890: Transactions Server Side Defense - Prevents hanging transactions.
  • KIP-797: Duplicate Listener on Port for IPv4/IPv6 - Supports same port on different IP stacks.
  • KIP-868: Metadata Transactions - Improves KRaft durability with atomic record batches.
  • KIP-923: Grace Period in Stream Table Join - Improves handling of out-of-order data in joins.
  • KRaft Production-Ready: Migration from ZooKeeper to KRaft is now production-ready.
  • Memory Performance: Reduces memory allocation during record deserialization.
 2023-10-10 6 53 70
3.5.2
(11 Dec 2023)		 17 new features, 183 improvements, 268 bug fixes 2023-12-11 35
3.5.1
(21 Jul 2023)		 17 new features, 183 improvements, 303 bug fixes 2023-07-21 8
3.5
(15 Jun 2023)		 What's New in Apache 3.5 17 new features, 183 improvements, 311 bug fixes
  • KIP-881: Rack-aware partition assignment for consumers (assignors)
  • KIP-887: Environment variable support for configurations.
  • KIP-900: SCRAM support for inter-broker authentication in KRaft.
  • KIP-903: Prevents replicas with stale broker epoch from joining ISR.
  • Kafka Streams Enhancements: Versioned state stores and Boolean Serde.
  • Kafka Connect: Improved MirrorMaker 2.0 and first-class offset support.
 2023-06-15 50 77
3.4.1
(6 June 2023)		 13 new features, 195 improvements, 274 bug fixes 2023-06-06 9 41
3.4.0
(7 Feb 2023)		 What's New in Apache Kafka 3.4
(7 Feb 2023)		 17 new features, 233 improvements, 388 bug fixes
  • KIP-866: Early access for migrating Kafka clusters from ZooKeeper to KRaft mode without downtime.
  • KIP-792: Adds generation field into consumer protocol.
  • KIP-830: Allows disabling JMX reporter.
  • KIP-840: Config file option for ConsoleProducer/ConsoleConsumer.
  • KIP-854: Separate configuration for producer ID expiry.
  • KIP-876: Time-based cluster metadata snapshots.
  • KIP-881: Rack-aware partition assignment for Kafka consumers (protocol change only).
  • Kafka Streams: Updates including KIP-770 and KIP-837.
 2023-02-07 4 29 73
3.3.2
(23 Jan 2023)		 21 new features, 259 improvements, 436 bug fixes 2023-01-23 3 23
3.3.1
(3 Oct 2022)		 21 new features, 262 improvements, 459 bug fixes 2022-10-03 2
3.3 (unannounced, use 3.3.1) What’s New in Apache Kafka 3.3 21 new features, 262 improvements, 461 bug fixes
  • KRaft Mode: Marked as production-ready for new clusters.
  • KIP-778: KRaft-to-KRaft upgrades without double rolling.
  • KIP-841: Improved availability by preventing fenced replicas from joining ISR.
  • KIP-794: Improved partitioning for non-keyed data.
  • KIP-709: Extended OffsetFetch RPC to handle multiple group IDs.
  • Kafka Streams Enhancements: New metrics for throughput and the ability to pause/resume topologies.
  • Kafka Connect: Exactly-once support for source connectors.
 2022-10-03 6 32 100
3.2.3
(19 Sep 2022)		 27 new features, 293 improvements, 544 bug fixes 2022-09-19 5
3.2.2 (Unreleased) 27 new features, 293 improvements, 549 bug fixes
3.2.1
(29 Jul 2022)		 27 new features, 293 improvements, 549 bug fixes 2022-07-29 1 12
3.2
(17 May 2022)		 What's New in Apache 3.2 27 new features, 294 improvements, 561 bug fixes
  • KIP-801: Introduces StandardAuthorizer for secure Kafka clusters without ZooKeeper.
  • KIP-704: Allows partition leaders to recover state after unclean leader election.
  • KIP-764: Configurable backlog size for TCP acceptor sockets.
  • KIP-788: Configure network thread pools per listener.
  • Kafka Streams: Rack awareness for standby replicas and improved Interactive Query interface.
  • Kafka Connect: Enhanced plugin listing and Unix time precision support.
 2022-05-17 2 36 65
3.1.2
(19 Sep 2022)		 29 new features, 325 improvements, 599 bug fixes 2022-09-19 4
3.1.1
(13 May 2022)		 29 new features, 325 improvements, 603 bug fixes 2022-05-13 5 23
3.1
(24 Jan 2022)		 What's New in Apache 3.1 29 new features, 330 improvements, 626 bug fixes
  • KIP-516: Topic IDs in FetchRequest for safer and more efficient topic data fetching.
  • KIP-773: Consistent metric latency naming measured in milliseconds and nanoseconds.
  • KIP-768: Enhanced SASL/OAUTHBEARER with OpenID Connect (OIDC) support.
  • KIP-748: New metrics for active and fenced broker counts.
  • Kafka Streams Improvements: Deprecated eager rebalancing, added TaskId in StreamsException, support for custom partitioners in foreign-key joins, and new metrics for blocked time.
 2022-01-24 1 31 47
3.0.2
(19 Sep 2022)		 30 new features, 359 improvements, 638 bug fixes 2022-09-19 10
3.0.1
(11 Mar 2022)		 30 new features, 359 improvements, 648 bug fixes 2022-03-11 2 25
3.0.0
(21 Sep 2021)		 What's New in Apache Kafka 3.0 30 new features, 361 improvements, 673 bug fixes
  • KRaft Mode Improvements: Enhanced metadata handling, snapshot support, and producer ID generation.
  • Deprecations: Java 8 and Scala 2.12 support deprecated
  • Delivery Guarantees: Producer defaults to idempotency and acks=all.
  • Consumer Features: Increased default session timeout and support for multiple group ID offset fetches.
  • Kafka Streams: Improved timestamp synchronization, new methods for task metadata, and updated SerDes defaults.
 2021-09-21 3 87 132
2.8.2
(19 Sep 2022)		 32 new features, 444 improvements, 754 bug fixes 2022-09-19 11
2.8.1
(17 Sep 2021)		 32 new features, 444 improvements, 765 bug fixes 2021-09-17 1 4 40
2.8.0
(19 Apr 2021)		 What's New in Apache Kafka 2.8 33 new features, 448 improvements, 805 bug fixes
  • KIP-500 Early Access: Ability to run Kafka without ZooKeeper using an internal Raft implementation.
  • Describe Cluster API: New API to query brokers for cluster information.
  • Mutual TLS Authentication: Enhanced support for SASL_SSL listeners.
  • JSON Structured Logs: Emitting JSON structured request/response logs.
  • Connection Rate Limits: Set limits on broker connection creation rates.
  • Topic Identifiers: Introduction of unique topic IDs.
 2021-04-19 3 55 95
2.7.2
(15 Nov 2021)		 36 new features, 497 improvements, 841 bug fixes 2021-11-15 2 21
2.7.1
(10 May 2021)		 36 new features, 499 improvements, 862 bug fixes 2021-05-10 4 38
2.7.0
(21 Dec 2020)		 What's New in Apache Kafka 2.7 36 new features, 503 improvements, 900 bug fixes
  • ZooKeeper Replacement: Progress on inter-broker API and Raft implementation.
  • Tiered Storage: Infinite scaling and faster rebalance times.
  • KIP-654: New TransactionAbortedException for handling aborted transactions.
  • PEM Support: SSL certificates in PEM format supported.
  • Connection Rate Limits: Ability to limit connection creation rates.
  • Operation Throttling: Throttle create topic, partition, and delete topic operations.
  • Kafka Streams: New exception handling and backward iteration in state stores.
 2020-12-21 5 55 91
2.6.3
(15 Nov 2021)		 41 new features, 550 improvements, 906 bug fixes 2021-11-15 1 8
2.6.2
(20 Apr 2021)		 41 new features, 551 improvements, 914 bug fixes 2021-04-20 4 40
2.6.1
(7 Jan 2021)		 41 new features, 555 improvements, 954 bug fixes 2021-01-07 3 37
2.6.0
(3 Aug 2020)		 What’s New in Apache Kafka 2.6 41 new features, 558 improvements, 991 bug fixes
  • Performance Enhancements: Improved broker shutdown, producer compression, and ACL efficiency.
  • Java 14 Support: Full compatibility with Java 14 and Scala 2.13.
  • Client Quota APIs: Simplified quota management via Admin Client.
  • Disk Metrics: New metrics for monitoring disk operations.
  • TLS 1.3 Enabled: Default for Java 11+.
  • Explicit Rebalance: Consumers can explicitly trigger rebalances.
 2020-08-03 7 74 175
2.5.1
(10 Aug 2020)		 48 new features, 625 improvements, 1104 bug fixes 2020-08-10 7 62
2.5.0
(15 Apr 2020)		 What's New in Apache Kafka 2.5 48 new features, 632 improvements, 1166 bug fixes
  • KIP-500: Steps towards removing ZooKeeper dependency.
  • EOS Enhancements: Improved exactly-once semantics.
  • Metrics and Security: New metrics for client versioning and ZooKeeper TLS support.
  • Protocol Enhancements: Better compatibility with L7 proxies.
  • Kafka Connect: Active topic tracking for connectors.
  • Kafka Streams: Added co-group operator and toTable() function.
  • Deprecations: Dropped Scala 2.11 support and set TLS 1.2 as the default SSL protocol.
 2020-04-15 7 65 92
2.4.1
(12 Mar 2020)		 55 new features, 694 improvements, 1223 bug fixes 2020-03-12 3 35
2.4.0
(16 Dec 2019)		 What's New in Apache Kafka 2.4 55 new features, 697 improvements, 1258 bug fixes
  1. KIP-392: Allows consumers to fetch from the closest replica, reducing cross-datacenter network costs.
  2. KIP-429: Incremental Cooperative Rebalancing for more efficient consumer rebalancing.
  3. KIP-455: New Admin API for replica reassignment.
  4. KIP-480: Sticky Partitioner improves message batching and reduces latency.
  5. KIP-482: Optional tagged fields in Kafka protocol for more efficient serialization.
  6. Kafka Connect: MirrorMaker 2.0 and extended support for headers.
  7. Kafka Streams: Non-key joins in KTable and custom processor names.
 2019-12-16
2.3.1
(24 Oct 2019)		 55 new features, 696 improvements, 1213 bug fixes 2019-10-24 1 45
2.3.0
(25 Jun 2019)		 What's New in Apache Kafka 2.3 55 new features, 697 improvements, 1258 bug fixes
  • KIP-351 & KIP-427: Enhanced monitoring for partitions with lost replicas.
  • KIP-354: Introduced maximum log compaction lag.
  • KIP-402: Improved fairness in SocketServer processors.
  • KIP-461: Better failure handling in the Replica Fetcher.
  • Kafka Connect: Incremental cooperative rebalancing and improved logging context.
  • Kafka Streams: Support for storing record timestamps in RocksDB and in-memory window/session stores.
 2019-06-25 7 47 87
2.2.2
(1 Dec 2019)		 36 2019-12-01 1 1
2.2.1
(1 Jun 2029)		 62 new features, 740 improvements, 1306 bug fixes 2019-06-01 4 39
2.2.0
(22 Mar 2019)		 What's New in Apache Kafka 2.2 62 new features, 744 improvements, 1345 bug fixes
  1. AdminClient Use: kafka-preferred-replica-election.sh and kafka-topics.sh now use AdminClient, removing the need for --zookeeper (KIP-183, KIP-337).
  2. KafkaConsumer Group ID: Default group.id changed to null to avoid unnecessary offset fetches/commits (KIP-289).
  3. Separate Planes: Controller and data planes are separated to improve stability (KIP-291).
  4. Control Requests: Detection of outdated control requests for increased stability (KIP-380).
  5. Consumer Group Size Limit: Configurable limit to protect cluster resources (KIP-389).
 2019-03-22 53 82
2.1.1
(15 Feb 2019)		 62 new features, 790 improvements, 1394 bug fixes 2019-02-15 7 33
2.1.0
(20 Nov 2018)		 62 new features, 797 improvements, 1427 bug fixes
  • Java 11 support
  • KIP-110: Zstandard compression.
  • KIP-91: Intuitive User Timeouts in The Producer.
  • KIP-211: Avoid expiring committed offsets for active consumer group.
 2018-11-20 1 63 98
2.0.1
(9 Nov 2018)		 63 new features, 853 improvements, 1485 bug fixes 2018-11-09 7 40
2.0.0
(30 Jul 2018)		 63 new features, 860 improvements, 1525 bug fixes
  • Enhanced security with prefixed ACLs for granular access control, OAuth2 bearer token authentication, and mandatory host name verification for SSL connections.
  • Better perfomance optimizations in replication, message down-conversions, and windowed aggregations in Kafka Streams
  • Expanded functionalities with message headers in Kafka Streams, dynamic SSL truststore updates, and improved error handling in Kafka Connect.
 2018-07-30 4 80
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment