AWS Lambda Adds Queue Mode for Self-Managed Kafka: Several Pollers Can Read One Partition Without Ordering, and Amazon MSK Is Not Yet Supported
On November 2, 2026, AWS announced queue mode for self-managed Kafka event source mappings in Lambda: built on Kafka share groups, several event pollers can read the same partition at once, each with up to 10 concurrent invocations, with no change to producers, topics or function code, but without partition ordering. The announcement says it covers self-managed clusters and managed offerings such as Confluent Cloud, while the developer guide says queue mode is for self-managed Kafka only, Amazon MSK does not currently support it, and the poller needs Kafka 4.2 or later.
On November 2, 2026, AWS announced that AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event source mappings. Until now, a Kafka event source mapping used the Kafka consumer group protocol, which assigns each partition to exactly one consumer in a group. A team that wanted more parallelism had to over-partition its topics or move traffic to a separate queuing system, which added infrastructure to run and producer code to maintain. The new queue mode is built on Kafka share groups, which let several event pollers read from the same partition at the same time.
AWS says each event poller supports up to 10 concurrent invocations, and that parallelism is no longer tied to partition count, with no change to producers, topic configuration or function code. A slow or failed event no longer holds up other events in the same partition. The announcement is plain about the cost of this: queue mode does not preserve partition ordering, and it is meant for workloads that do not need it. It names use cases where each event is an independent unit of work and speed matters more than order, such as image and video encoding, fraud detection, notification delivery and telemetry ingestion.
The requirements are narrower than the headline suggests. The announcement says the cluster must have share groups enabled, such as Apache Kafka 4.2 or later, and that the feature is available in all AWS commercial Regions where self-managed Kafka event source mapping is available. It says the feature covers clusters that customers run themselves and managed offerings such as Confluent Cloud. The Lambda developer guide goes further. It says share groups became available in Apache Kafka 4.0, but the Kafka Queue Poller requires version 4.2 or later, and that queue mode is for self-managed clusters only. In its words, Amazon MSK does not currently support queue mode, and AWS will add MSK support once MSK offers Kafka 4.2 or later with share groups enabled. A team that runs Amazon MSK should therefore not read the announcement as covering its cluster.
To use it, create a self-managed Kafka event source mapping with ConsumptionMode set to Queue, through the console, the Lambda API, the AWS CLI, CloudFormation, SAM or CDK. If no mode is given, the mapping uses the default Stream mode. The guide compares the two. Stream keeps per-partition ordering and allows up to 5 concurrent invocations per poller. Queue gives no ordering guarantee and allows up to 10. Both list up to 5 MB/s of throughput per poller, but the on-demand section for queue mode lists up to 1 MB/s per poller, with at most 5 pollers, 5 MB/s in total and 50 concurrent invocations. Provisioned mode lets you set minimum and maximum pollers, from 1 to 200 for the minimum and 1 to 2,000 for the maximum, and the guide lists up to 5 MB/s per poller there. Because the pages describe different limits for different modes, read the mode you will actually run.
Several familiar settings change. MaximumBatchingWindowInSeconds is not supported in queue mode, and specifying it returns a validation error, because the Kafka share consumer needs explicit acknowledgement of every record in a batch. StartingPosition and StartingPositionTimestamp are not supported either, since queue mode always reads from the share group's committed position. MaximumRetryAttempts defaults to 0, and the broker performs retries when no on-failure destination is configured. Event filtering with up to 5 patterns, schema registries, on-failure destinations and event source mapping logging are supported. Queue event source mappings share event poller units only with other queue mappings. The general Kafka guide also warns that mappings process each event at least once, so function code should be idempotent.
On cost, the announcement says standard Lambda pricing applies, and that customers who configure provisioned mode also pay for Event Poller Unit usage. A sensible trial starts with a non-critical topic: confirm the cluster version and that share groups are on, create a queue mapping in a test account, send a burst, and measure throughput, duplicates and failed-record handling. Decide first whether any consumer needs order, because queue mode removes it. The post gives no benchmark of throughput gains, so any speedup claim has to come from your own test.
Sources
FAQ
What problem does Lambda's Kafka queue mode solve?
Before, each partition went to one consumer in a group, so more parallelism meant over-partitioning or a separate queue system. Queue mode uses Kafka share groups so several pollers read the same partition, each with up to 10 concurrent invocations, with no producer, topic or function change. The cost is that partition ordering is not preserved.
Can Amazon MSK users use this feature?
The announcement says it covers self-managed clusters and managed offerings such as Confluent Cloud, but the developer guide says queue mode is for self-managed Kafka only and Amazon MSK does not currently support it, pending a Kafka 4.2 or later offering with share groups. MSK users should not assume they can use it yet.
Which limits should be checked before relying on it?
Check Kafka 4.2 or later, share groups, ordering and idempotency. Queue mode has no batching window or starting position and permits five filter patterns. MaximumRetryAttempts defaults to 0, but without an on-failure destination the Kafka broker retries failed records. Test the throughput limits of the on-demand or provisioned mode actually used.