Reference · built on requestConnectorStreams, queues & webhooks

Apache Kafka

Events from Kafka topics through the Consumer, Producer, and Admin APIs over Kafka's binary protocol on TCP.

Request accessPricing ↗

About

Apache Kafka is an event log run by the Apache Software Foundation. Producers write events to topics. A topic is split into partitions, each an ordered commit log spread across brokers. Every event carries a key, a value, a timestamp, and optional headers. Events with the same key land on the same partition, and a consumer reads a partition in the exact order it was written. Retention is set per topic by time or size, and log compaction keeps at least the last value for each key.

Through Fibric, a consumer group of its own reads the topics you name, keeps its own offsets, and hands each event to an operator. When an operator wants to write back, it proposes a produce, an offset commit, or a topic change for your approval, and the connector calls the broker once.

This is a reference listing. It documents what Fibric would read from Apache Kafka and what it could propose, based on the vendor's published interfaces. Fibric builds it under a managed deployment when you request it; selecting it here installs nothing.

Inputs

  • Events on the topic-partitions you name, read through the Consumer API in the order each partition was written
  • The key, value, timestamp, and headers of each event
  • Consumer group position per partition: the offset of the next message to consume, read with OffsetFetch
  • Topic, partition, and broker metadata from the Metadata and ListOffsets requests
  • Topic configuration and retention settings inspected through the Admin API
  • Quota throttling, seen as the delay a broker adds to responses for a client over its byte-rate or request-rate quota

Proposed actions

  • Target capability: propose producing an event to a topic through the Producer API, batched with the Produce request
  • Target capability: propose committing this connector's own consumer group offsets with OffsetCommit
  • Target capability: propose creating a topic through the Admin API's CreateTopics request
  • Target capability: propose a topic retention or compaction setting change through the Admin API

Proposed actions are target capabilities. Every action runs propose-first and needs a validated deployment and the appropriate permissions.

What you can build

  • Watch an order topic for events that need a person

    Order and payment events flow through a topic. The connector reads them in partition order and hands each to the operator, which flags the orders that need a hold or a call before they ship.

    With Order Risk

  • Catch a consumer that has stopped moving

    The operator compares each group's committed offset with the partition's latest offset. When lag grows and stays, it raises the pipeline that stalled and the topic behind it.

    With Data Freshness, Integration Health

  • Replay a window of events

    After a downstream outage, the operator proposes resetting this connector's offsets to a start point within retention. You approve. The events are read again and the receipt records the range.

    With Webhook Replay

  • Keep quota headroom

    When brokers begin delaying a client's responses, the operator reports which client-id or user hit its byte-rate or request-rate quota and proposes a quota change.

    With Quota Headroom

Requirements

  • Network reach to the brokers. Kafka speaks a binary protocol over TCP, and a request must go to the broker that leads the partition
  • A principal with authorization to read the named topics and to manage its own consumer group
  • For writes, authorization on the target topic and, for topic changes, Admin API rights
  • Retention long enough on each topic for any replay you expect. Old data is discarded by time or size
Authentication
A broker listener with SASL (GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, or OAUTHBEARER) or SSL client authentication, plus ACLs that authorize read and write on the topics and group.

Limits

  • Kafka publishes no HTTP API of its own. Reads and writes go over the binary protocol, so brokers must be reachable on their listener port
  • Ordering holds within a partition, not across a topic. Two events on different partitions have no order between them
  • Brokers throttle clients that exceed network bandwidth or request rate quotas by delaying responses
  • The broker has a configurable maximum request size. A request over it disconnects the socket

Access and pricing

Reference listing. Fibric builds the connector under a managed deployment when you request it. Your quote covers the build, capabilities, usage, and support.

Request Apache Kafka ↗

Questions and answers

Does Kafka expose a REST API?
No. Kafka documents five APIs (Admin, Producer, Consumer, Kafka Streams, Kafka Connect) that speak a binary protocol over TCP. HTTP access comes from vendors that wrap Kafka, such as Confluent Cloud and Redpanda, which have their own listings.
What order does the connector read events in?
Kafka guarantees that a consumer of a topic-partition reads events in the same order they were written. Events with the same key go to the same partition. There is no ordering guarantee across partitions of one topic.
How is the connection authenticated?
Kafka authenticates clients with SSL or SASL. The documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, and OAUTHBEARER. Authorization of read and write operations is pluggable, and ACLs are the built-in option.
Ask about Apache Kafka

Ask about the capabilities and requirements in this listing.

For project-specific requirements, contact Fibric.

This connector listing is developed, published, and supported by Fibric. It describes integration with Apache Kafka through published interfaces. Third-party names and logos identify the systems an integration connects to; they are the property of their respective owners, who are not affiliated with Fibric and do not sponsor or endorse this listing. Trademark policy