Reference · built on requestConnectorStreams, queues & webhooks

Amazon Kinesis Data Streams

Records from Kinesis data stream shards through GetRecords or SubscribeToShard, plus stream and consumer state.

Request accessPricing ↗

About

Amazon Kinesis Data Streams is a hosted stream service from Amazon Web Services. A data stream is made of shards. In provisioned mode you set the shard count, and each shard takes up to 1 MB or 1,000 records per second of writes and serves up to 2 MB per second of reads. In on-demand mode the service scales shards for you. Producers call PutRecord or PutRecords. Consumers either poll a shard with GetShardIterator and GetRecords, sharing that 2 MB per second, or register with RegisterStreamConsumer and receive records pushed over HTTP/2 through SubscribeToShard with their own 2 MB per second per shard.

Through Fibric, an operator reads records from the shards of a stream and the stream's state. It proposes a write, a consumer registration, or a retention change. You approve. The connector makes the call once.

This is a reference listing. It documents what Fibric would read from Amazon Kinesis Data Streams and what it could propose, based on the vendor's published interfaces. Fibric builds it under a managed deployment when you request it; selecting it here installs nothing.

Inputs

  • Records from each shard through GetShardIterator and GetRecords, up to 10,000 records or 10 MB per call
  • Records pushed over HTTP/2 through SubscribeToShard when the connector is a registered enhanced fan-out consumer
  • The shard list from ListShards and stream description from DescribeStreamSummary, including capacity mode and retention
  • Registered consumers on a stream from ListStreamConsumers and DescribeStreamConsumer
  • ProvisionedThroughputExceededException responses, which mark a shard read past its rate
  • Account limits from DescribeLimits

Proposed actions

  • Target capability: propose writing records with PutRecords, up to 500 records and 10 MiB per request
  • Target capability: propose registering this connector as an enhanced fan-out consumer with RegisterStreamConsumer
  • Target capability: propose a longer retention with IncreaseStreamRetentionPeriod, up to 8760 hours
  • Target capability: propose a shard count change with UpdateShardCount on a provisioned stream

Proposed actions are target capabilities. Every action runs propose-first and needs a validated deployment and the appropriate permissions.

What you can build

  • Read an event stream without touching other consumers

    The connector registers as an enhanced fan-out consumer and receives records over SubscribeToShard with its own throughput. Other readers on the stream keep theirs.

    With Integration Health

  • Catch a shard that is being throttled

    ProvisionedThroughputExceededException starts appearing on a shard. The operator names the stream and shard and proposes UpdateShardCount, or a switch to on-demand mode, for your approval.

    With Quota Headroom

  • Keep records long enough to replay

    Retention sits at the 24 hour minimum. Before a risky release, the operator proposes IncreaseStreamRetentionPeriod so the window covers the rollback plan.

    With Webhook Replay, Data Freshness

  • Tie stream cost to consumers

    Enhanced fan-out adds a data retrieval cost and a consumer-shard hour cost. The operator lists registered consumers and flags ones that no longer subscribe.

    With Cost Anomaly

Requirements

  • An AWS account with a data stream in the Region the connector runs against
  • An IAM policy allowing kinesis:DescribeStreamSummary, kinesis:ListShards, kinesis:GetShardIterator, and kinesis:GetRecords on the stream ARN
  • For enhanced fan-out, kinesis:DescribeStreamConsumer and kinesis:SubscribeToShard on the consumer ARN as well
  • For writes, kinesis:PutRecord and kinesis:PutRecords on the stream ARN
  • Cross-account reads need a resource-based policy set with PutResourcePolicy. Wildcards are not allowed in it
Authentication
AWS IAM credentials, normally an IAM role, allowed kinesis: actions on the stream ARN, with a resource-based policy on the stream and consumer ARN for cross-account access.

Limits

  • GetRecords allows five read transactions per second per shard and 2 MB per second, shared by every polling consumer on that shard
  • A shard iterator expires 5 minutes after it is returned
  • Up to 20 registered enhanced fan-out consumers per stream, or 50 in On-demand Advantage mode
  • A stream can switch between on-demand and provisioned modes twice within 24 hours

Access and pricing

Reference listing. Fibric builds the connector under a managed deployment when you request it. Your quote covers the build, capabilities, usage, and support.

Request Amazon Kinesis Data Streams ↗

Questions and answers

What is the difference between GetRecords and SubscribeToShard?
GetRecords is a pull over HTTP. Every consumer polling a shard shares a fixed 2 MB per second. SubscribeToShard pushes records over HTTP/2 to a registered enhanced fan-out consumer, and each such consumer gets up to 2 MB per second per shard on its own.
How long are records kept?
The minimum retention period is 24 hours. IncreaseStreamRetentionPeriod can raise it to 8760 hours, which is 365 days. DecreaseStreamRetentionPeriod lowers it again. Both calls are rate-limited per stream.
Can the connector read a stream in another AWS account?
Yes. Attach a resource-based policy to the data stream with PutResourcePolicy naming the connector's principal. Enhanced fan-out sharing needs a policy on both the stream ARN and the consumer ARN. Wildcards, service principals, and federated principals are not accepted in that policy.
Ask about Amazon Kinesis Data Streams

Ask about the capabilities and requirements in this listing.

For project-specific requirements, contact Fibric.

This connector listing is developed, published, and supported by Fibric. It describes integration with Amazon Kinesis Data Streams through published interfaces. Third-party names and logos identify the systems an integration connects to; they are the property of their respective owners, who are not affiliated with Fibric and do not sponsor or endorse this listing. Trademark policy