Connect your app to Kafka

Your environment's Kafka is a real Apache Kafka cluster (3 brokers, SCRAM authentication). Durability comes from S3, not broker replication — brokers are stateless, and every byte they write lands in your bucket, which is what survives a broker restart. Any Kafka client works — there is no PerunsCloud SDK.

There are two ways in, and they are genuinely different.

From inside your environment

This is the normal case: a Flink job, or anything else running in your cluster.

Use the bootstrap address and credentials from the Kafka tab:

bootstrap.servers=peruns-kafka-kafka-bootstrap.kafka.svc:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule \
  required username="tenant-app" password="<from the Kafka tab>";

SASL_PLAINTEXT is correct here. The traffic never leaves your cluster, and network policies already restrict who can reach the brokers to your own jobs. If you prefer TLS inside the cluster too, port 9093 speaks the same SCRAM over TLS.

Use the Read from Kafka preset

If this is a Flink job, the preset does all of the above for you and turns on exactly-once checkpointing. Skipping checkpointing on a Kafka consumer means offsets are re-read or lost on every restart, which looks a lot like data loss.

From your laptop or another network

Your brokers are not on the public internet. That is deliberate — an exposed Kafka is a data breach waiting for a weak password — so external access goes through an encrypted tunnel you open on demand.

Install cloudflared, then:

cloudflared access tcp \
  --hostname kafka-peruns-<org>-<env>.peruns.cloud \
  --url localhost:9092

Leave that running. It forwards localhost:9092 on your machine to your brokers. Now point any Kafka client at localhost:9092 with the same SCRAM credentials:

kcat -b localhost:9092 \
  -X security.protocol=SASL_PLAINTEXT \
  -X sasl.mechanism=SCRAM-SHA-512 \
  -X sasl.username=tenant-app \
  -X sasl.password='<from the Kafka tab>' \
  -L

This is a development path, not a production one

The tunnel is a process on your machine. It is right for a console consumer, a local test, or debugging a topic at 2am. It is the wrong way to run a production application — for that, deploy the application into the environment, where it reaches the brokers directly with no tunnel, no extra hop and no laptop in the path.

If you need durable external access for a production system outside PerunsCloud, tell us. It is a supported thing to add; it is just not the default, because most people asking for it actually want the in-cluster path.

Zero-RPO status

The Kafka tab shows which engine is behind your brokers and, on a dual-region environment, a replication badge. It is worth knowing what each state actually claims, because the badge is deliberately honest about what we can and cannot see.

Zero-RPO: active (both regions) means every acknowledged write is already in both regions' object storage — losing a region loses no acknowledged data. Zero-RPO suspended means writes are being accepted against one region only, so data written during that window is not yet replicated; the count next to it is how many objects are waiting to be copied back. Restoring parity is the catch-up after such a window, and zero-RPO resumes on its own when it finishes. Produce paused is the default stance during a remote-region outage: rather than accept writes it cannot replicate, the cluster stops accepting them, which trades availability for keeping RPO at zero. Switching that trade-off for the duration of an incident is a deliberate decision — ask us. Finally, Metrics unavailable means exactly that: our monitoring could not read the replication state, so we are not claiming anything in either direction. It is grey rather than red on purpose — your brokers' own readiness is reported separately, right above it, and is unaffected. A single-region environment shows Single region instead, which is not a fault: cross-region replication is a Gold feature.

Python and Java

Once the tunnel is up (or you are running in-cluster), it is ordinary Kafka:

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    "orders",
    bootstrap_servers="localhost:9092",     # or the in-cluster bootstrap
    security_protocol="SASL_PLAINTEXT",
    sasl_mechanism="SCRAM-SHA-512",
    sasl_plain_username="tenant-app",
    sasl_plain_password="...",
    group_id="my-consumer",
    auto_offset_reset="earliest",
)

Things that will bite you

Writes are acknowledged in hundreds of milliseconds, not single digits. This is the one number that will surprise you if you have run Kafka before, and it is the direct cost of the durability described above: a produce is acknowledged once the data is in your S3 bucket, not once it has reached another broker's disk. A classic Kafka cluster answers in single-digit milliseconds and can lose the write if the wrong machines die together; this one answers in hundreds and cannot.

It is a latency cost, not a throughput one. Batching absorbs it almost completely, and every mainstream client batches by default — so a pipeline moving millions of messages is unaffected, while a request/response pattern that produces one message and waits for the ack feels every millisecond. If you have such a path, raise linger.ms and let the client group writes, or keep it off Kafka. Do not tune this by lowering durability; there is no setting that makes the bucket faster.

You can watch it: Produce latency p99 on the Kafka dashboard in Grafana.

A topic must exist first. Auto-creation is off. A producer writing to an unknown topic gets UNKNOWN_TOPIC_OR_PARTITION rather than a silently-created one-partition topic that skips the partition and retention guardrails the Kafka tab enforces. Create topics from there instead.

Suspending an environment discards topic data. Suspend deletes the whole environment's compute, and the brokers' cluster metadata goes with it — resume rebuilds the cluster from your declarative configuration rather than restoring it. Your topic definitions and credentials come back; the messages do not. Flink state in S3 survives — that is the durable thing. See Checkpoints & storage.

Consumer lag is on the Kafka tab and in Grafana. If a pipeline "feels slow", that is the number to look at first.