We are integrating fluentbit into Kafka via Kubernetes deployed using strimzi.io and we hit our first issue.
When creating a KafkaUser it will not create the secret needed for tls in our fluent namespace
kind: KafkaUser
metadata:
name: fluent
namespace: fluent
labels:
strimzi.io/cluster: debezium-cluster
spec:
authentication:
type: tls
authorization:
type: simple
acls:
– resource:
name: ‘*’
patternType: literal
type: topic
operation: All
– resource:
name: ‘*’
patternType: literal
type: group
operation: All
– resource:
type: cluster
operation: All
it simply sits there like
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY
fluent debezium-cluster tls simple
Reading up it seems that this is a long running issue and although there is a fix for java applications, it would appear you need to deploy something else to mirror the secret generated into the fluent namespace.
In one of the comments it lead us to https://config-syncer.com/docs/v0.14.7/setup/install/ which had a comment about another tool emberstack/kubernetes-reflector.
Installation is pretty straight forward.
helm repo update
helm upgrade –install reflector -n emberstack –create-namespace emberstack/reflector
Then you need to annotate the KafkaUser yaml and apply it
kind: KafkaUser
metadata:
name: fluent
namespace: kakfa
labels:
strimzi.io/cluster: kakfa-cluster
spec:
authentication:
type: tls
template:
secret:
metadata:
annotations:
reflector.v1.k8s.emberstack.com/reflection-allowed: “true”
reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: “fluent”
authorization:
type: simple
acls:
– resource:
name: ‘*’
patternType: literal
type: topic
operation: All
– resource:
name: ‘*’
patternType: literal
type: group
operation: All
– resource:
type: cluster
operation: All
and then finally create a empty secret in the fluent namespace and annotate it to mirror the secret created previously.
kind: Secret
metadata:
name: fluent
namespace: fluent
annotations:
reflector.v1.k8s.emberstack.com/reflects: “kafka/fluent”
type: Opaque
When completed the secret is mirrored (and maintained)
NAME TYPE DATA AGE
fluent Opaque 5 26m
kubectl get secret fluent -n fluent
NAME TYPE DATA AGE
fluent Opaque 5 19m
You can now reference the secret in your config.
Read how to sync the Kafka Cluster CA certificate into your namespace to enable the sync of the Kafka Cluster CA Certificate.
