Introduction
Hello 👋, In this post we’ll see about sending traces from a demo app to AWS X-Ray via the ADOT(AWS Distro for OpenTelemetry) collector. We would then visualize this on Grafana. Note that we’d deploy the workloads on a kubernetes cluster.
Here is a picture of what we are trying to accomplish:
Alright, let’s get started!!!
Namespace
We shall deploy the workloads on a separate namespace. Let’s create one.
Credentials
Store the AWS credentials as a kubernetes secret.
–from-literal=AWS_ACCESS_KEY_ID=<access-key-id>
–from-literal=AWS_SECRET_ACCESS_KEY=<aws-secret-access-key>
-n adot-traces-demo
ADOT Config
Set the ADOT config in a file.
exporters:
awsxray:
region: ap-south-2
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
service:
pipelines:
traces:
exporters:
– awsxray
receivers:
– otlp
And create a config map with this file.
ADOT Deployment
Setup deployment spec in a file, that injects the secret we created earlier as environment variables and the configmap as a volume.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: adot-collector
name: adot-collector
spec:
replicas: 1
selector:
matchLabels:
app: adot-collector
template:
metadata:
labels:
app: adot-collector
spec:
containers:
– args:
– ‘–config=/etc/adot-config.yaml’
envFrom:
– secretRef:
name: aws-credentials
image: public.ecr.aws/aws-observability/aws-otel-collector:latest
name: adot-collector
volumeMounts:
– mountPath: /etc/adot-config.yaml
name: config-volume
subPath: adot-config.yaml
volumes:
– configMap:
name: adot-config
name: config-volume
Create the deployment.
The pod in the deployment should be running.
NAME READY STATUS RESTARTS AGE
adot-collector-7cbf849b89-b4bkl 1/1 Running 0 3m26s
ADOT Service
We can expose the ADOT deployment with a service spec that exposes the grpc port 4317 as follows.
apiVersion: v1
kind: Service
metadata:
name: adot-collector-service
spec:
selector:
app: adot-collector
ports:
– protocol: TCP
port: 4317
targetPort: 4317
We can now create the service.
The endpoint IP should match with the pod IP.
NAME ENDPOINTS AGE
adot-collector-service 10.1.3.187:4317 22s
$ kubectl get po -n adot-traces-demo -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
adot-collector-7cbf849b89-b4bkl 1/1 Running 0 7m11s 10.1.3.187 docker-desktop <none> <none>
Demo app
We can now deploy the sample demo app which can send traces to ADOT collector, with the following manifest.
apiVersion: apps/v1
kind: Deployment
metadata:
name: xk6-tracing
spec:
replicas: 1
selector:
matchLabels:
app: xk6-tracing
template:
metadata:
labels:
app: xk6-tracing
spec:
containers:
– env:
– name: ENDPOINT
value: adot-collector-service:4317
image: ghcr.io/grafana/xk6-client-tracing:v0.0.2
name: xk6-tracing
Let’s create the deployment.
Both the ADOT collector and k6-tracing pods should now be running.
NAME READY STATUS RESTARTS AGE
adot-collector-7cbf849b89-b4bkl 1/1 Running 0 14m
xk6-tracing-69b48fcfd9-bjzbd 1/1 Running 0 24s
X-Ray
We can now headover to AWS X-Ray, in ap-south-2 region that we mentioned in the adot-config.
The nodes(services) shown in the screenshot belong to our demo application. We could filter for traces that passes through a particular service name for ex. article service, like below.
If we click on a single trace we should be able to see a complete service map for that trace, that shows all the services that trace traverses.
If we go a further down on this we should be able to see the details for segments/spans with in this trace.
Grafana
So we far we were able to see the traces in AWS X-Ray, we can do a similar exercise on Grafana. I am using a Grafana Cloud Free subscription for this lab.
Go to Connections, Add a new connection and search for X-Ray and install it.
You can then go to datasources, add a new X-Ray datasource with the access key id, secret access key, and default region(I have chosen ap-south-2 which matches with adot config).
All good, we can try adding a new panel, go to dashboards > new dashboard and a new visualization with table as panel type and a sample query for ex. service(id(name: “article-service” ))
We can click on one of the traces we should take us to the explore view where we can see the node graph(service map)
We should also see the trace explorer that shows the individual spans.
Okay so we reached this far, that was some fun exploring traces on AWS and Grafana with Open Telemetry. Thank you for reading !!!
CleanUp
Just delete the namespace with kubectl delete ns adot-traces-demo and it should remove the workloads we deployed from kubernetes and stop sending any new data to the cloud.
