Datadog APM tracing and logging

Recently, I have set up my micro services application onto datadog APM with tracing and logging.

something like,

so that we can follow through, for example, any http request or user_id from HTTP request, controller, service, authentication, database, and etc, with the same tracing ID and logs detailed.

Here are the steps I have followed:

1. Set up datadog account and access, so that we will get the datadog API key, which is to be used to upload the data to datadog server
2. install the datadog agent onto the server, we are using kubernetes on AWS (EKS) here, and have installed it as a daemonset (so that we have one for each node) with terraform.

set up the necessary environment variable in the daemonset

and expose the necessary ports

3. from the python services, leverage on libraries, for example, https://pypi.org/project/JSON-log-formatter/0.1.0/, to log the messages into JSON format

format = "%(asctime)s %(levelname)s [%(name)s]-- '[dd.trace_id=%(dd.trace_id)s dd.span_id=%(dd.span_id)s] ' -- %(message)s"
json = jsonlogger.JsonFormatter(format)
handler = logging.StreamHandler()
handler.setFormatter(json)

so that these logs would be in JSON format, and datadog would be able to parse the logs into tags & attributes.

4. then configure the datadog tracing

from ddtrace import config, patch_all, tracer

def setup_datadog():
    tracer.configure(
        hostname=os.environ["DD_AGENT_HOST"],#set up in the deployment.yaml, the env attribute
    )
    tracer.set_tags({"env": "production"})
    config.flask["service_name"] = "checkout-service" #this could also set up in the deployment.yaml, the env attribute
    config.flask["analytics_enabled"] = True
    config.flask["extra_error_codes"] = [401, 403]
    patch_all()

5. finally, then annotate the entry point function, with

@tracer.wrap(service="checkout-service")

sample configuration in the deployment.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
...
  annotations:
    ad.datadoghq.com/checkout.logs: '[{"source":"python", "service": "checkout-service"}]'
spec:
  replicas: 1
...
  template:
    metadata:
    spec:
      containers:
      -  name: checkout
...
        env:
          - name: DD_AGENT_HOST
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
          - name: DD_SERVICE
            value: checkout-service
          - name: DD_TRACE_ANALYTICS_ENABLED
            value: "true"
...
      restartPolicy: Always
status: {}