Recently, I have set up my micro services application onto datadog APM with tracing and logging.
something like,
so that we can follow through, for example, any http request or user_id from HTTP request, controller, service, authentication, database, and etc, with the same tracing ID and logs detailed.
Here are the steps I have followed:
- Set up datadog account and access, so that we will get the datadog API key, which is to be used to upload the data to datadog server
- install the datadog agent onto the server, we are using kubernetes on AWS (EKS) here, and have installed it as a daemonset (so that we have one for each node) with terraform.
set up the necessary environment variable in the daemonset
and expose the necessary ports
3. from the python services, leverage on libraries, for example, https://pypi.org/project/JSON-log-formatter/0.1.0/, to log the messages into JSON format
format = "%(asctime)s %(levelname)s [%(name)s]-- '[dd.trace_id=%(dd.trace_id)s dd.span_id=%(dd.span_id)s] ' -- %(message)s"
json = jsonlogger.JsonFormatter(format)handler = logging.StreamHandler()
handler.setFormatter(json)
so that these logs would be in JSON format, and datadog would be able to parse the logs into tags & attributes.
4. then configure the datadog tracing
from ddtrace import config, patch_all, tracer
def setup_datadog():
tracer.configure(
hostname=os.environ["DD_AGENT_HOST"],#set up in the deployment.yaml, the env attribute
)
tracer.set_tags({"env": "production"})
config.flask["service_name"] = "checkout-service" #this could also set up in the deployment.yaml, the env attribute
config.flask["analytics_enabled"] = True
config.flask["extra_error_codes"] = [401, 403]
patch_all()
5. finally, then annotate the entry point function, with
@tracer.wrap(service="checkout-service")
sample configuration in the deployment.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
...
annotations:
ad.datadoghq.com/checkout.logs: '[{"source":"python", "service": "checkout-service"}]'
spec:
replicas: 1
...
template:
metadata:
spec:
containers:
- name: checkout
...
env:
- name: DD_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: DD_SERVICE
value: checkout-service
- name: DD_TRACE_ANALYTICS_ENABLED
value: "true"
...
restartPolicy: Always
status: {}