> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://cla-demo.postmansolutions.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server.

# Telemetry and Monitoring Guide

## Overview

This comprehensive guide covers the complete observability stack for applications deployed on Azure Kubernetes Service (AKS), including:

1. **Infrastructure Deployment**: Setting up Azure Managed Prometheus and Grafana using Terraform

2. **Application Telemetry**: Enabling Application Insights logging in Node.js applications using the framework

3. **Enterprise Monitoring**: Integrating Dynatrace with Prometheus metrics from AKS clusters

By following this guide, you'll establish a complete observability pipeline that captures infrastructure metrics, application logs, and provides enterprise-grade monitoring capabilities.

---

## Table of Contents

* [Part 1: Deploying Prometheus and Grafana on Terraform](#part-1-deploying-prometheus-and-grafana-on-terraform)

\- [TL;DR](#tldr)

\- [Prerequisites](#prerequisites)

\- [Stage 1: Configure Monitoring Infrastructure](#stage-1-configure-monitoring-infrastructure)

\- [Stage 2: Configure Grafana Data Source Tracking](#stage-2-configure-grafana-data-source-tracking)

\- [Stage 3: Configure Team Permissions in Grafana](#stage-3-configure-team-permissions-in-grafana)

\- [Complete Workflow Summary](#complete-workflow-summary)

\- [Validation](#validation)

\- [Troubleshooting](#troubleshooting)

* [Part 2: Enabling Application Telemetry](#part-2-enabling-application-telemetry)

\- [Application Insights Configuration](#application-insights-configuration)

\- [Authentication Modes](#authentication-modes)

\- [Environment-Specific Configuration](#environment-specific-configuration)

* [Part 3: Integrating Dynatrace with Prometheus](#part-3-integrating-dynatrace-with-prometheus)

\- [Overview](#dynatrace-overview)

\- [Deployment Options](#deployment-options)

\- [Terraform Configuration](#terraform-configuration)

\- [Validation and Testing](#dynatrace-validation)

---

## Part 1: Deploying Prometheus and Grafana on Terraform

### TL;DR

**Quick Summary**: Enable Azure Managed Prometheus monitoring for AKS clusters in three stages:

**Stage 1 (Application/Operations Workspace)**: Update AKS module to v1.1.19+, create Azure Monitor Workspace (AMW), Data Collection Endpoint (DCE), Data Collection Rule (DCR), and associate with AKS cluster. Grant Grafana UAMI "Monitoring Data Reader" role. Optionally configure Prometheus recording rules.

**Stage 2 (DevOps Workspace)**: Add the AMW resource ID to the `prometheus_azure_monitor_workspaces` list in `az-devops-lz-tf/shared_services/config/prd.eastus2.auto.tfvars` to enable automatic Grafana data source creation.

**Stage 3 (Grafana Permissions)**: Configure team access by adding team definitions in `az-devops-lz-tf/grafana/config/prd.eastus2.auto.tfvars` with Entra ID group mappings and appropriate permissions (Query/Admin).

**Key Requirements**: AKS module ≥ v1.1.19, Grafana UAMI principal ID, access to both application and DevOps repositories.

This guide provides step-by-step instructions for enabling Azure Managed Prometheus monitoring on AKS clusters. This is a three-stage operation that requires configuration in both the application/operations workspace and the DevOps workspace.

### Prerequisites

* AKS cluster module version 1.1.19 or higher

* User Assigned Managed Identity (UAMI) for AKS

* Grafana workspace UAMI principal ID (for monitoring data reader role)

* Access to both application and DevOps landing zone repositories

### Stage 1: Configure Monitoring Infrastructure

#### Example Implementation

See the full working example in:

* **Repository**: `az-platform-lz-tf`

* **Path**: `operations/sandbox_shared_services/`

* **Files**:

\- `main.tf` (lines 349-446)

\- `variables.tf` (lines 315-338)

\- `config/sbx.eastus2.auto.tfvars` (lines 100+)

#### Step 1: Update Module Version

Ensure your AKS cluster module is at version 1.1.19 or higher and add the value `true` for `monitor_metrics_enabled`:

You can also include the optional variables `prometheus_labels_allowed` and `prometheus_annotations_allowed` for fine-grained control over metric metadata collection:

* **prometheus\_labels\_allowed**: Specifies which Kubernetes labels should be scraped and attached to Prometheus metrics. This controls the number of unique metric combinations by limiting the label dimensions collected. Use this when you need specific labels for filtering/grouping (e.g., app, environment, version) while excluding labels with many unique values that could cause storage and performance issues.

* **prometheus\_annotations\_allowed**: Specifies which Kubernetes annotations should be scraped and attached to Prometheus metrics. Annotations often contain deployment metadata, version information, or operational context. Use this to capture useful operational data (e.g., `deployment.kubernetes.io/revision`, custom deployment IDs) without collecting all annotations.

**Why use these variables?**

* **Reduce metric explosion**: Prevents too many unique metric combinations from labels with many unique values (e.g., pod IDs, timestamps, dynamic values)

* **Control costs**: Lower storage and query costs by limiting metadata dimensions

* **Improve performance**: Faster queries and reduced memory usage in Prometheus/Grafana

* **Selective monitoring**: Include only the labels/annotations relevant to your dashboards and alerts

**Example usage:**

```terraform

module "aks_cluster" {

  source  = "us.spacelift.io/claconnect/aks_cluster/azurerm"

  version = "~> 1.1.19"  # or higher

  

  monitor_metrics_enabled = true

  

  # Optional: Control which labels and annotations are scraped

  prometheus_labels_allowed = [

    "app",

    "environment",

    "component",

    "version"

  ]

  

  prometheus_annotations_allowed = [

    "deployment.kubernetes.io/revision",

    "prometheus.io/scrape",

    "prometheus.io/port"

  ]

  

  # ... existing configuration

}

```

#### Step 2: Add Required Variables

Add the following variables to your `variables.tf`:

```terraform

variable "grafana_uami_principal_id" {

  description = "The User Assigned Managed Identity principal ID for Grafana integration with Azure Monitor."

  type        = string

  default     = "a2f0c2f6-68c8-44ae-9a12-b6a5990f94cd"  # DevOps Grafana UAMI

}

  

variable "monitor_workspace_name" {

  description = "Name of the Azure Monitor Workspace for managed Prometheus."

  type        = string

  default     = null

}

  

variable "prometheus_rule_group_enabled" {

  description = "Whether Prometheus rule groups are enabled."

  type        = bool

  default     = true

}

  

variable "prometheus_rule_group_interval" {

  description = "Evaluation interval for Prometheus rule groups in ISO 8601 duration format (PT1M to PT15M)."

  type        = string

  default     = "PT1M"

}

```

#### Step 3: Create Azure Monitor Workspace (AMW)

Add the Azure Monitor Workspace module to your `main.tf`:

```terraform

module "monitor_workspace" {

  source  = "us.spacelift.io/claconnect/monitor_workspace/azurerm"

  version = "~> 1.0"

  

  # Required inputs

  location            = var.location

  resource_group_name = module.resource_group.name

  workspace_name      = var.monitor_workspace_name

}

```

#### Step 4: Create Data Collection Endpoint (DCE)

Add the Data Collection Endpoint module:

```terraform

module "monitor_data_collection_endpoint" {

  source  = "us.spacelift.io/claconnect/monitor_data_collection_endpoint/azurerm"

  version = "~> 1.0"

  

  # Required inputs

  location                              = var.location

  monitor_data_collection_endpoint_name = "${var.monitor_workspace_name}-dce"

  resource_group_name                   = module.resource_group.name

  kind                                  = "Linux"

}

```

#### Step 5: Create Data Collection Rule (DCR)

Add the Data Collection Rule module:

```terraform

module "monitor_data_collection_rule" {

  source  = "us.spacelift.io/claconnect/monitor_data_collection_rule/azurerm"

  version = "~> 1.0"

  

  # Required inputs

  data_collection_rule_name   = "${var.monitor_workspace_name}-dcr"

  data_collection_endpoint_id = module.monitor_data_collection_endpoint.id

  location                    = var.location

  resource_group_name         = module.resource_group.name

  tags                        = var.tags

  

  # Data flow configuration

  data_flow = [{

    streams      = ["Microsoft-PrometheusMetrics"]

    destinations = ["MonitoringAccount1"]

  }]

  

  # Data sources configuration

  data_sources = {

    prometheus_forwarder = [{

      streams = ["Microsoft-PrometheusMetrics"]

      name    = "PrometheusDataSource"

    }]

  }

  

  # Destinations configuration

  destinations = {

    monitor_account = [{

      monitor_account_id = module.monitor_workspace.id

      name               = "MonitoringAccount1"

    }]

  }

}

```

#### Step 6: Create Data Collection Rule Association (DCRA)

Associate the DCR with each AKS cluster:

```terraform

module "monitor_data_collection_rule_association" {

  source  = "us.spacelift.io/claconnect/monitor_data_collection_rule_association/azurerm"

  version = "~> 1.0"

  

  for_each = var.aks_clusters

  

  # Required inputs

  description             = "Association of data collection rule. Deleting this association will break the data collection for this AKS Cluster."

  name                    = "${each.key}-DCR-association"

  data_collection_rule_id = module.monitor_data_collection_rule.id

  target_resource_id      = module.aks_cluster[each.key].id

}

```

#### Step 7: Configure Grafana Role Assignment

Grant the Grafana UAMI access to read monitoring data:

```terraform

module "role_assignment_grafana" {

  source  = "us.spacelift.io/claconnect/role_assignment/azurerm"

  version = "1.1.0"

  

  principal_id    = var.grafana_uami_principal_id

  role_definition = "Monitoring Data Reader"

  scope           = module.monitor_workspace.id

}

```

#### Step 8: Configure Prometheus Rule Groups (Optional)

Add recording rules for node metrics (This is an example, use whatever is necessary for your apps needs):

```terraform

locals {

  # Node recording rules for Prometheus

  node_recording_rules = [

    {

      enabled    = true

      record     = "instance:node_num_cpu:sum"

      expression = "count without (cpu, mode) (  node_cpu_seconds_total{job=\"node\",mode=\"idle\"})"

    },

    {

      enabled    = true

      record     = "instance:node_cpu_utilisation:rate5m"

      expression = "1 - avg without (cpu) (  sum without (mode) (rate(node_cpu_seconds_total{job=\"node\", mode=~\"idle|iowait|steal\"}[5m])))"

    },

    {

      enabled    = true

      record     = "instance:node_load1_per_cpu:ratio"

      expression = "(  node_load1{job=\"node\"}/  instance:node_num_cpu:sum{job=\"node\"})"

    },

    {

      enabled    = true

      record     = "instance:node_memory_utilisation:ratio"

      expression = "1 - (  (    node_memory_MemAvailable_bytes{job=\"node\"}    or    (      node_memory_Buffers_bytes{job=\"node\"}      +      node_memory_Cached_bytes{job=\"node\"}      +      node_memory_MemFree_bytes{job=\"node\"}      +      node_memory_Slab_bytes{job=\"node\"}    )  )/  node_memory_MemTotal_bytes{job=\"node\"})"

    },

    {

      enabled    = true

      record     = "instance:node_vmstat_pgmajfault:rate5m"

      expression = "rate(node_vmstat_pgmajfault{job=\"node\"}[5m])"

    },

    {

      enabled    = true

      record     = "instance_device:node_disk_io_time_seconds:rate5m"

      expression = "rate(node_disk_io_time_seconds_total{job=\"node\", device!=\"\"}[5m])"

    },

    {

      enabled    = true

      record     = "instance_device:node_disk_io_time_weighted_seconds:rate5m"

      expression = "rate(node_disk_io_time_weighted_seconds_total{job=\"node\", device!=\"\"}[5m])"

    },

    {

      enabled    = true

      record     = "instance:node_network_receive_bytes_excluding_lo:rate5m"

      expression = "sum without (device) (  rate(node_network_receive_bytes_total{job=\"node\", device!=\"lo\"}[5m]))"

    },

    {

      enabled    = true

      record     = "instance:node_network_transmit_bytes_excluding_lo:rate5m"

      expression = "sum without (device) (  rate(node_network_transmit_bytes_total{job=\"node\", device!=\"lo\"}[5m]))"

    },

    {

      enabled    = true

      record     = "instance:node_network_receive_drop_excluding_lo:rate5m"

      expression = "sum without (device) (  rate(node_network_receive_drop_total{job=\"node\", device!=\"lo\"}[5m]))"

    },

    {

      enabled    = true

      record     = "instance:node_network_transmit_drop_excluding_lo:rate5m"

      expression = "sum without (device) (  rate(node_network_transmit_drop_total{job=\"node\", device!=\"lo\"}[5m]))"

    }

  ]

}

  

module "node_recording_rules" {

  source   = "us.spacelift.io/claconnect/monitor_alert_prometheus_rule_group/azurerm"

  version  = "1.1.0"

  for_each = var.aks_clusters

  

  # Required inputs

  name                = "NodeRecordingRules-${each.key}"

  location            = var.location

  resource_group_name = module.resource_group.name

  description         = "Node recording rules for ${each.key}"

  scopes              = [module.monitor_workspace.id, module.aks_cluster[each.key].id]

  

  # Optional inputs

  cluster_name       = each.key

  interval           = var.prometheus_rule_group_interval

  rule_group_enabled = var.prometheus_rule_group_enabled

  rules              = local.node_recording_rules

  tags               = var.tags

}

```

#### Step 9: Update Configuration File

Add the required values to your `.auto.tfvars` file:

```terraform

monitor_workspace_name = "sbx-eus2-prom-ops-amw-001"  # Customize per environment

  

# Optional: Override defaults if needed

prometheus_rule_group_enabled  = true

prometheus_rule_group_interval = "PT1M"

grafana_uami_principal_id      = "a2f0c2f6-68c8-44ae-9a12-b6a5990f94cd" ## Or leave the default in the variable.

```

#### Step 10: Apply the Configuration

Submit a PR and merge the code changes into the main branch to trigger a Spacelift run.

After successful deployment, note the Azure Monitor Workspace ID from the output. You will need this for Stage 2.

### Stage 2: Configure Grafana Data Source Tracking

⚠️ **IMPORTANT**: The Azure Monitor Workspace ID from Stage 1 must be merged into the DevOps shared services configuration to enable data source tracking in the Grafana workspace.

This step ensures that Grafana can discover and connect to your newly created Prometheus data source by integrating the Azure Monitor Workspace into the Grafana instance configuration.

#### Step 1: Locate the Azure Monitor Workspace ID

From Stage 1, retrieve the Azure Monitor Workspace resource ID. It will be in this format:

```

/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Monitor/accounts/{workspace-name}

```

**Example:**

```

/subscriptions/a3fde7ee-8b51-4aec-8f30-3751a39b3c14/resourceGroups/sbx-eus2-operations-shared-services-rg-001/providers/Microsoft.Monitor/accounts/sbx-eus2-prom-ops-amw-001

```

You can retrieve this from:

* **Terraform output**: `module.monitor_workspace.id`

* **Azure Portal**: Navigate to the AMW → Properties → Resource ID

* **Azure CLI**: `az monitor account show --name <workspace-name> --resource-group <rg-name> --query id`

#### Step 2: Create a Pull Request in az-devops-lz-tf

Navigate to the DevOps Landing Zone repository (`az-devops-lz-tf`) and locate the shared services configuration.

**File to Edit**: `shared_services/config/prd.eastus2.auto.tfvars`

Add your Azure Monitor Workspace ID to the `prometheus_azure_monitor_workspaces` list:

```terraform

prometheus_azure_monitor_workspaces = [

  # Existing workspaces

  "/subscriptions/a3fde7ee-8b51-4aec-8f30-3751a39b3c14/resourceGroups/sbx-eus2-operations-shared-services-rg-001/providers/Microsoft.Monitor/accounts/sbx-eus2-prom-ops-amw-001",

  

  # Add your new workspace here

  "/subscriptions/{your-subscription-id}/resourceGroups/{your-rg-name}/providers/Microsoft.Monitor/accounts/{your-workspace-name}"

]

```

**Important Notes:**

* This is a list, so multiple Azure Monitor Workspaces can be tracked

* Each workspace added here will automatically create a Grafana data source

* The data source will be named: `Managed_Prometheus_{workspace-name}`

* Ensure proper comma separation between list items

#### Step 3: Understand the Integration Flow

The Grafana module in `shared_services/main.tf` automatically integrates Azure Monitor Workspaces:

```terraform

module "dashboard_grafana" {

  source  = "us.spacelift.io/claconnect/dashboard_grafana/azurerm"

  version = "~> 1.0"

  

  # ... other configuration

  

  # This parameter links AMW IDs to Grafana

  azure_monitor_workspace_integrations = var.prometheus_azure_monitor_workspaces

}

```

When Terraform applies this configuration:

1. Grafana creates a managed data source for each AMW ID

2. The Grafana UAMI (configured in Stage 1) authenticates to the AMW

3. The data source becomes queryable in Grafana as `Managed_Prometheus_{workspace-name}`

#### Step 4: Verify Variable Definition

Ensure the variable exists in `shared_services/variables.tf`:

```terraform

variable "prometheus_azure_monitor_workspaces" {

  type        = list(string)

  description = "The resource IDs of the connected Azure Monitor Workspaces."

  default     = []

}

```

This variable should already exist in the DevOps shared services configuration. If not, add it.

#### Step 5: Submit the Pull Request

Create a PR with the following details:

**PR Title**: Add Azure Monitor Workspace for \{cluster-name} Prometheus Integration

**PR Description:**

```markdown

## Summary

  

Adds Azure Monitor Workspace integration for {cluster-name} to enable Prometheus metrics in Grafana.

  

## Changes

  

- Added Azure Monitor Workspace ID to `prometheus_azure_monitor_workspaces` list in `prd.eastus2.auto.tfvars`

  

## Azure Monitor Workspace Details

  

- **Name**: {workspace-name}

- **Subscription**: {subscription-name}

- **Resource Group**: {resource-group-name}

- **Region**: {region}

  

## Related Work

  

- Stage 1 completed in {application-lz-repo} PR #{pr-number}

- DCR, DCE, and DCRA configured for AKS cluster

- Grafana UAMI has Monitoring Reader role on AMW

  

## Testing

  

- [ ] Terraform plan shows Grafana module update

- [ ] No other resources affected

- [ ] After apply, verify data source appears in Grafana

```

#### Step 6: Apply and Verify

After the PR is merged and Terraform is applied:

**Check Grafana Data Sources:**

1. Navigate to Grafana → Configuration → Data Sources

2. Verify new data source: `Managed_Prometheus_{workspace-name}`

3. Click on the data source and select "Save & Test"

4. Should show: "Successfully queried the Azure Monitor Workspace"

**Test Prometheus Queries:**

1. Go to Explore in Grafana

2. Select your new data source

3. Run a test query: `up`

4. Should return metrics from your AKS cluster

**Verify Metrics Flow:**

1. Go to Dashboards → Azure Managed Prometheus

2. Select a Dashboard to confirm metrics visibility

#### Step 7: Configure Team Permissions (If Applicable)

If using the Grafana team management from earlier in this conversation, update the teams configuration in `az-devops-lz-tf/grafana/config/prd.eastus2.auto.tfvars`:

```terraform

teams = {

  "{cluster-team-name}" = {

    monitor_workspace_name = "{your-workspace-name}"

    external_groups        = ["{azure-ad-group-id}"]

    permissions = [

      {

        permission = "Admin"

      },

      {

        built_in_role = "Admin"

        permission = "Admin"

      }

    ]

  }

}

```

This ensures the appropriate teams have access to query the Prometheus data source in Grafana.

### Stage 3: Configure Team Permissions in Grafana

After the Azure Monitor Workspace has been integrated into Grafana (Stage 2), you need to configure team permissions to control access to the Prometheus data source.

#### Step 1: Identify Required Information

Before creating the PR, gather the following information:

* **Monitor Workspace Name**: From Stage 1 (e.g., `sbx-eus2-prom-ops-amw-001`)

* **Entra ID Group ID**: The Object ID of the Azure AD/Entra group that should have access

\- Navigate to Azure Portal → Entra ID → Groups

\- Find your group → Overview → Object ID

\- Example: `b76d9990-2716-4ead-8cef-abb3cff4e60d`

* **Team Name**: A descriptive name for the Grafana team (typically matches cluster or application name)

#### Step 2: Create Pull Request in az-devops-lz-tf

Navigate to the DevOps Landing Zone repository (`az-devops-lz-tf/grafana`) and locate the Grafana configuration.

**File to Edit**: `grafana/config/prd.eastus2.auto.tfvars`

Add a new team configuration to the `teams` map:

```terraform

teams = {

  # Existing teams

  "az-kt-sandbox-owners" = {

    monitor_workspace_name = "sbx-eus2-prom-ops-amw-001"

    external_groups        = ["b76d9990-2716-4ead-8cef-abb3cff4e60d"]

    permissions = [

      {

        permission = "Query"

      },

      {

        permission    = "Admin"

        built_in_role = "Admin"

      }

    ]

  },

  

  # Add your new team here

  "{your-team-name}" = {

    monitor_workspace_name = "{your-workspace-name}"

    external_groups        = ["{your-entra-group-id}"]

    permissions = [

      {

        permission = "Query"

      }

    ]

  }

}

```

#### Step 3: Configure Permission Levels

Choose the appropriate permission level for your team:

**Query Permission (Read-Only)**

```terraform

permissions = [

  {

    permission = "Query"

  }

]

```

* **Use case**: Teams that only need to view and query metrics

* **Access**: Can run queries, view dashboards, but cannot modify the data source

**Query + Built-in Role**

```terraform

permissions = [

  {

    permission    = "Query"

    built_in_role = "Viewer"  # Options: Viewer, Editor, Admin

  }

]

```

* **Use case**: Teams needing broader Grafana access beyond just the data source

* **Options**:

\- `Viewer`: Can view dashboards and data sources

\- `Editor`: Can create and edit dashboards

\- `Admin`: Full Grafana administration rights

**Admin Permission**

```terraform

permissions = [

  {

    permission = "Query"

  },

  {

    permission    = "Admin"

    built_in_role = "Admin"

  }

]

```

* **Use case**: Platform/DevOps teams managing the data source

* **Access**: Full control over the data source configuration

#### Step 4: Understand the Configuration Structure

The Grafana configuration creates the following resources:

```terraform

# Creates a Grafana team

module "grafana_team" {

  source  = "us.spacelift.io/claconnect/grafana_team/grafana"

  version = "~> 1.0"

  

  for_each = var.teams

  name     = each.key  # Team name from the map key

}

  

# Links Entra ID group to Grafana team

module "grafana_team_external_group" {

  source  = "us.spacelift.io/claconnect/grafana_team_external_group/grafana"

  version = "~> 1.0"

  

  for_each = var.teams

  team_id   = module.grafana_team[each.key].id

  group_ids = each.value.external_groups  # Your Entra group ID

}

  

# Grants team permissions to the data source

module "grafana_data_source_permissions" {

  source  = "us.spacelift.io/claconnect/grafana_data_source_permissions/grafana"

  version = "1.0.2"

  

  for_each = var.teams

  datasource_uid = data.grafana_data_source.data_source[each.key].uid

  permissions    = each.value.permissions

  team_id        = module.grafana_team[each.key].id

}

  

# Automatically looks up data source by workspace name

data "grafana_data_source" "data_source" {

  for_each = var.teams

  name     = "Managed_Prometheus_${each.value.monitor_workspace_name}"

}

```

**Key Points:**

* The team name (map key) becomes the Grafana team name

* The `monitor_workspace_name` is used to automatically find the correct data source

* Data source name format: `Managed_Prometheus_{workspace-name}`

* Multiple Entra ID groups can be linked to a single team

#### Step 5: Verify Configuration Prerequisites

Before submitting the PR, ensure:

* ✅ **Stage 2 Complete**: Azure Monitor Workspace is added to `shared_services/config/prd.eastus2.auto.tfvars`

* ✅ **Terraform Applied**: The Grafana data source exists in Grafana

* ✅ **Entra Group Exists**: The specified group ID is valid and accessible

* ✅ **Group Members**: Users are already members of the Entra ID group

#### Step 6: Submit the Pull Request

Create a PR with the following details:

**PR Title**: Add Grafana Team and Permissions for \{team-name}

**PR Description:**

```markdown

## Summary

  

Configures Grafana team and data source permissions for {team-name} to access Prometheus metrics from {cluster-name}.

  

## Changes

  

- Added team `{team-name}` to `grafana/config/prd.eastus2.auto.tfvars`

- Linked Entra ID group `{group-name}` ({group-id})

- Granted `Query` permission to data source `Managed_Prometheus_{workspace-name}`

  

## Team Configuration Details

  

- **Team Name**: {team-name}

- **Entra ID Group**: {group-name} ({group-id})

- **Workspace**: {workspace-name}

- **Permissions**: Query (read-only access)

  

## Prerequisites Completed

  

- [x] Azure Monitor Workspace integrated in Stage 2

- [x] Data source `Managed_Prometheus_{workspace-name}` exists in Grafana

- [x] Entra ID group created with appropriate members

- [x] Verified workspace name matches AMW resource

  

## Testing Plan

  

After apply:

  

- [ ] Verify team appears in Grafana Teams

- [ ] Confirm Entra group members can log into Grafana

- [ ] Test data source access with team member account

- [ ] Validate Query permissions work as expected

```

#### Step 7: Apply and Verify

After the PR is merged and Terraform is applied:

**Verify Team Creation:**

1. Log into Grafana as an admin

2. Navigate to Configuration → Teams

3. Confirm your new team exists

**Check Group Mapping:**

1. Click on the team name

2. Go to "External group sync"

3. Verify the Entra ID group is linked

**Verify Data Source Permissions:**

1. Navigate to Configuration → Data Sources

2. Select `Managed_Prometheus_{workspace-name}`

3. Click on "Permissions" tab

4. Confirm your team has the configured permissions

**Test User Access:**

1. Have a member of the Entra ID group log into Grafana

2. Navigate to Explore

3. Select the data source from the dropdown

4. Run a test query: `up{cluster="{cluster-name}"}`

5. Verify metrics are returned

**Validate Permission Restrictions:**

1. Confirm users can query but not modify the data source (if Query-only)

2. Test that users cannot access other data sources (if not granted)

#### Step 8: Troubleshooting Common Issues

**Data Source Not Found**

* **Error**: `[GET /datasources/name/{name}] getDataSourceByName (status 404): {}`

* **Resolution**:

\- Ensure Stage 2 is complete and Terraform has been applied

\- Verify the workspace name matches exactly (case-sensitive)

\- Check data source exists: Grafana → Configuration → Data Sources

**Entra Group Not Syncing**

* **Resolution**:

\- Verify the group Object ID is correct

\- Check Grafana has proper Entra ID integration configured

\- Wait 5-10 minutes for sync to occur

\- Users may need to log out and back in

**Permission Validation Error**

* **Error**: `Invalid value for "value" parameter: argument must not be null.`

* **Resolution**:

\- Ensure `built_in_role` is either omitted or set to a valid value

\- Valid values: `Viewer`, `Editor`, `Admin`

\- If using Query-only permission, omit `built_in_role` entirely

#### Step 9: Multiple Teams (Optional)

You can configure multiple teams for different access patterns:

```terraform

teams = {

  # Read-only access for developers

  "{cluster-name}-developers" = {

    monitor_workspace_name = "{workspace-name}"

    external_groups        = ["{developer-group-id}"]

    permissions = [

      {

        permission = "Query"

      }

    ]

  }

  

  # Admin access for platform team

  "{cluster-name}-platform" = {

    monitor_workspace_name = "{workspace-name}"

    external_groups        = ["{platform-team-group-id}"]

    permissions = [

      {

        permission = "Query"

      },

      {

        permission    = "Admin"

        built_in_role = "Admin"

      }

    ]

  }

  

  # Editor access for SRE team

  "{cluster-name}-sre" = {

    monitor_workspace_name = "{workspace-name}"

    external_groups        = ["{sre-group-id}"]

    permissions = [

      {

        permission    = "Query"

        built_in_role = "Editor"

      }

    ]

  }

}

```

Each team will:

* Have access to the same data source

* Have different permission levels

* Be linked to different Entra ID groups

* Appear as separate teams in Grafana

### Complete Workflow Summary

#### Stage 1: Application/Operations Landing Zone

Deploy monitoring infrastructure:

* Azure Monitor Workspace (AMW)

* Data Collection Endpoint (DCE)

* Data Collection Rule (DCR)

* Data Collection Rule Association (DCRA)

* Prometheus Rule Groups

* Grafana UAMI role assignment

#### Stage 2: DevOps Landing Zone - Shared Services

Integrate AMW with Grafana:

* Add AMW resource ID to `prometheus_azure_monitor_workspaces` list

* Grafana automatically creates managed data source

* Data source named: `Managed_Prometheus_{workspace-name}`

#### Stage 3: DevOps Landing Zone - Grafana

Configure team access:

* Create Grafana team

* Link Entra ID group to team

* Grant data source permissions to team

* Verify user access and permissions

**Result**: Teams can now access Prometheus metrics from AKS clusters through Grafana with appropriate permission levels based on their role.

### Validation

After completing all stages, verify the setup:

**Check Data Collection Rule Association:**

1. Navigate to your AKS cluster in Azure Portal

2. Go to Settings → Insights → Data Collection Settings

3. Verify the DCR association is present

**Verify Prometheus Metrics:**

1. Navigate to the Azure Monitor Workspace in Azure Portal

2. Query some basic metrics (e.g., `up`, `kube_node_info`)

**Check Grafana Data Source:**

1. Log into Grafana

2. Navigate to Configuration → Data Sources

3. Verify the `Managed_Prometheus_<workspace-name>` data source exists

4. Test the connection and run a simple query

**Verify Permissions:**

1. Ensure the Grafana UAMI has "Monitoring Reader" role on the AMW

2. Check role assignments in the AMW's Access Control (IAM) blade

### Troubleshooting

**No data appearing in Grafana**

* Verify the DCR association is created for the AKS cluster

* Check that the query endpoint URL in Grafana matches the AMW's query endpoint

* Ensure the Grafana UAMI has proper permissions

**Data Collection Rule issues**

* Verify the DCE is in the same region as the AKS cluster

* Check that the data flow, data sources, and destinations are correctly configured

* Review activity logs for any deployment errors

**Role assignment errors**

* Confirm the Grafana UAMI principal ID is correct

* Verify the UAMI has been properly created in the DevOps subscription

* Check for any RBAC propagation delays (can take 5-10 minutes)

### Additional Resources

* [Azure Monitor Workspace Documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/azure-monitor-workspace-overview)

* [Azure Managed Prometheus](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/prometheus-metrics-overview)

* [Data Collection Rules](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview)

---

## Part 2: Enabling Application Telemetry

This section covers how to configure Application Insights logging in your Node.js applications using the `framework-nodejs-logging` package version 2.1.3.

### Application Insights Configuration

Following the new pattern below, update the logging files in `/internal` to follow this pattern, using version `framework-nodejs-logging@2.1.3` with the **Direct Construction** method.

Test your settings by clearing npm cache (if needed): `npm cache clean --force` and running `npm install` from this git repository.

### Authentication Modes

The `AppInsightsLogConfigProfile` supports two authentication modes:

#### 1. Traditional Authentication (API Key)

Use this mode when you have an instrumentation key embedded in the connection string:

```typescript

import { Logger, ConfigProfilesEnum, resolveConfigProfileByName } from 'framework-nodejs-logging';

  

const profile = resolveConfigProfileByName(

  ConfigProfilesEnum.AppInsightsLogConfigProfile,

  'InstrumentationKey=xxx;IngestionEndpoint=https://xxx.applicationinsights.azure.com/'

);

  

const logger = Logger.create('info', profile);

logger.info('Hello from traditional auth');

```

#### 2. Managed Identity Authentication (Recommended for Azure)

Use this mode when deploying to Azure services with managed identity enabled. This is more secure as it eliminates the need to store API keys.

**System-Assigned Managed Identity:**

```typescript

import { Logger, ConfigProfilesEnum, resolveConfigProfileByName } from 'framework-nodejs-logging';

import { ManagedIdentityCredential } from '@azure/identity';

  

// Create credential for system-assigned managed identity

const credential = new ManagedIdentityCredential();

  

const profile = resolveConfigProfileByName(

  ConfigProfilesEnum.AppInsightsLogConfigProfile,

  'InstrumentationKey=xxx;IngestionEndpoint=https://xxx.applicationinsights.azure.com/',

  credential

);

  

const logger = Logger.create('info', profile);

logger.info('Hello from managed identity');

```

**User-Assigned Managed Identity:**

```typescript

import { ManagedIdentityCredential } from '@azure/identity';

  

// Provide the client ID for user-assigned managed identity

const credential = new ManagedIdentityCredential({

  clientId: 'your-user-assigned-identity-client-id'

});

  

const profile = resolveConfigProfileByName(

  ConfigProfilesEnum.AppInsightsLogConfigProfile,

  'InstrumentationKey=xxx;IngestionEndpoint=https://xxx.applicationinsights.azure.com/',

  credential

);

```

**Local Development:**

For local development, use `DefaultAzureCredential` which tries multiple authentication methods:

```typescript

import { DefaultAzureCredential } from '@azure/identity';

  

const credential = new DefaultAzureCredential();

  

const profile = resolveConfigProfileByName(

  ConfigProfilesEnum.AppInsightsLogConfigProfile,

  'InstrumentationKey=xxx;IngestionEndpoint=https://xxx.applicationinsights.azure.com/',

  credential

);

```

**Direct Construction:**

You can also create the profile directly:

```typescript

import { AppInsightsLogConfigProfile, Logger } from 'framework-nodejs-logging';

import { ManagedIdentityCredential } from '@azure/identity';

  

const connectionString = 'InstrumentationKey=xxx;IngestionEndpoint=https://xxx.applicationinsights.azure.com/';

const credential = new ManagedIdentityCredential();

const profile = new AppInsightsLogConfigProfile(connectionString, credential);

const logger = Logger.create('info', profile);

```

### Important Notes

1. **Connection String Format**: When using managed identity, your connection string **must include the `IngestionEndpoint`**. The library will validate this and throw an error if it's missing.

2. **Azure RBAC Permissions**: Ensure your managed identity has the **Monitoring Metrics Publisher** role assigned to the Application Insights resource.

3. **Environment-Specific Configuration**: It's recommended to use environment variables to configure authentication dynamically:

```typescript

import { ManagedIdentityCredential, DefaultAzureCredential } from '@azure/identity';

  

const connectionString = process.env.DOCAI_APPLICATIONINSIGHTS_CONNECTION_STRING!;

const useManagedIdentity = process.env.NODE_ENV === 'production';

  

// Use managed identity in production, local credentials in development

const credential = useManagedIdentity

  ? new ManagedIdentityCredential()

  : new DefaultAzureCredential();

  

const profile = resolveConfigProfileByName(

  ConfigProfilesEnum.AppInsightsLogConfigProfile,

  connectionString,

  credential

);

```

### Environment-Specific Configuration

**Recommended Pattern for Production Applications:**

```typescript

// src/internal/frameworkAdapter/registerLogger.ts

import { Logger, AppInsightsLogConfigProfile } from 'framework-nodejs-logging';

import { ManagedIdentityCredential, DefaultAzureCredential } from '@azure/identity';

  

function createLogger() {

  const connectionString = process.env.DOCAI_APPLICATIONINSIGHTS_CONNECTION_STRING;

  const environment = process.env.NODE_ENV || 'development';

  const logLevel = process.env.LOG_LEVEL || 'info';

  

  if (!connectionString) {

    throw new Error('DOCAI_APPLICATIONINSIGHTS_CONNECTION_STRING environment variable is required');

  }

  

  // Use managed identity in production, DefaultAzureCredential for development

  const credential = environment === 'production'

    ? new ManagedIdentityCredential()

    : new DefaultAzureCredential();

  

  const profile = new AppInsightsLogConfigProfile(connectionString, credential);

  return Logger.create(logLevel, profile);

}

  

export const logger = createLogger();

```

**Integration with Fastify:**

```typescript

// src/app.ts

import Fastify from 'fastify';

import { logger } from './internal/frameworkAdapter/registerLogger';

  

const app = Fastify({

  logger: logger,

  disableRequestLogging: false,

});

  

// Log incoming requests

app.addHook('onRequest', async (request, reply) => {

  logger.info({

    msg: 'Incoming request',

    method: request.method,

    url: request.url,

    headers: request.headers,

  });

});

  

// Log responses

app.addHook('onResponse', async (request, reply) => {

  logger.info({

    msg: 'Request completed',

    method: request.method,

    url: request.url,

    statusCode: reply.statusCode,

    responseTime: reply.getResponseTime(),

  });

});

  

export default app;

```

### Best Practices

1. **Never log sensitive data or PII** (Personally Identifiable Information)

2. **Use structured logging** with consistent field names

3. **Configure appropriate log levels** for different environments

4. **Monitor Application Insights ingestion** to control costs

5. **Use correlation IDs** to trace requests across services

6. **Set up alerts** for critical errors and performance thresholds

---

## Part 3: Integrating Dynatrace with Prometheus

### Dynatrace Overview

Dynatrace is an enterprise-grade observability platform that provides advanced monitoring, AI-powered analytics, and automated root cause analysis. By integrating Dynatrace with Azure Managed Prometheus, you can:

* **Centralize observability**: Consolidate Prometheus metrics with Dynatrace's full-stack monitoring

* **AI-powered insights**: Leverage Davis AI for automatic problem detection and root cause analysis

* **Advanced analytics**: Use Dynatrace's query language (DQL) for complex metric analysis

* **Unified dashboards**: Combine infrastructure, application, and business metrics in one platform

* **Enterprise features**: Access SLO tracking, service-level monitoring, and compliance reporting

### Architecture Overview

```

┌─────────────────────────────────────────────────────────────────┐

│                      Azure AKS Cluster                          │

│                                                                 │

│  ┌──────────────┐         ┌────────────────┐                  │

│  │  Application │────────►│   Prometheus   │                  │

│  │    Pods      │ metrics │   (Managed)    │                  │

│  └──────────────┘         └────────┬───────┘                  │

│                                     │                           │

│                           ┌─────────▼───────────┐              │

│                           │  Dynatrace OneAgent │              │

│                           │     (DaemonSet)     │              │

│                           └─────────┬───────────┘              │

└─────────────────────────────────────┼──────────────────────────┘

                                      │

                          ┌───────────▼───────────┐

                          │ Azure Monitor         │

                          │ Workspace (AMW)       │

                          └───────────┬───────────┘

                                      │

                          ┌───────────▼───────────┐

                          │  Dynatrace ActiveGate │

                          │  (Prometheus Scraper) │

                          └───────────┬───────────┘

                                      │

                          ┌───────────▼───────────┐

                          │   Dynatrace SaaS      │

                          │   (Cluster/Tenant)    │

                          └───────────────────────┘

```

### Deployment Options

There are two primary approaches to integrate Dynatrace with Prometheus on AKS:

#### Option 1: OneAgent + ActiveGate (Recommended)

**Pros:**

* Full-stack observability (infrastructure + application + Prometheus metrics)

* Automatic instrumentation of applications

* AI-powered problem detection

* Simplified deployment and management

**Cons:**

* Requires Dynatrace OneAgent license

* More resource consumption on cluster nodes

#### Option 2: ActiveGate Only (Metrics-focused)

**Pros:**

* Lower resource consumption

* Focused on metrics collection only

* Lower cost (doesn't require full OneAgent license)

**Cons:**

* No automatic application instrumentation

* Limited to Prometheus metrics only

* No full-stack visibility

### Terraform Configuration

#### Prerequisites

Before implementing Dynatrace integration, ensure you have:

1. **Dynatrace Tenant/Cluster**: Active Dynatrace SaaS or Managed environment

2. **Dynatrace API Token**: With the following scopes:

\- `activeGateTokenManagement.create`

\- `entities.read`

\- `metrics.ingest`

\- `InstallerDownload`

3. **Dynatrace PaaS Token**: For OneAgent deployment

4. **Azure Monitor Workspace**: Deployed from Part 1 of this guide

5. **Terraform Access**: Permissions to deploy resources in AKS cluster

#### Step 1: Add Required Variables

Add Dynatrace-specific variables to your `variables.tf`:

```terraform

variable "dynatrace_enabled" {

  description = "Enable Dynatrace integration for Prometheus metrics"

  type        = bool

  default     = false

}

  

variable "dynatrace_environment_id" {

  description = "Dynatrace environment ID (e.g., abc12345)"

  type        = string

  default     = ""

}

  

variable "dynatrace_api_url" {

  description = "Dynatrace API URL (e.g., https://abc12345.live.dynatrace.com/api)"

  type        = string

  default     = ""

}

  

variable "dynatrace_api_token" {

  description = "Dynatrace API token for authentication"

  type        = string

  sensitive   = true

  default     = ""

}

  

variable "dynatrace_paas_token" {

  description = "Dynatrace PaaS token for OneAgent deployment"

  type        = string

  sensitive   = true

  default     = ""

}

  

variable "dynatrace_oneagent_enabled" {

  description = "Deploy Dynatrace OneAgent for full-stack monitoring"

  type        = bool

  default     = true

}

  

variable "dynatrace_activegate_enabled" {

  description = "Deploy Dynatrace ActiveGate for Prometheus scraping"

  type        = bool

  default     = true

}

  

variable "prometheus_remote_write_enabled" {

  description = "Enable Prometheus remote write to Dynatrace"

  type        = bool

  default     = true

}

```

#### Step 2: Create Dynatrace Namespace

Create a dedicated namespace for Dynatrace components:

```terraform

resource "kubernetes_namespace" "dynatrace" {

  count = var.dynatrace_enabled ? 1 : 0

  

  metadata {

    name = "dynatrace"

    labels = {

      "managed-by" = "terraform"

      "purpose"    = "monitoring"

    }

  }

  

  depends_on = [module.aks_cluster]

}

```

#### Step 3: Create Kubernetes Secrets for Dynatrace

Store Dynatrace tokens securely in Kubernetes secrets:

```terraform

resource "kubernetes_secret" "dynatrace_tokens" {

  count = var.dynatrace_enabled ? 1 : 0

  

  metadata {

    name      = "dynatrace-tokens"

    namespace = kubernetes_namespace.dynatrace[0].metadata[0].name

  }

  

  data = {

    api-token  = var.dynatrace_api_token

    paas-token = var.dynatrace_paas_token

  }

  

  type = "Opaque"

}

```

#### Step 4: Deploy Dynatrace OneAgent (Optional)

Deploy OneAgent as a DaemonSet for full-stack monitoring:

```terraform

resource "helm_release" "dynatrace_oneagent" {

  count = var.dynatrace_enabled && var.dynatrace_oneagent_enabled ? 1 : 0

  

  name       = "dynatrace-oneagent"

  repository = "https://raw.githubusercontent.com/Dynatrace/dynatrace-operator/main/config/helm/repos/stable"

  chart      = "dynatrace-operator"

  version    = "0.14.2"  # Use latest stable version

  namespace  = kubernetes_namespace.dynatrace[0].metadata[0].name

  

  values = [

    yamlencode({

      platform = "kubernetes"

      operator = {

        image = "dynatrace/dynatrace-operator"

      }

  

      oneAgent = {

        enabled = true

        name    = "oneagent"

        apiUrl = var.dynatrace_api_url

        # Reference the secret created above

        tokens = kubernetes_secret.dynatrace_tokens[0].metadata[0].name

  

        # Resource limits

        resources = {

          requests = {

            cpu    = "100m"

            memory = "512Mi"

          }

          limits = {

            cpu    = "500m"

            memory = "1Gi"

          }

        }

  

        # Environment-specific settings

        env = [

          {

            name  = "ONEAGENT_ENABLE_VOLUME_STORAGE"

            value = "true"

          }

        ]

  

        # Node selector for specific node pools (optional)

        nodeSelector = {

          "kubernetes.io/os" = "linux"

        }

  

        # Tolerations for node taints (optional)

        tolerations = []

      }

  

      # Enable Kubernetes monitoring

      kubernetesMonitoring = {

        enabled = true

      }

    })

  ]

  

  depends_on = [

    kubernetes_namespace.dynatrace,

    kubernetes_secret.dynatrace_tokens

  ]

}

```

#### Step 5: Deploy Dynatrace ActiveGate

Deploy ActiveGate for Prometheus metrics scraping:

```terraform

resource "kubernetes_deployment" "dynatrace_activegate" {

  count = var.dynatrace_enabled && var.dynatrace_activegate_enabled ? 1 : 0

  

  metadata {

    name      = "dynatrace-activegate"

    namespace = kubernetes_namespace.dynatrace[0].metadata[0].name

    labels = {

      app = "dynatrace-activegate"

    }

  }

  

  spec {

    replicas = 1

  

    selector {

      match_labels = {

        app = "dynatrace-activegate"

      }

    }

  

    template {

      metadata {

        labels = {

          app = "dynatrace-activegate"

        }

      }

  

      spec {

        container {

          name  = "activegate"

          image = "dynatrace/dynatrace-activegate:latest"

  

          env {

            name  = "DT_ID"

            value = var.dynatrace_environment_id

          }

  

          env {

            name = "DT_TENANT_TOKEN"

            value_from {

              secret_key_ref {

                name = kubernetes_secret.dynatrace_tokens[0].metadata[0].name

                key  = "api-token"

              }

            }

          }

  

          env {

            name  = "DT_CAPABILITIES"

            value = "restInterface,kubernetes_monitoring,metrics_ingest"

          }

  

          resources {

            requests = {

              cpu    = "500m"

              memory = "1Gi"

            }

            limits = {

              cpu    = "1000m"

              memory = "2Gi"

            }

          }

  

          port {

            name           = "https"

            container_port = 9999

            protocol       = "TCP"

          }

  

          liveness_probe {

            http_get {

              path   = "/rest/health"

              port   = 9999

              scheme = "HTTPS"

            }

            initial_delay_seconds = 90

            period_seconds        = 15

            timeout_seconds       = 5

          }

  

          readiness_probe {

            http_get {

              path   = "/rest/health"

              port   = 9999

              scheme = "HTTPS"

            }

            initial_delay_seconds = 30

            period_seconds        = 10

            timeout_seconds       = 5

          }

        }

      }

    }

  }

  

  depends_on = [

    kubernetes_namespace.dynatrace,

    kubernetes_secret.dynatrace_tokens

  ]

}

```

#### Step 6: Create ActiveGate Service

Expose ActiveGate service for internal access:

```terraform

resource "kubernetes_service" "dynatrace_activegate" {

  count = var.dynatrace_enabled && var.dynatrace_activegate_enabled ? 1 : 0

  

  metadata {

    name      = "dynatrace-activegate"

    namespace = kubernetes_namespace.dynatrace[0].metadata[0].name

  }

  

  spec {

    selector = {

      app = "dynatrace-activegate"

    }

  

    port {

      name        = "https"

      port        = 9999

      target_port = 9999

      protocol    = "TCP"

    }

  

    type = "ClusterIP"

  }

  

  depends_on = [kubernetes_deployment.dynatrace_activegate]

}

```

#### Step 7: Configure Prometheus Remote Write

Configure Azure Managed Prometheus to send metrics to Dynatrace:

```terraform

resource "kubernetes_config_map" "prometheus_remote_write" {

  count = var.dynatrace_enabled && var.prometheus_remote_write_enabled ? 1 : 0

  

  metadata {

    name      = "prometheus-remote-write-config"

    namespace = "kube-system"

  }

  

  data = {

    "remote-write.yaml" = yamlencode({

      remote_write = [

        {

          url = "${var.dynatrace_api_url}/v2/metrics/ingest"

          bearer_token_file = "/etc/prometheus/secrets/dynatrace-tokens/api-token"

          queue_config = {

            capacity          = 10000

            max_shards        = 50

            min_shards        = 1

            max_samples_per_send = 5000

            batch_send_deadline  = "5s"

            min_backoff          = "30ms"

            max_backoff          = "100ms"

          }

  

          write_relabel_configs = [

            {

              source_labels = ["__name__"]

              regex         = "up|container_.*|kube_.*|node_.*"

              action        = "keep"

            }

          ]

        }

      ]

    })

  }

  

  depends_on = [kubernetes_secret.dynatrace_tokens]

}

```

#### Step 8: Update Configuration File

Add Dynatrace-specific values to your `.auto.tfvars` file:

```terraform

# Dynatrace Configuration

dynatrace_enabled             = true

dynatrace_environment_id      = "abc12345"  # Your Dynatrace environment ID

dynatrace_api_url             = "https://abc12345.live.dynatrace.com/api"

dynatrace_oneagent_enabled    = true

dynatrace_activegate_enabled  = true

prometheus_remote_write_enabled = true

  

# Store tokens in Azure Key Vault or use Terraform Cloud variables

# dynatrace_api_token  = "dt0c01.xxx"  # Reference from secure storage

# dynatrace_paas_token = "dt0c01.yyy"  # Reference from secure storage

```

#### Step 9: Secure Secrets Management

**Recommended Approach: Azure Key Vault Integration**

```terraform

# Reference secrets from Azure Key Vault

data "azurerm_key_vault_secret" "dynatrace_api_token" {

  name         = "dynatrace-api-token"

  key_vault_id = data.azurerm_key_vault.secrets.id

}

  

data "azurerm_key_vault_secret" "dynatrace_paas_token" {

  name         = "dynatrace-paas-token"

  key_vault_id = data.azurerm_key_vault.secrets.id

}

  

# Use in variables

locals {

  dynatrace_api_token  = data.azurerm_key_vault_secret.dynatrace_api_token.value

  dynatrace_paas_token = data.azurerm_key_vault_secret.dynatrace_paas_token.value

}

```

#### Step 10: Apply Configuration

1. **Store secrets in Azure Key Vault**:

```bash

az keyvault secret set --vault-name <vault-name> --name dynatrace-api-token --value "dt0c01.xxx"

az keyvault secret set --vault-name <vault-name> --name dynatrace-paas-token --value "dt0c01.yyy"

```

2. **Create a PR** with all Dynatrace configuration

3. **Merge to main branch** to trigger Spacelift run

4. **Monitor deployment** in Terraform output

### Dynatrace Validation

After deployment, verify the integration:

#### Step 1: Verify OneAgent Deployment

```bash

# Check OneAgent pods are running

kubectl get pods -n dynatrace -l app.kubernetes.io/name=oneagent

  

# Check OneAgent logs

kubectl logs -n dynatrace -l app.kubernetes.io/name=oneagent --tail=50

```

Expected output: OneAgent pods should be in `Running` state on all nodes.

#### Step 2: Verify ActiveGate Deployment

```bash

# Check ActiveGate pod status

kubectl get pods -n dynatrace -l app=dynatrace-activegate

  

# Check ActiveGate logs

kubectl logs -n dynatrace -l app=dynatrace-activegate --tail=50

  

# Test ActiveGate health endpoint

kubectl exec -n dynatrace -it <activegate-pod> -- curl -k https://localhost:9999/rest/health

```

Expected output: ActiveGate should report healthy status.

#### Step 3: Verify Dynatrace UI

1. **Log into Dynatrace**: Navigate to your Dynatrace tenant

2. **Check Hosts**: Infrastructure → Hosts → Verify AKS nodes appear

3. **Check Kubernetes**: Kubernetes → Clusters → Find your AKS cluster

4. **Check Metrics**: Metrics → Explore metrics → Search for Prometheus metrics

5. **Verify Ingestion**: Settings → Monitoring → Monitored entities → Kubernetes

#### Step 4: Query Prometheus Metrics in Dynatrace

1. Navigate to **Metrics** in Dynatrace

2. Search for common Prometheus metrics:

\- `up`: Monitor all targets

\- `container_cpu_usage_seconds_total`: Container CPU metrics

\- `kube_pod_info`: Kubernetes pod information

\- `node_memory_MemAvailable_bytes`: Node memory metrics

3. Create custom dashboards combining:

\- Prometheus infrastructure metrics

\- Application Insights telemetry

\- Dynatrace AI insights

#### Step 5: Configure Dynatrace Alerting

Create alerting rules in Dynatrace:

1. **Navigate to Settings** → Anomaly detection → Custom events for alerting

2. **Create metric events** for Prometheus metrics:

**Example: High CPU Alert**

```yaml

Metric: container_cpu_usage_seconds_total

Aggregation: Average

Threshold: > 80%

Duration: 5 minutes

Severity: Resource

Alert on: All matching entities

```

**Example: Pod Crash Alert**

```yaml

Metric: kube_pod_container_status_restarts_total

Aggregation: Rate

Threshold: > 5 restarts per minute

Duration: 2 minutes

Severity: Availability

Alert on: All matching entities

```

### Advanced Configuration

#### Metric Filtering

To control which Prometheus metrics are sent to Dynatrace, configure metric filters:

```terraform

write_relabel_configs = [

  # Keep only specific metric prefixes

  {

    source_labels = ["__name__"]

    regex         = "(up|container_.*|kube_pod_.*|kube_node_.*|node_cpu_.*|node_memory_.*)"

    action        = "keep"

  },

  # Drop high-cardinality metrics

  {

    source_labels = ["__name__"]

    regex         = ".*_bucket"

    action        = "drop"

  },

  # Add custom labels

  {

    target_label = "cluster_name"

    replacement  = "prod-aks-cluster"

  }

]

```

#### Custom Dashboards

Create Dynatrace dashboards combining multiple data sources:

1. **Infrastructure View**: Combine Prometheus node metrics with Dynatrace host monitoring

2. **Application Performance**: Merge Application Insights traces with Kubernetes metrics

3. **Business Metrics**: Correlate application logs with infrastructure health

#### SLO Configuration

Define Service Level Objectives in Dynatrace:

1. **Availability SLO**:

\- Metric: `up{job="kubernetes-nodes"}`

\- Target: 99.9% availability

\- Timeframe: 30 days

2. **Performance SLO**:

\- Metric: `http_request_duration_seconds`

\- Target: 95% of requests \< 500ms

\- Timeframe: 7 days

### Cost Optimization

To optimize Dynatrace costs:

1. **Filter Metrics**: Only send high-value metrics to Dynatrace

2. **Use Recording Rules**: Pre-aggregate metrics in Prometheus before sending

3. **Adjust Retention**: Configure appropriate retention periods in Dynatrace

4. **Monitor DPM Usage**: Track Davis Data Units (DDU) consumption

5. **Leverage Sampling**: Use trace sampling for high-volume applications

### Troubleshooting Dynatrace Integration

**OneAgent Not Reporting**

* **Verify tokens**: Check API and PaaS tokens are valid

* **Check connectivity**: Ensure AKS can reach Dynatrace endpoints

* **Review logs**: `kubectl logs -n dynatrace -l app.kubernetes.io/name=oneagent`

* **Validate RBAC**: Ensure OneAgent has proper Kubernetes permissions

**ActiveGate Not Scraping Metrics**

* **Check configuration**: Verify Prometheus endpoint is accessible

* **Test connectivity**: Use `curl` from ActiveGate pod to AMW query endpoint

* **Review capabilities**: Ensure `metrics_ingest` capability is enabled

* **Check firewall rules**: Verify network security groups allow traffic

**Metrics Not Appearing in Dynatrace**

* **Verify ingestion**: Check Dynatrace Settings → Monitoring overview

* **Review filters**: Ensure `write_relabel_configs` aren't dropping metrics

* **Check metric names**: Dynatrace may transform Prometheus metric names

* **Validate credentials**: Confirm API token has `metrics.ingest` scope

**High DDU Consumption**

* **Analyze metric cardinality**: Identify high-cardinality metrics

* **Implement filtering**: Use `write_relabel_configs` to drop unnecessary metrics

* **Use recording rules**: Pre-aggregate metrics to reduce data points

* **Review retention policies**: Adjust retention based on requirements

### Security Considerations

1. **Token Management**:

\- Store Dynatrace tokens in Azure Key Vault

\- Rotate tokens regularly (every 90 days)

\- Use separate tokens for different environments

2. **Network Security**:

\- Use Private Link for Dynatrace connectivity (if available)

\- Implement network policies to restrict Dynatrace component communication

\- Enable TLS for all Dynatrace communications

3. **RBAC**:

\- Grant minimal Kubernetes permissions to Dynatrace components

\- Use dedicated service accounts for OneAgent and ActiveGate

\- Regular audit of Dynatrace access permissions

4. **Data Privacy**:

\- Configure metric relabeling to strip sensitive labels

\- Review Dynatrace data residency requirements

\- Implement data masking for PII in logs

### Best Practices Summary

1. **Start with ActiveGate Only**: Test Prometheus integration before deploying full OneAgent

2. **Use Managed Identity**: Leverage Azure managed identities where possible

3. **Monitor Resource Usage**: Track OneAgent and ActiveGate resource consumption

4. **Implement Gradual Rollout**: Deploy to sandbox/dev environments first

5. **Create Runbooks**: Document troubleshooting procedures for common issues

6. **Set Up Alerting**: Configure alerts for Dynatrace component health

7. **Regular Reviews**: Periodically review metrics being sent to optimize costs

8. **Leverage AI**: Use Dynatrace Davis AI for anomaly detection and root cause analysis

---

## Conclusion

This comprehensive guide has covered:

1. **Infrastructure Setup**: Deploying Azure Managed Prometheus and Grafana using Terraform with proper RBAC and team permissions

2. **Application Telemetry**: Configuring Application Insights with managed identity authentication for secure logging

3. **Enterprise Monitoring**: Integrating Dynatrace with Prometheus for AI-powered observability and advanced analytics

By following these three parts, you've established a complete, enterprise-grade observability stack that provides:

* **Infrastructure monitoring** through Prometheus and Grafana

* **Application insights** through Application Insights and structured logging

* **AI-powered analytics** through Dynatrace integration

* **Unified visibility** across the entire stack

This architecture enables your teams to proactively monitor, troubleshoot, and optimize applications running on Azure AKS with industry-leading tools and best practices.

### Next Steps

1. **Create Custom Dashboards**: Build team-specific dashboards in both Grafana and Dynatrace

2. **Set Up Alerting**: Configure alerts based on your SLOs and operational requirements

3. **Implement SLOs**: Define and track Service Level Objectives across your services

4. **Optimize Costs**: Regular review of metrics ingestion and storage costs

5. **Train Teams**: Conduct training sessions on using the observability tools effectively

6. **Document Runbooks**: Create operational runbooks for common scenarios

### Additional Resources

* [Azure Monitor Documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/)

* [Dynatrace Kubernetes Monitoring](https://www.dynatrace.com/support/help/technology-support/cloud-platforms/kubernetes)

* [Prometheus Best Practices](https://prometheus.io/docs/practices/)

* [Grafana Dashboard Gallery](https://grafana.com/grafana/dashboards/)

* [Application Insights Documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview)