Telemetry and Monitoring Guide

Overview

This comprehensive guide covers the complete observability stack for applications deployed on Azure Kubernetes Service (AKS), including:

1. Infrastructure Deployment: Setting up Azure Managed Prometheus and Grafana using Terraform

2. Application Telemetry: Enabling Application Insights logging in Node.js applications using the framework

3. Enterprise Monitoring: Integrating Dynatrace with Prometheus metrics from AKS clusters

By following this guide, you’ll establish a complete observability pipeline that captures infrastructure metrics, application logs, and provides enterprise-grade monitoring capabilities.

Table of Contents

• Part 1: Deploying Prometheus and Grafana on Terraform

  - TL;DR

  - Prerequisites

  - Stage 1: Configure Monitoring Infrastructure

  - Stage 2: Configure Grafana Data Source Tracking

  - Stage 3: Configure Team Permissions in Grafana

  - Complete Workflow Summary

  - Validation

  - Troubleshooting

• Part 2: Enabling Application Telemetry

  - Application Insights Configuration

  - Authentication Modes

  - Environment-Specific Configuration

• Part 3: Integrating Dynatrace with Prometheus

  - Overview

  - Deployment Options

  - Terraform Configuration

  - Validation and Testing

Part 1: Deploying Prometheus and Grafana on Terraform

TL;DR

Quick Summary: Enable Azure Managed Prometheus monitoring for AKS clusters in three stages:

Stage 1 (Application/Operations Workspace): Update AKS module to v1.1.19+, create Azure Monitor Workspace (AMW), Data Collection Endpoint (DCE), Data Collection Rule (DCR), and associate with AKS cluster. Grant Grafana UAMI “Monitoring Data Reader” role. Optionally configure Prometheus recording rules.

Stage 2 (DevOps Workspace): Add the AMW resource ID to the prometheus_azure_monitor_workspaces list in az-devops-lz-tf/shared_services/config/prd.eastus2.auto.tfvars to enable automatic Grafana data source creation.

Stage 3 (Grafana Permissions): Configure team access by adding team definitions in az-devops-lz-tf/grafana/config/prd.eastus2.auto.tfvars with Entra ID group mappings and appropriate permissions (Query/Admin).

Key Requirements: AKS module ≥ v1.1.19, Grafana UAMI principal ID, access to both application and DevOps repositories.

This guide provides step-by-step instructions for enabling Azure Managed Prometheus monitoring on AKS clusters. This is a three-stage operation that requires configuration in both the application/operations workspace and the DevOps workspace.

Prerequisites

• AKS cluster module version 1.1.19 or higher

• User Assigned Managed Identity (UAMI) for AKS

• Grafana workspace UAMI principal ID (for monitoring data reader role)

• Access to both application and DevOps landing zone repositories

Stage 1: Configure Monitoring Infrastructure

Example Implementation

See the full working example in:

• Repository: az-platform-lz-tf

• Path: operations/sandbox_shared_services/

• Files:

  - main.tf (lines 349-446)

  - variables.tf (lines 315-338)

  - config/sbx.eastus2.auto.tfvars (lines 100+)

Step 1: Update Module Version

Ensure your AKS cluster module is at version 1.1.19 or higher and add the value true for monitor_metrics_enabled:

You can also include the optional variables prometheus_labels_allowed and prometheus_annotations_allowed for fine-grained control over metric metadata collection:

• prometheus_labels_allowed: Specifies which Kubernetes labels should be scraped and attached to Prometheus metrics. This controls the number of unique metric combinations by limiting the label dimensions collected. Use this when you need specific labels for filtering/grouping (e.g., app, environment, version) while excluding labels with many unique values that could cause storage and performance issues.

• prometheus_annotations_allowed: Specifies which Kubernetes annotations should be scraped and attached to Prometheus metrics. Annotations often contain deployment metadata, version information, or operational context. Use this to capture useful operational data (e.g., deployment.kubernetes.io/revision, custom deployment IDs) without collecting all annotations.

Why use these variables?

• Reduce metric explosion: Prevents too many unique metric combinations from labels with many unique values (e.g., pod IDs, timestamps, dynamic values)

• Control costs: Lower storage and query costs by limiting metadata dimensions

• Improve performance: Faster queries and reduced memory usage in Prometheus/Grafana

• Selective monitoring: Include only the labels/annotations relevant to your dashboards and alerts

Example usage:

module "aks_cluster" {
  source  = "us.spacelift.io/claconnect/aks_cluster/azurerm"
  version = "~> 1.1.19"  # or higher
  
  monitor_metrics_enabled = true
  
  # Optional: Control which labels and annotations are scraped
  prometheus_labels_allowed = [
    "app",
    "environment",
    "component",
    "version"
  ]
  
  prometheus_annotations_allowed = [
    "deployment.kubernetes.io/revision",
    "prometheus.io/scrape",
    "prometheus.io/port"
  ]
  
  # ... existing configuration
}

Step 2: Add Required Variables

Add the following variables to your variables.tf:

variable "grafana_uami_principal_id" {
  description = "The User Assigned Managed Identity principal ID for Grafana integration with Azure Monitor."
  type        = string
  default     = "a2f0c2f6-68c8-44ae-9a12-b6a5990f94cd"  # DevOps Grafana UAMI
}
  
variable "monitor_workspace_name" {
  description = "Name of the Azure Monitor Workspace for managed Prometheus."
  type        = string
  default     = null
}
  
variable "prometheus_rule_group_enabled" {
  description = "Whether Prometheus rule groups are enabled."
  type        = bool
  default     = true
}
  
variable "prometheus_rule_group_interval" {
  description = "Evaluation interval for Prometheus rule groups in ISO 8601 duration format (PT1M to PT15M)."
  type        = string
  default     = "PT1M"
}

Step 3: Create Azure Monitor Workspace (AMW)

Add the Azure Monitor Workspace module to your main.tf:

module "monitor_workspace" {
  source  = "us.spacelift.io/claconnect/monitor_workspace/azurerm"
  version = "~> 1.0"
  
  # Required inputs
  location            = var.location
  resource_group_name = module.resource_group.name
  workspace_name      = var.monitor_workspace_name
}

Step 4: Create Data Collection Endpoint (DCE)

Add the Data Collection Endpoint module:

module "monitor_data_collection_endpoint" {
  source  = "us.spacelift.io/claconnect/monitor_data_collection_endpoint/azurerm"
  version = "~> 1.0"
  
  # Required inputs
  location                              = var.location
  monitor_data_collection_endpoint_name = "${var.monitor_workspace_name}-dce"
  resource_group_name                   = module.resource_group.name
  kind                                  = "Linux"
}

Step 5: Create Data Collection Rule (DCR)

Add the Data Collection Rule module:

module "monitor_data_collection_rule" {
  source  = "us.spacelift.io/claconnect/monitor_data_collection_rule/azurerm"
  version = "~> 1.0"
  
  # Required inputs
  data_collection_rule_name   = "${var.monitor_workspace_name}-dcr"
  data_collection_endpoint_id = module.monitor_data_collection_endpoint.id
  location                    = var.location
  resource_group_name         = module.resource_group.name
  tags                        = var.tags
  
  # Data flow configuration
  data_flow = [{
    streams      = ["Microsoft-PrometheusMetrics"]
    destinations = ["MonitoringAccount1"]
  }]
  
  # Data sources configuration
  data_sources = {
    prometheus_forwarder = [{
      streams = ["Microsoft-PrometheusMetrics"]
      name    = "PrometheusDataSource"
    }]
  }
  
  # Destinations configuration
  destinations = {
    monitor_account = [{
      monitor_account_id = module.monitor_workspace.id
      name               = "MonitoringAccount1"
    }]
  }
}

Step 6: Create Data Collection Rule Association (DCRA)

Associate the DCR with each AKS cluster:

module "monitor_data_collection_rule_association" {
  source  = "us.spacelift.io/claconnect/monitor_data_collection_rule_association/azurerm"
  version = "~> 1.0"
  
  for_each = var.aks_clusters
  
  # Required inputs
  description             = "Association of data collection rule. Deleting this association will break the data collection for this AKS Cluster."
  name                    = "${each.key}-DCR-association"
  data_collection_rule_id = module.monitor_data_collection_rule.id
  target_resource_id      = module.aks_cluster[each.key].id
}

Step 7: Configure Grafana Role Assignment

Grant the Grafana UAMI access to read monitoring data:

module "role_assignment_grafana" {
  source  = "us.spacelift.io/claconnect/role_assignment/azurerm"
  version = "1.1.0"
  
  principal_id    = var.grafana_uami_principal_id
  role_definition = "Monitoring Data Reader"
  scope           = module.monitor_workspace.id
}

Step 8: Configure Prometheus Rule Groups (Optional)

Add recording rules for node metrics (This is an example, use whatever is necessary for your apps needs):

locals {
  # Node recording rules for Prometheus
  node_recording_rules = [
    {
      enabled    = true
      record     = "instance:node_num_cpu:sum"
      expression = "count without (cpu, mode) (  node_cpu_seconds_total{job=\"node\",mode=\"idle\"})"
    },
    {
      enabled    = true
      record     = "instance:node_cpu_utilisation:rate5m"
      expression = "1 - avg without (cpu) (  sum without (mode) (rate(node_cpu_seconds_total{job=\"node\", mode=~\"idle|iowait|steal\"}[5m])))"
    },
    {
      enabled    = true
      record     = "instance:node_load1_per_cpu:ratio"
      expression = "(  node_load1{job=\"node\"}/  instance:node_num_cpu:sum{job=\"node\"})"
    },
    {
      enabled    = true
      record     = "instance:node_memory_utilisation:ratio"
      expression = "1 - (  (    node_memory_MemAvailable_bytes{job=\"node\"}    or    (      node_memory_Buffers_bytes{job=\"node\"}      +      node_memory_Cached_bytes{job=\"node\"}      +      node_memory_MemFree_bytes{job=\"node\"}      +      node_memory_Slab_bytes{job=\"node\"}    )  )/  node_memory_MemTotal_bytes{job=\"node\"})"
    },
    {
      enabled    = true
      record     = "instance:node_vmstat_pgmajfault:rate5m"
      expression = "rate(node_vmstat_pgmajfault{job=\"node\"}[5m])"
    },
    {
      enabled    = true
      record     = "instance_device:node_disk_io_time_seconds:rate5m"
      expression = "rate(node_disk_io_time_seconds_total{job=\"node\", device!=\"\"}[5m])"
    },
    {
      enabled    = true
      record     = "instance_device:node_disk_io_time_weighted_seconds:rate5m"
      expression = "rate(node_disk_io_time_weighted_seconds_total{job=\"node\", device!=\"\"}[5m])"
    },
    {
      enabled    = true
      record     = "instance:node_network_receive_bytes_excluding_lo:rate5m"
      expression = "sum without (device) (  rate(node_network_receive_bytes_total{job=\"node\", device!=\"lo\"}[5m]))"
    },
    {
      enabled    = true
      record     = "instance:node_network_transmit_bytes_excluding_lo:rate5m"
      expression = "sum without (device) (  rate(node_network_transmit_bytes_total{job=\"node\", device!=\"lo\"}[5m]))"
    },
    {
      enabled    = true
      record     = "instance:node_network_receive_drop_excluding_lo:rate5m"
      expression = "sum without (device) (  rate(node_network_receive_drop_total{job=\"node\", device!=\"lo\"}[5m]))"
    },
    {
      enabled    = true
      record     = "instance:node_network_transmit_drop_excluding_lo:rate5m"
      expression = "sum without (device) (  rate(node_network_transmit_drop_total{job=\"node\", device!=\"lo\"}[5m]))"
    }
  ]
}
  
module "node_recording_rules" {
  source   = "us.spacelift.io/claconnect/monitor_alert_prometheus_rule_group/azurerm"
  version  = "1.1.0"
  for_each = var.aks_clusters
  
  # Required inputs
  name                = "NodeRecordingRules-${each.key}"
  location            = var.location
  resource_group_name = module.resource_group.name
  description         = "Node recording rules for ${each.key}"
  scopes              = [module.monitor_workspace.id, module.aks_cluster[each.key].id]
  
  # Optional inputs
  cluster_name       = each.key
  interval           = var.prometheus_rule_group_interval
  rule_group_enabled = var.prometheus_rule_group_enabled
  rules              = local.node_recording_rules
  tags               = var.tags
}

Step 9: Update Configuration File

Add the required values to your .auto.tfvars file:

monitor_workspace_name = "sbx-eus2-prom-ops-amw-001"  # Customize per environment
  
# Optional: Override defaults if needed
prometheus_rule_group_enabled  = true
prometheus_rule_group_interval = "PT1M"
grafana_uami_principal_id      = "a2f0c2f6-68c8-44ae-9a12-b6a5990f94cd" ## Or leave the default in the variable.

Step 10: Apply the Configuration

Submit a PR and merge the code changes into the main branch to trigger a Spacelift run.

After successful deployment, note the Azure Monitor Workspace ID from the output. You will need this for Stage 2.

Stage 2: Configure Grafana Data Source Tracking

⚠️ IMPORTANT: The Azure Monitor Workspace ID from Stage 1 must be merged into the DevOps shared services configuration to enable data source tracking in the Grafana workspace.

This step ensures that Grafana can discover and connect to your newly created Prometheus data source by integrating the Azure Monitor Workspace into the Grafana instance configuration.

Step 1: Locate the Azure Monitor Workspace ID

From Stage 1, retrieve the Azure Monitor Workspace resource ID. It will be in this format:

/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Monitor/accounts/{workspace-name}

Example:

/subscriptions/a3fde7ee-8b51-4aec-8f30-3751a39b3c14/resourceGroups/sbx-eus2-operations-shared-services-rg-001/providers/Microsoft.Monitor/accounts/sbx-eus2-prom-ops-amw-001

You can retrieve this from:

• Terraform output: module.monitor_workspace.id

• Azure Portal: Navigate to the AMW → Properties → Resource ID

• Azure CLI: az monitor account show --name <workspace-name> --resource-group <rg-name> --query id

Step 2: Create a Pull Request in az-devops-lz-tf

Navigate to the DevOps Landing Zone repository (az-devops-lz-tf) and locate the shared services configuration.

File to Edit: shared_services/config/prd.eastus2.auto.tfvars

Add your Azure Monitor Workspace ID to the prometheus_azure_monitor_workspaces list:

prometheus_azure_monitor_workspaces = [
  # Existing workspaces
  "/subscriptions/a3fde7ee-8b51-4aec-8f30-3751a39b3c14/resourceGroups/sbx-eus2-operations-shared-services-rg-001/providers/Microsoft.Monitor/accounts/sbx-eus2-prom-ops-amw-001",
  
  # Add your new workspace here
  "/subscriptions/{your-subscription-id}/resourceGroups/{your-rg-name}/providers/Microsoft.Monitor/accounts/{your-workspace-name}"
]

Important Notes:

• This is a list, so multiple Azure Monitor Workspaces can be tracked

• Each workspace added here will automatically create a Grafana data source

• The data source will be named: Managed_Prometheus_{workspace-name}

• Ensure proper comma separation between list items

Step 3: Understand the Integration Flow

The Grafana module in shared_services/main.tf automatically integrates Azure Monitor Workspaces:

module "dashboard_grafana" {
  source  = "us.spacelift.io/claconnect/dashboard_grafana/azurerm"
  version = "~> 1.0"
  
  # ... other configuration
  
  # This parameter links AMW IDs to Grafana
  azure_monitor_workspace_integrations = var.prometheus_azure_monitor_workspaces
}

When Terraform applies this configuration:

1. Grafana creates a managed data source for each AMW ID

2. The Grafana UAMI (configured in Stage 1) authenticates to the AMW

3. The data source becomes queryable in Grafana as Managed_Prometheus_{workspace-name}

Step 4: Verify Variable Definition

Ensure the variable exists in shared_services/variables.tf:

variable "prometheus_azure_monitor_workspaces" {
  type        = list(string)
  description = "The resource IDs of the connected Azure Monitor Workspaces."
  default     = []
}

This variable should already exist in the DevOps shared services configuration. If not, add it.

Step 5: Submit the Pull Request

Create a PR with the following details:

PR Title: Add Azure Monitor Workspace for {cluster-name} Prometheus Integration

PR Description:

## Summary
  
Adds Azure Monitor Workspace integration for {cluster-name} to enable Prometheus metrics in Grafana.
  
## Changes
  
- Added Azure Monitor Workspace ID to `prometheus_azure_monitor_workspaces` list in `prd.eastus2.auto.tfvars`
  
## Azure Monitor Workspace Details
  
- **Name**: {workspace-name}
- **Subscription**: {subscription-name}
- **Resource Group**: {resource-group-name}
- **Region**: {region}
  
## Related Work
  
- Stage 1 completed in {application-lz-repo} PR #{pr-number}
- DCR, DCE, and DCRA configured for AKS cluster
- Grafana UAMI has Monitoring Reader role on AMW
  
## Testing
  
- [ ] Terraform plan shows Grafana module update
- [ ] No other resources affected
- [ ] After apply, verify data source appears in Grafana

Step 6: Apply and Verify

After the PR is merged and Terraform is applied:

Check Grafana Data Sources:

1. Navigate to Grafana → Configuration → Data Sources

2. Verify new data source: Managed_Prometheus_{workspace-name}

3. Click on the data source and select “Save & Test”

4. Should show: “Successfully queried the Azure Monitor Workspace”

Test Prometheus Queries:

1. Go to Explore in Grafana

2. Select your new data source

3. Run a test query: up

4. Should return metrics from your AKS cluster

Verify Metrics Flow:

1. Go to Dashboards → Azure Managed Prometheus

2. Select a Dashboard to confirm metrics visibility

Step 7: Configure Team Permissions (If Applicable)

If using the Grafana team management from earlier in this conversation, update the teams configuration in az-devops-lz-tf/grafana/config/prd.eastus2.auto.tfvars:

teams = {
  "{cluster-team-name}" = {
    monitor_workspace_name = "{your-workspace-name}"
    external_groups        = ["{azure-ad-group-id}"]
    permissions = [
      {
        permission = "Admin"
      },
      {
        built_in_role = "Admin"
        permission = "Admin"
      }
    ]
  }
}

This ensures the appropriate teams have access to query the Prometheus data source in Grafana.

Stage 3: Configure Team Permissions in Grafana

After the Azure Monitor Workspace has been integrated into Grafana (Stage 2), you need to configure team permissions to control access to the Prometheus data source.

Step 1: Identify Required Information

Before creating the PR, gather the following information:

• Monitor Workspace Name: From Stage 1 (e.g., sbx-eus2-prom-ops-amw-001)

• Entra ID Group ID: The Object ID of the Azure AD/Entra group that should have access

  - Navigate to Azure Portal → Entra ID → Groups

  - Find your group → Overview → Object ID

  - Example: b76d9990-2716-4ead-8cef-abb3cff4e60d

• Team Name: A descriptive name for the Grafana team (typically matches cluster or application name)

Step 2: Create Pull Request in az-devops-lz-tf

Navigate to the DevOps Landing Zone repository (az-devops-lz-tf/grafana) and locate the Grafana configuration.

File to Edit: grafana/config/prd.eastus2.auto.tfvars

Add a new team configuration to the teams map:

teams = {
  # Existing teams
  "az-kt-sandbox-owners" = {
    monitor_workspace_name = "sbx-eus2-prom-ops-amw-001"
    external_groups        = ["b76d9990-2716-4ead-8cef-abb3cff4e60d"]
    permissions = [
      {
        permission = "Query"
      },
      {
        permission    = "Admin"
        built_in_role = "Admin"
      }
    ]
  },
  
  # Add your new team here
  "{your-team-name}" = {
    monitor_workspace_name = "{your-workspace-name}"
    external_groups        = ["{your-entra-group-id}"]
    permissions = [
      {
        permission = "Query"
      }
    ]
  }
}

Step 3: Configure Permission Levels

Choose the appropriate permission level for your team:

Query Permission (Read-Only)

permissions = [
  {
    permission = "Query"
  }
]

• Use case: Teams that only need to view and query metrics

• Access: Can run queries, view dashboards, but cannot modify the data source

Query + Built-in Role

permissions = [
  {
    permission    = "Query"
    built_in_role = "Viewer"  # Options: Viewer, Editor, Admin
  }
]

• Use case: Teams needing broader Grafana access beyond just the data source

• Options:

  - Viewer: Can view dashboards and data sources

  - Editor: Can create and edit dashboards

  - Admin: Full Grafana administration rights

Admin Permission

permissions = [
  {
    permission = "Query"
  },
  {
    permission    = "Admin"
    built_in_role = "Admin"
  }
]

• Use case: Platform/DevOps teams managing the data source

• Access: Full control over the data source configuration

Step 4: Understand the Configuration Structure

The Grafana configuration creates the following resources:

# Creates a Grafana team
module "grafana_team" {
  source  = "us.spacelift.io/claconnect/grafana_team/grafana"
  version = "~> 1.0"
  
  for_each = var.teams
  name     = each.key  # Team name from the map key
}
  
# Links Entra ID group to Grafana team
module "grafana_team_external_group" {
  source  = "us.spacelift.io/claconnect/grafana_team_external_group/grafana"
  version = "~> 1.0"
  
  for_each = var.teams
  team_id   = module.grafana_team[each.key].id
  group_ids = each.value.external_groups  # Your Entra group ID
}
  
# Grants team permissions to the data source
module "grafana_data_source_permissions" {
  source  = "us.spacelift.io/claconnect/grafana_data_source_permissions/grafana"
  version = "1.0.2"
  
  for_each = var.teams
  datasource_uid = data.grafana_data_source.data_source[each.key].uid
  permissions    = each.value.permissions
  team_id        = module.grafana_team[each.key].id
}
  
# Automatically looks up data source by workspace name
data "grafana_data_source" "data_source" {
  for_each = var.teams
  name     = "Managed_Prometheus_${each.value.monitor_workspace_name}"
}

Key Points:

• The team name (map key) becomes the Grafana team name

• The monitor_workspace_name is used to automatically find the correct data source

• Data source name format: Managed_Prometheus_{workspace-name}

• Multiple Entra ID groups can be linked to a single team

Step 5: Verify Configuration Prerequisites

Before submitting the PR, ensure:

• ✅ Stage 2 Complete: Azure Monitor Workspace is added to shared_services/config/prd.eastus2.auto.tfvars

• ✅ Terraform Applied: The Grafana data source exists in Grafana

• ✅ Entra Group Exists: The specified group ID is valid and accessible

• ✅ Group Members: Users are already members of the Entra ID group

Step 6: Submit the Pull Request

Create a PR with the following details:

PR Title: Add Grafana Team and Permissions for {team-name}

PR Description:

## Summary
  
Configures Grafana team and data source permissions for {team-name} to access Prometheus metrics from {cluster-name}.
  
## Changes
  
- Added team `{team-name}` to `grafana/config/prd.eastus2.auto.tfvars`
- Linked Entra ID group `{group-name}` ({group-id})
- Granted `Query` permission to data source `Managed_Prometheus_{workspace-name}`
  
## Team Configuration Details
  
- **Team Name**: {team-name}
- **Entra ID Group**: {group-name} ({group-id})
- **Workspace**: {workspace-name}
- **Permissions**: Query (read-only access)
  
## Prerequisites Completed
  
- [x] Azure Monitor Workspace integrated in Stage 2
- [x] Data source `Managed_Prometheus_{workspace-name}` exists in Grafana
- [x] Entra ID group created with appropriate members
- [x] Verified workspace name matches AMW resource
  
## Testing Plan
  
After apply:
  
- [ ] Verify team appears in Grafana Teams
- [ ] Confirm Entra group members can log into Grafana
- [ ] Test data source access with team member account
- [ ] Validate Query permissions work as expected

Step 7: Apply and Verify

After the PR is merged and Terraform is applied:

Verify Team Creation:

1. Log into Grafana as an admin

2. Navigate to Configuration → Teams

3. Confirm your new team exists

Check Group Mapping:

1. Click on the team name

2. Go to “External group sync”

3. Verify the Entra ID group is linked

Verify Data Source Permissions:

1. Navigate to Configuration → Data Sources

2. Select Managed_Prometheus_{workspace-name}

3. Click on “Permissions” tab

4. Confirm your team has the configured permissions

Test User Access:

1. Have a member of the Entra ID group log into Grafana

2. Navigate to Explore

3. Select the data source from the dropdown

4. Run a test query: up{cluster="{cluster-name}"}

5. Verify metrics are returned

Validate Permission Restrictions:

1. Confirm users can query but not modify the data source (if Query-only)

2. Test that users cannot access other data sources (if not granted)

Step 8: Troubleshooting Common Issues

Data Source Not Found

• Error: [GET /datasources/name/{name}] getDataSourceByName (status 404): {}

• Resolution:

  - Ensure Stage 2 is complete and Terraform has been applied

  - Verify the workspace name matches exactly (case-sensitive)

  - Check data source exists: Grafana → Configuration → Data Sources

Entra Group Not Syncing

• Resolution:

  - Verify the group Object ID is correct

  - Check Grafana has proper Entra ID integration configured

  - Wait 5-10 minutes for sync to occur

  - Users may need to log out and back in

Permission Validation Error

• Error: Invalid value for "value" parameter: argument must not be null.

• Resolution:

  - Ensure built_in_role is either omitted or set to a valid value

  - Valid values: Viewer, Editor, Admin

  - If using Query-only permission, omit built_in_role entirely

Step 9: Multiple Teams (Optional)

You can configure multiple teams for different access patterns:

teams = {
  # Read-only access for developers
  "{cluster-name}-developers" = {
    monitor_workspace_name = "{workspace-name}"
    external_groups        = ["{developer-group-id}"]
    permissions = [
      {
        permission = "Query"
      }
    ]
  }
  
  # Admin access for platform team
  "{cluster-name}-platform" = {
    monitor_workspace_name = "{workspace-name}"
    external_groups        = ["{platform-team-group-id}"]
    permissions = [
      {
        permission = "Query"
      },
      {
        permission    = "Admin"
        built_in_role = "Admin"
      }
    ]
  }
  
  # Editor access for SRE team
  "{cluster-name}-sre" = {
    monitor_workspace_name = "{workspace-name}"
    external_groups        = ["{sre-group-id}"]
    permissions = [
      {
        permission    = "Query"
        built_in_role = "Editor"
      }
    ]
  }
}

Each team will:

• Have access to the same data source

• Have different permission levels

• Be linked to different Entra ID groups

• Appear as separate teams in Grafana

Complete Workflow Summary

Stage 1: Application/Operations Landing Zone

Deploy monitoring infrastructure:

• Azure Monitor Workspace (AMW)

• Data Collection Endpoint (DCE)

• Data Collection Rule (DCR)

• Data Collection Rule Association (DCRA)

• Prometheus Rule Groups

• Grafana UAMI role assignment

Stage 2: DevOps Landing Zone - Shared Services

Integrate AMW with Grafana:

• Add AMW resource ID to prometheus_azure_monitor_workspaces list

• Grafana automatically creates managed data source

• Data source named: Managed_Prometheus_{workspace-name}

Stage 3: DevOps Landing Zone - Grafana

Configure team access:

• Create Grafana team

• Link Entra ID group to team

• Grant data source permissions to team

• Verify user access and permissions

Result: Teams can now access Prometheus metrics from AKS clusters through Grafana with appropriate permission levels based on their role.

Validation

After completing all stages, verify the setup:

Check Data Collection Rule Association:

1. Navigate to your AKS cluster in Azure Portal

2. Go to Settings → Insights → Data Collection Settings

3. Verify the DCR association is present

Verify Prometheus Metrics:

1. Navigate to the Azure Monitor Workspace in Azure Portal

2. Query some basic metrics (e.g., up, kube_node_info)

Check Grafana Data Source:

1. Log into Grafana

2. Navigate to Configuration → Data Sources

3. Verify the Managed_Prometheus_<workspace-name> data source exists

4. Test the connection and run a simple query

Verify Permissions:

1. Ensure the Grafana UAMI has “Monitoring Reader” role on the AMW

2. Check role assignments in the AMW’s Access Control (IAM) blade

Troubleshooting

No data appearing in Grafana

• Verify the DCR association is created for the AKS cluster

• Check that the query endpoint URL in Grafana matches the AMW’s query endpoint

• Ensure the Grafana UAMI has proper permissions

Data Collection Rule issues

• Verify the DCE is in the same region as the AKS cluster

• Check that the data flow, data sources, and destinations are correctly configured

• Review activity logs for any deployment errors

Role assignment errors

• Confirm the Grafana UAMI principal ID is correct

• Verify the UAMI has been properly created in the DevOps subscription

• Check for any RBAC propagation delays (can take 5-10 minutes)

Additional Resources

• Azure Monitor Workspace Documentation

• Azure Managed Prometheus

• Data Collection Rules

Part 2: Enabling Application Telemetry

This section covers how to configure Application Insights logging in your Node.js applications using the framework-nodejs-logging package version 2.1.3.

Application Insights Configuration

Following the new pattern below, update the logging files in /internal to follow this pattern, using version framework-nodejs-logging@2.1.3 with the Direct Construction method.

Test your settings by clearing npm cache (if needed): npm cache clean --force and running npm install from this git repository.

Authentication Modes

The AppInsightsLogConfigProfile supports two authentication modes:

1. Traditional Authentication (API Key)

Use this mode when you have an instrumentation key embedded in the connection string:

import { Logger, ConfigProfilesEnum, resolveConfigProfileByName } from 'framework-nodejs-logging';
  
const profile = resolveConfigProfileByName(
  ConfigProfilesEnum.AppInsightsLogConfigProfile,
  'InstrumentationKey=xxx;IngestionEndpoint=https://xxx.applicationinsights.azure.com/'
);
  
const logger = Logger.create('info', profile);
logger.info('Hello from traditional auth');

2. Managed Identity Authentication (Recommended for Azure)

Use this mode when deploying to Azure services with managed identity enabled. This is more secure as it eliminates the need to store API keys.

System-Assigned Managed Identity:

import { Logger, ConfigProfilesEnum, resolveConfigProfileByName } from 'framework-nodejs-logging';
import { ManagedIdentityCredential } from '@azure/identity';
  
// Create credential for system-assigned managed identity
const credential = new ManagedIdentityCredential();
  
const profile = resolveConfigProfileByName(
  ConfigProfilesEnum.AppInsightsLogConfigProfile,
  'InstrumentationKey=xxx;IngestionEndpoint=https://xxx.applicationinsights.azure.com/',
  credential
);
  
const logger = Logger.create('info', profile);
logger.info('Hello from managed identity');

User-Assigned Managed Identity:

import { ManagedIdentityCredential } from '@azure/identity';
  
// Provide the client ID for user-assigned managed identity
const credential = new ManagedIdentityCredential({
  clientId: 'your-user-assigned-identity-client-id'
});
  
const profile = resolveConfigProfileByName(
  ConfigProfilesEnum.AppInsightsLogConfigProfile,
  'InstrumentationKey=xxx;IngestionEndpoint=https://xxx.applicationinsights.azure.com/',
  credential
);

Local Development:

For local development, use DefaultAzureCredential which tries multiple authentication methods:

import { DefaultAzureCredential } from '@azure/identity';
  
const credential = new DefaultAzureCredential();
  
const profile = resolveConfigProfileByName(
  ConfigProfilesEnum.AppInsightsLogConfigProfile,
  'InstrumentationKey=xxx;IngestionEndpoint=https://xxx.applicationinsights.azure.com/',
  credential
);

Direct Construction:

You can also create the profile directly:

import { AppInsightsLogConfigProfile, Logger } from 'framework-nodejs-logging';
import { ManagedIdentityCredential } from '@azure/identity';
  
const connectionString = 'InstrumentationKey=xxx;IngestionEndpoint=https://xxx.applicationinsights.azure.com/';
const credential = new ManagedIdentityCredential();
const profile = new AppInsightsLogConfigProfile(connectionString, credential);
const logger = Logger.create('info', profile);

Important Notes

1. Connection String Format: When using managed identity, your connection string must include the IngestionEndpoint. The library will validate this and throw an error if it’s missing.

2. Azure RBAC Permissions: Ensure your managed identity has the Monitoring Metrics Publisher role assigned to the Application Insights resource.

3. Environment-Specific Configuration: It’s recommended to use environment variables to configure authentication dynamically:

import { ManagedIdentityCredential, DefaultAzureCredential } from '@azure/identity';
  
const connectionString = process.env.DOCAI_APPLICATIONINSIGHTS_CONNECTION_STRING!;
const useManagedIdentity = process.env.NODE_ENV === 'production';
  
// Use managed identity in production, local credentials in development
const credential = useManagedIdentity
  ? new ManagedIdentityCredential()
  : new DefaultAzureCredential();
  
const profile = resolveConfigProfileByName(
  ConfigProfilesEnum.AppInsightsLogConfigProfile,
  connectionString,
  credential
);

Environment-Specific Configuration

Recommended Pattern for Production Applications:

// src/internal/frameworkAdapter/registerLogger.ts
import { Logger, AppInsightsLogConfigProfile } from 'framework-nodejs-logging';
import { ManagedIdentityCredential, DefaultAzureCredential } from '@azure/identity';
  
function createLogger() {
  const connectionString = process.env.DOCAI_APPLICATIONINSIGHTS_CONNECTION_STRING;
  const environment = process.env.NODE_ENV || 'development';
  const logLevel = process.env.LOG_LEVEL || 'info';
  
  if (!connectionString) {
    throw new Error('DOCAI_APPLICATIONINSIGHTS_CONNECTION_STRING environment variable is required');
  }
  
  // Use managed identity in production, DefaultAzureCredential for development
  const credential = environment === 'production'
    ? new ManagedIdentityCredential()
    : new DefaultAzureCredential();
  
  const profile = new AppInsightsLogConfigProfile(connectionString, credential);
  return Logger.create(logLevel, profile);
}
  
export const logger = createLogger();

Integration with Fastify:

// src/app.ts
import Fastify from 'fastify';
import { logger } from './internal/frameworkAdapter/registerLogger';
  
const app = Fastify({
  logger: logger,
  disableRequestLogging: false,
});
  
// Log incoming requests
app.addHook('onRequest', async (request, reply) => {
  logger.info({
    msg: 'Incoming request',
    method: request.method,
    url: request.url,
    headers: request.headers,
  });
});
  
// Log responses
app.addHook('onResponse', async (request, reply) => {
  logger.info({
    msg: 'Request completed',
    method: request.method,
    url: request.url,
    statusCode: reply.statusCode,
    responseTime: reply.getResponseTime(),
  });
});
  
export default app;

Best Practices

1. Never log sensitive data or PII (Personally Identifiable Information)

2. Use structured logging with consistent field names

3. Configure appropriate log levels for different environments

4. Monitor Application Insights ingestion to control costs

5. Use correlation IDs to trace requests across services

6. Set up alerts for critical errors and performance thresholds

Part 3: Integrating Dynatrace with Prometheus

Dynatrace Overview

Dynatrace is an enterprise-grade observability platform that provides advanced monitoring, AI-powered analytics, and automated root cause analysis. By integrating Dynatrace with Azure Managed Prometheus, you can:

• Centralize observability: Consolidate Prometheus metrics with Dynatrace’s full-stack monitoring

• AI-powered insights: Leverage Davis AI for automatic problem detection and root cause analysis

• Advanced analytics: Use Dynatrace’s query language (DQL) for complex metric analysis

• Unified dashboards: Combine infrastructure, application, and business metrics in one platform

• Enterprise features: Access SLO tracking, service-level monitoring, and compliance reporting

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                      Azure AKS Cluster                          │
│                                                                 │
│  ┌──────────────┐         ┌────────────────┐                  │
│  │  Application │────────►│   Prometheus   │                  │
│  │    Pods      │ metrics │   (Managed)    │                  │
│  └──────────────┘         └────────┬───────┘                  │
│                                     │                           │
│                           ┌─────────▼───────────┐              │
│                           │  Dynatrace OneAgent │              │
│                           │     (DaemonSet)     │              │
│                           └─────────┬───────────┘              │
└─────────────────────────────────────┼──────────────────────────┘
                                      │
                          ┌───────────▼───────────┐
                          │ Azure Monitor         │
                          │ Workspace (AMW)       │
                          └───────────┬───────────┘
                                      │
                          ┌───────────▼───────────┐
                          │  Dynatrace ActiveGate │
                          │  (Prometheus Scraper) │
                          └───────────┬───────────┘
                                      │
                          ┌───────────▼───────────┐
                          │   Dynatrace SaaS      │
                          │   (Cluster/Tenant)    │
                          └───────────────────────┘

Deployment Options

There are two primary approaches to integrate Dynatrace with Prometheus on AKS:

Option 1: OneAgent + ActiveGate (Recommended)

Pros:

• Full-stack observability (infrastructure + application + Prometheus metrics)

• Automatic instrumentation of applications

• AI-powered problem detection

• Simplified deployment and management

Cons:

• Requires Dynatrace OneAgent license

• More resource consumption on cluster nodes

Option 2: ActiveGate Only (Metrics-focused)

Pros:

• Lower resource consumption

• Focused on metrics collection only

• Lower cost (doesn’t require full OneAgent license)

Cons:

• No automatic application instrumentation

• Limited to Prometheus metrics only

• No full-stack visibility

Terraform Configuration

Prerequisites

Before implementing Dynatrace integration, ensure you have:

1. Dynatrace Tenant/Cluster: Active Dynatrace SaaS or Managed environment

2. Dynatrace API Token: With the following scopes:

   - activeGateTokenManagement.create

   - entities.read

   - metrics.ingest

   - InstallerDownload

3. Dynatrace PaaS Token: For OneAgent deployment

4. Azure Monitor Workspace: Deployed from Part 1 of this guide

5. Terraform Access: Permissions to deploy resources in AKS cluster

Step 1: Add Required Variables

Add Dynatrace-specific variables to your variables.tf:

variable "dynatrace_enabled" {
  description = "Enable Dynatrace integration for Prometheus metrics"
  type        = bool
  default     = false
}
  
variable "dynatrace_environment_id" {
  description = "Dynatrace environment ID (e.g., abc12345)"
  type        = string
  default     = ""
}
  
variable "dynatrace_api_url" {
  description = "Dynatrace API URL (e.g., https://abc12345.live.dynatrace.com/api)"
  type        = string
  default     = ""
}
  
variable "dynatrace_api_token" {
  description = "Dynatrace API token for authentication"
  type        = string
  sensitive   = true
  default     = ""
}
  
variable "dynatrace_paas_token" {
  description = "Dynatrace PaaS token for OneAgent deployment"
  type        = string
  sensitive   = true
  default     = ""
}
  
variable "dynatrace_oneagent_enabled" {
  description = "Deploy Dynatrace OneAgent for full-stack monitoring"
  type        = bool
  default     = true
}
  
variable "dynatrace_activegate_enabled" {
  description = "Deploy Dynatrace ActiveGate for Prometheus scraping"
  type        = bool
  default     = true
}
  
variable "prometheus_remote_write_enabled" {
  description = "Enable Prometheus remote write to Dynatrace"
  type        = bool
  default     = true
}

Step 2: Create Dynatrace Namespace

Create a dedicated namespace for Dynatrace components:

resource "kubernetes_namespace" "dynatrace" {
  count = var.dynatrace_enabled ? 1 : 0
  
  metadata {
    name = "dynatrace"
    labels = {
      "managed-by" = "terraform"
      "purpose"    = "monitoring"
    }
  }
  
  depends_on = [module.aks_cluster]
}

Step 3: Create Kubernetes Secrets for Dynatrace

Store Dynatrace tokens securely in Kubernetes secrets:

resource "kubernetes_secret" "dynatrace_tokens" {
  count = var.dynatrace_enabled ? 1 : 0
  
  metadata {
    name      = "dynatrace-tokens"
    namespace = kubernetes_namespace.dynatrace[0].metadata[0].name
  }
  
  data = {
    api-token  = var.dynatrace_api_token
    paas-token = var.dynatrace_paas_token
  }
  
  type = "Opaque"
}

Step 4: Deploy Dynatrace OneAgent (Optional)

Deploy OneAgent as a DaemonSet for full-stack monitoring:

resource "helm_release" "dynatrace_oneagent" {
  count = var.dynatrace_enabled && var.dynatrace_oneagent_enabled ? 1 : 0
  
  name       = "dynatrace-oneagent"
  repository = "https://raw.githubusercontent.com/Dynatrace/dynatrace-operator/main/config/helm/repos/stable"
  chart      = "dynatrace-operator"
  version    = "0.14.2"  # Use latest stable version
  namespace  = kubernetes_namespace.dynatrace[0].metadata[0].name
  
  values = [
    yamlencode({
      platform = "kubernetes"
      operator = {
        image = "dynatrace/dynatrace-operator"
      }
  
      oneAgent = {
        enabled = true
        name    = "oneagent"
        apiUrl = var.dynatrace_api_url
        # Reference the secret created above
        tokens = kubernetes_secret.dynatrace_tokens[0].metadata[0].name
  
        # Resource limits
        resources = {
          requests = {
            cpu    = "100m"
            memory = "512Mi"
          }
          limits = {
            cpu    = "500m"
            memory = "1Gi"
          }
        }
  
        # Environment-specific settings
        env = [
          {
            name  = "ONEAGENT_ENABLE_VOLUME_STORAGE"
            value = "true"
          }
        ]
  
        # Node selector for specific node pools (optional)
        nodeSelector = {
          "kubernetes.io/os" = "linux"
        }
  
        # Tolerations for node taints (optional)
        tolerations = []
      }
  
      # Enable Kubernetes monitoring
      kubernetesMonitoring = {
        enabled = true
      }
    })
  ]
  
  depends_on = [
    kubernetes_namespace.dynatrace,
    kubernetes_secret.dynatrace_tokens
  ]
}