Microservice Guide

Overview

This document serves as the authoritative guide for developing microservices within this project. It outlines best practices, architecture decisions, and coding standards to ensure maintainability, scalability, and efficiency. The microservices are built using Node.js, TypeScript, and RESTful APIs, with Pino for logging. They also integrate with Azure (database TBD).

1. Microservice Architecture

1.1 Principles

Single Responsibility: Each service should handle one specific function.
- A user service that handles user registration, authentication, and profile management.
Statelessness: Avoid storing session data in microservices.
- Use JWT tokens for authentication instead of server-side sessions.
Resilience: Implement retry logic, circuit breakers, and timeouts.
- Use libraries like axios-retry for retry logic and opossum for circuit breakers.
  - Circuit Breakers: A circuit breaker is a design pattern used to detect failures and encapsulate the logic of preventing a failure from constantly recurring during maintenance, temporary external system failure, or unexpected system difficulties. When a service fails repeatedly, the circuit breaker trips and further calls to the service are blocked for a certain period.
    - It wouldnt need to be called on every function but those that have a higher chance of failure
Scalability: Services should be horizontally scalable.
- Deploy services in a Kubernetes cluster to easily scale out.
Observability: Use logs, metrics, and distributed tracing.
- Integrate with tools like Prometheus for metrics and Jaeger for tracing.
  - Prometheus: An open-source systems monitoring and alerting toolkit. It collects and stores metrics as time series data, i.e., metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.
    - Use Prometheus to monitor CPU usage, memory usage, request rates, error rates, etc.
  - Jaeger: An open-source, end-to-end distributed tracing tool. It is used for monitoring and troubleshooting microservices-based distributed systems.
    - Use Jaeger to trace requests as they flow through various microservices, helping to identify bottlenecks and performance issues.

Why Metrics and Tracing are Important

Metrics: Metrics provide quantitative data about the performance and health of your services. They help in identifying trends, detecting anomalies, and triggering alerts when something goes wrong. Metrics are crucial for capacity planning, performance tuning, and ensuring the reliability of your services.
- Monitoring request rates and error rates can help identify issues before they impact users.
Tracing: Distributed tracing helps in understanding the flow of requests through various microservices. It provides insights into the latency and performance of each service, helping to identify bottlenecks and optimize the overall system.
- Tracing a request from the frontend to the backend can help identify which service is causing delays.

Azure Monitoring and Logging

Azure provides built-in monitoring and logging tools such as Azure Monitor, Application Insights, and Log Analytics. These tools can be used to collect, analyze, and act on telemetry data from your applications and infrastructure.

Azure Monitor: Provides full-stack monitoring for applications and infrastructure.
Application Insights: An application performance management (APM) service for web developers to monitor live applications, detect performance anomalies, and diagnose issues.
Log Analytics: Collects and analyzes log data from various sources.

While Azure provides robust monitoring and logging capabilities, integrating tools like Prometheus and Jaeger can provide additional flexibility and insights, especially in multi-cloud or hybrid environments.

1.2 Service Structure

/microservice-name
│── src
│   ├── controllers
│   ├── services
│   ├── models
│   ├── routes
│   ├── middlewares
│   ├── utils
│── tests
│── config
│── .env
│── index.ts
│── Dockerfile
│── README.md

1.3 Directory Structure Explanation

src: The main source code directory for the microservice.
- controllers: Contains the logic for handling incoming HTTP requests and sending responses. Controllers act as intermediaries between the client and the service layer (e.g., userController.ts to handle user-related endpoints).
- services: Contains the business logic of the application. Services perform operations and interact with models to fulfill the requests (e.g., userService.ts to handle user-related business logic).
- models: Defines the data structures and schemas used in the application. Models represent the shape of the data.
- routes: Defines the API endpoints and maps them to the corresponding controller methods (e.g., userRoutes.ts to define routes like GET /users, POST /users).
- middlewares: Contains middleware functions that process requests before they reach the controllers. Middlewares can be used for authentication, logging, error handling, etc. (e.g., authMiddleware.ts to handle JWT authentication).
- utils: Contains utility functions and helpers that can be used throughout the application (e.g., logger.ts to configure and use the Pino logger).
tests: Contains unit and integration tests for the application. Tests ensure that the code behaves as expected and helps prevent regressions (e.g., userController.test.ts to test the user controller).
config: Contains configuration files and settings for the application. This can include database configurations, environment-specific settings, etc. (e.g., databaseConfig.ts to configure database connections).
.env: Environment variables file to store sensitive configurations and settings. This file should not be committed to version control (e.g., .env to store database connection strings, API keys, etc.).
index.ts: The entry point of the application. This file initializes the server, sets up middleware, and starts listening for incoming requests (e.g., index.ts to set up Express server and routes).
Dockerfile: Defines the Docker image for the microservice. This file contains instructions for building the Docker image, including installing dependencies and running the application (e.g., Dockerfile to build and run the microservice in a container).

2. Technology Stack

Runtime: Node.js
Language: TypeScript
API Standard: REST
Logging: Pino
Cloud Provider: Azure
Testing: Vitest
Containerization: Docker/Kubernetes
Package Manager: npm

3. Best Practices

3.1 TypeScript Guidelines

Always define explicit types and avoid any.
- const user: User = { id: 1, name: 'John Doe' };
Use interfaces and types for API payloads.

interface User {
  id: number;
  name: string;
}

Enable strict mode in tsconfig.json.

{
  "compilerOptions": {
    "strict": true
  }
}

3.2 REST API Design

Use RESTful resource naming conventions (e.g., GET /users/{id}).
Use proper HTTP status codes (200 for success, 400 for bad requests, 500 for internal errors).
Use OpenAPI to define endpoint URLs and methods.
Use JSON Schema to define data types and structures for HTTP message bodies.
- This ensures that the same validation logic is applied across different layers (e.g., request validation, message bus messages, and document database validation).
Example of using OpenAPI and JSON Schema for validation:

openapi: 3.0.0
info:
  title: Example API
  version: 1.0.0
paths:
  /users:
    post:
      summary: Create a user
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/User'
      responses:
        '201':
          description: User created
components:
  schemas:
    User:
      type: object
      required:
        - username
        - password
      properties:
        username:
          type: string
          minLength: 3
          maxLength: 30
        password:
          type: string
          pattern: '^[a-zA-Z0-9]{3,30}$'

Example of using JSON Schema for message bus messages:

{
  "$schema": "http://....",
  "type": "object",
  "required": ["username", "password"],
  "properties": {
    "username": {
      "type": "string",
      "minLength": 3,
      "maxLength": 30
    },
    "password": {
      "type": "string",
      "pattern": "^[a-zA-Z0-9]{3,30}$"
    }
  }
}

Follow OpenAPI/Swagger documentation standards.
- Use swagger-jsdoc to generate Swagger documentation from JSDoc comments.

3.3 Logging with Pino

Log at appropriate levels: info, warn, error, debug.
Structure logs in JSON for better observability.
Mask sensitive data before logging.

const pino = require('pino');
const logger = pino({
  redact: ['req.headers.authorization']
});

3.4 Security

Use environment variables for sensitive configurations.
- Store secrets in .env file and use dotenv to load them.
Implement JWT-based authentication if required.
- Use jsonwebtoken to sign and verify tokens.
Ensure proper CORS handling.
- Use cors middleware to configure CORS.
Validate all inputs to prevent injections.
- Use libraries like express-validator for input validation.
Encrypt sensitive data: Use encryption for sensitive data both at rest and in transit.
- Use libraries like crypto for encrypting data before storing it in the database.
- Use HTTPS to encrypt data in transit.
Use security headers: Implement security headers to protect against common vulnerabilities.
- Use helmet middleware to set various HTTP headers.
Rate limiting: Implement rate limiting to prevent abuse and attacks. (TBD)
- Use express-rate-limit to limit repeated requests to public APIs.
Audit logging: Maintain logs of security-related events for auditing purposes.
- Log authentication attempts, access to sensitive data, and configuration changes.
Regular security assessments: Conduct regular security assessments and vulnerability scans.
- Use tools like OWASP ZAP or Nessus for vulnerability scanning.
Secure dependencies: Regularly update dependencies and use tools to check for known vulnerabilities.
- Use npm audit to identify and fix vulnerabilities in dependencies.
Role-based access control (RBAC): Implement RBAC to restrict access based on user roles.
- Use libraries like casbin to manage roles and permissions.
Data masking: Mask sensitive data in logs and responses.
- Ensure that sensitive information like passwords are not logged or exposed in error messages.

3.5 Error Handling

Use a centralized error handling middleware to manage errors consistently across the application.

// Configuration object to determine error handling behavior
const config = {
  showErrorDetails: process.env.SHOW_ERROR_DETAILS === 'true'
};
app.use((err, req, res, next) => {
  console.error(err.stack);
  res.status(500).json({
    message: 'An unexpected error occurred. Please try again later.',
    error: config.showErrorDetails ? err : {}
  });
});

Provide meaningful error messages.
Do not expose internal stack traces in production.

3.6 Fastify Best Practices

Use Plugins: Leverage Fastify’s plugin system to encapsulate and reuse functionality.

// Configuration object to determine error handling behavior
const config = {
  showErrorDetails: process.env.SHOW_ERROR_DETAILS === 'true'
};
app.use((err, req, res, next) => {
  console.error(err.stack);
  res.status(500).json({
    message: 'An unexpected error occurred. Please try again later.',
    error: config.showErrorDetails ? err : {}
  });
});

Schema Validation: Use Fastify’s built-in schema validation to validate request and response data

const schema = {
  body: {
    type: 'object',
    required: ['username', 'password'],
    properties: {
      username: { type: 'string', minLength: 3, maxLength: 30 },
      password: { type: 'string', pattern: '^[a-zA-Z0-9]{3,30}$' }
    }
  }
};
fastify.post('/signup', { schema }, async (request, reply) => {
  // handle signup
});

Error Handling: Use Fastify’s error handling hooks to manage errors.

fastify.setErrorHandler(function (error, request, reply) {
  request.log.error(error);
  reply.status(500).send({ message: 'An unexpected error occurred. Please try again later.' });
});

Performance: Optimize performance by using Fastify’s asynchronous nature and avoiding blocking operations.

4. Deployment & CI/CD

Use Docker for containerization.
- Create a Dockerfile to define the container image.
Store secrets securely using Azure Key Vault or a secret manager.
Automate builds and deployments with GitHub Actions or Azure DevOps Pipelines.
- Use a GitHub Actions workflow to build and deploy the service.

5. Proof of Concept & Demo

A proof of concept (PoC) will be developed to demonstrate a functional microservice following the above guidelines. This includes:

Setting up a basic CRUD service.
- Create endpoints for creating, reading, updating, and deleting resources.
Connecting to Azure.
- Use @azure/storage-blob to interact with Azure Blob Storage.
Implementing structured logging with Pino.
Writing unit tests with Jest.
- Write tests for controllers, services, and models.
Deploying the microservice via Docker.
- Use docker-compose to define and run multi-container Docker applications.

Overview

1. Microservice Architecture

1.1 Principles

Why Metrics and Tracing are Important

Azure Monitoring and Logging

1.2 Service Structure

1.3 Directory Structure Explanation

2. Technology Stack

3. Best Practices

3.1 TypeScript Guidelines

Use interfaces and types for API payloads.

Enable strict mode in `tsconfig.json`.

3.2 REST API Design

3.3 Logging with Pino

Mask sensitive data before logging.

3.4 Security

3.5 Error Handling

3.6 Fastify Best Practices

4. Deployment & CI/CD

5. Proof of Concept & Demo

References

Overview

1. Microservice Architecture

1.1 Principles

Why Metrics and Tracing are Important

Azure Monitoring and Logging

1.2 Service Structure

1.3 Directory Structure Explanation

2. Technology Stack

3. Best Practices

3.1 TypeScript Guidelines

Use interfaces and types for API payloads.

Enable strict mode in tsconfig.json.

3.2 REST API Design

3.3 Logging with Pino

Mask sensitive data before logging.

3.4 Security

3.5 Error Handling

3.6 Fastify Best Practices

4. Deployment & CI/CD

5. Proof of Concept & Demo

References

Enable strict mode in `tsconfig.json`.