Maxat Akbanov's blog

Linux Namespaces and cgroups: Building Blocks of Modern Containerization

Maxat Akbanov — Fri, 29 Nov 2024 18:37:13 GMT

Linux namespaces and cgroups (control groups) are foundational Linux kernel features used to provide process isolation and resource management. They are core components of containerization technologies like Docker, Kubernetes, Podman and other orchestration platforms.

💡

According to the official documentation of Control Group v2 - cgroup is never capitalized. The singular form is used to designate the whole feature and also as a qualifier as in cgroup controllers. When explicitly referring to multiple individual control groups, the plural form cgroups is used.

These features were first introduced into Linux kernel back in 2002. However, the real container support was added into Linux kernel in 2013.

Linux Namespaces

According to Wikipedia, Linux namespaces:

are a feature of the Linux kernel that partition kernel resources such that one set of processes sees one set of resources, while another set of processes sees a different set of resources.

By resources Linux kernel sees as process IDs, hostnames, user IDs, file names, some names associated with network access, and Inter-process communication.

Types of Namespaces

mnt (Mount Namespace):
- Isolates the file system mount points.
- Processes in different mount namespaces can have different views of the file system hierarchy.
pid (Process ID Namespace):
- Isolates the process ID number space.
- Processes in a PID namespace see a separate set of process IDs, starting at 1 for the init process of that namespace.
net (Network Namespace):
- Isolates network-related resources like network interfaces, routing tables, and IP addresses.
- Each namespace can have its own network stack.
ipc (Inter-Process Communication Namespace):
- Isolates IPC resources such as shared memory, semaphores, and message queues.
  
  💡
  
  To learn more about IPC, check out this video: Linux Internals: Interprocess Communication
uts (UNIX Timesharing System Namespace):
- Isolates hostname and domain name.
- Allows containers to have their own hostname.
- As a result, it allows a single system to appear to have different host and domain names to different processes.
user (User Namespace):
- Isolates user and group IDs.
- Processes in a user namespace can have a different set of user IDs, including root privileges, without affecting the host system. The first process created in a new namespace has PID 1 and child processes are assigned subsequent PIDs. If a child process is created with its own PID namespace, it has PID 1 in that namespace as well as its PID in the parent process namespace.
  
  Image credits: blog.nginx.org

Namespace Example

To create a namespace for isolating the process ID space:

unshare --user --pid --map-root-user --mount-proc --fork bash

This command creates a new namespace with its own user and PID namespaces.

--map-root-user - maps the root user to the new namespace. The user in new namespace will have root permissions
--mount-proc - mounts a new proc filesystem
--fork - ensures that a new process is started immediately in the newly created namespace(s), separating it from the parent process
bash - run the bash command as the new process in the newly created namespace

As you can see, the new namespace only sees its own processes:

Linux Control Groups (cgroups)

cgroups provide resource management by allowing you to allocate, prioritize, deny, or limit system resources (CPU, memory, disk I/O, etc.) for processes. The official definition is:

cgroup - is a mechanism to organize processes hierarchically and distribute system resources along the hierarchy in a controlled and configurable manner.

Key Features

Resource Limiting:
You can configure cgroup to set limits on CPU, memory, disk I/O, and network bandwidth a process can use.
Prioritization:
You can control how much of a resource (CPU, disk, or network) a process can use compared to processes in another cgroup when there is resource contention.
Accounting:
Track resource usage of processes or groups of processes at the cgroup level
Control:
Change the status (freeze, stopped, or restarted) of processes or terminate groups of processes.

Hierarchy and Controllers

cgroups is largely composed of two parts - the core and controllers. cgroup core is primarily responsible for hierarchically organizing processes. A cgroup controller is usually responsible for distributing a specific type of system resource along the hierarchy although there are controllers which can serve purposes other than resource distribution.

Limits CPU usage.
Limits memory usage and manages swapping behavior.
Controls block device I/O.
Tags network packets for Quality of Service (QoS).
Manages access to device nodes.

So basically you use cgroups to control how much of a given key resource (CPU, memory, network, and disk I/O) can be accessed or used by a process or set of processes. Cgroups are a key component of containers because there are often multiple processes running in a container that you need to control together. In a Kubernetes environment, cgroups can be used to implement resource requests and limits and corresponding QoS classes at the pod level.

The following diagram illustrates how when you allocate a particular percentage of available system resources to a cgroup (in this case cgroup1), the remaining percentage is available to other cgroups (and individual processes) on the system.

Image credits: blog.nginx.org

Practical example of using cgroup

Heres an example of the script to create a cgroup, limit CPU usage to 50%, and validate the behavior of the restricted process.

#!/bin/bash# Define variablesCGROUP_NAME="my_cgroup"CGROUP_PATH="/sys/fs/cgroup/cpu/$CGROUP_NAME"CPU_LIMIT=50000       # 50% CPU usage (quota in microseconds)CPU_PERIOD=100000     # Period in microseconds (default is 100ms)# Step 1: Create a new cgroupecho "Creating cgroup at $CGROUP_PATH..."mkdir -p $CGROUP_PATH# Step 2: Set CPU usage limitsecho "Setting CPU limits..."echo $CPU_LIMIT > $CGROUP_PATH/cpu.cfs_quota_usecho $CPU_PERIOD > $CGROUP_PATH/cpu.cfs_period_us# Step 3: Launch a process to test the CPU limitecho "Starting a CPU-intensive process (infinite loop)..."# Launch a background CPU-intensive processbash -c "while :; do :; done" &PROCESS_PID=$!echo "Process started with PID $PROCESS_PID"# Step 4: Add the process to the cgroupecho "Adding process $PROCESS_PID to cgroup..."echo $PROCESS_PID > $CGROUP_PATH/cgroup.procs# Step 5: Monitor CPU usage for the processecho "Monitoring CPU usage (press Ctrl+C to exit)..."while true; do    CPU_USAGE=$(ps -p $PROCESS_PID -o %cpu=)    echo "CPU Usage of PID $PROCESS_PID: $CPU_USAGE%"    sleep 2done

Execute the script:

sudo ./cgroup.sh

Explanation of the Script

Create a cgroup:
- The script creates a directory in /sys/fs/cgroup/cpu/ for the new cgroup.
Set CPU limits:
- cpu.cfs_quota_us: Specifies the total time (in microseconds) the cgroup is allowed to run on the CPU within each cpu.cfs_period_us period.
- cpu.cfs_period_us: Defines the time period (in microseconds) used to calculate CPU quotas. Default is 1000 microseconds.
Run a CPU-intensive process:
- An infinite loop (while :; do :; done) generates a CPU load for testing.
Assign the process to the cgroup:
- The script uses echo $PROCESS_PID > $CGROUP_PATH/cgroup.procs to attach the process to the cgroup.
Monitor the CPU usage:
- The ps command checks the process's CPU usage percentage in real time.

Observe the CPU Usage:

The script displays the CPU usage of the process every 2 seconds. The %CPU value should not exceed 50%, confirming the limitation.

Summary table for namespaces vs. cgroups

Feature	Namespaces	cgroups
Purpose	Isolates system resources.	Manages and limits resource usage.
Scope	Provides isolation.	Provides control and accounting.
Examples	Process ID, filesystem, network.	CPU, memory, disk I/O.
Use Case	Container isolation.	Resource allocation for containers.

References:

]]>

Stop Leaving Passwords in Your Linux Shell History

Maxat Akbanov — Wed, 27 Nov 2024 10:37:57 GMT

Sometimes when you have to check functioning of service or troubleshoot connection of apps in Development or Test environments, you need to SSH into server and run commands with sensitive data.

For example you may run the applications docker container with specific env variables in order to check the containers connection to the database:

docker run -d --name app \-p 80:3000 \-e DB_HOST=my-db.us-east-1.rds.amazonaws.com \-e DB_USER=<username> \-e DB_PASSWORD=<password>-e DB_NAME=my_dbdkr.ecr.us-east-1.amazonaws.com/app-repo:v1.0.0

As you can see, here we type sensitive values of DB_USER and DB_PASSWORD right into the shell. If not properly managed, these values end up in shell history where everyone who has access to the server can view it.

To avoid storing sensitive commands in the shell history, follow these best practices:

Use a Space Before the Command

If your shell supports it (e.g., bash), prefix the command with a space. This relies on the HISTCONTROL environment variable being set to ignorespace. Using the HISTCONTROL variable you can control how bash stores your command history. You can tell it to ignore duplicate commands and/or to ignore commands that have leading whitespace.

export HISTCONTROL=ignorespace

If HISTCONTROL variable has the value of ignoreboth then the setting for using the space prefix is set.

echo $HISTCONTROL

Remember that this setting is temporary for the duration of running shell session.

You can set the flags in your ~/.bashrc file or in the global /etc/bash.bashrc file. The following command would append it to your ~/.bashrc file:

echo "HISTCONTROL=ignoreboth" >>~/.bashrc

Disable History Temporarily

Disable history logging for the current shell session before running the command:

set +o historypsql -h <db_endpoint> -U <db_name> -p 5432set -o history

Use a Secure Method to Read Passwords

Use tools like read with -s to hide the password input:

read -sp "Enter Password: " PASSWORDdocker run -d --name app \-p 80:3000 \-e DB_HOST=my-db.us-east-1.rds.amazonaws.com \-e DB_USER=<username> \-e DB_PASSWORD=$PASSWORD-e DB_NAME=my_dbdkr.ecr.us-east-1.amazonaws.com/app-repo:v1.0.0

Remove Specific Entries from History

If a sensitive command has been accidentally logged, remove it from the history:

history -d <line_number>

Or clear the entire history:

history -c

Set Alias to Clean History After Logout

In your .bash_aliases file set the following logout command to clean the history and exit the session:

alias logout="history -c && exit"

Set Patterns to Ignore Specific Commands

The HISTIGNORE environment variable controls which commands are entered into the history and which are ignored. It contains a list of patterns separated by colons (:) that when matched, drops a command rather than includes it in the file. For example to ignore all export commands:

HISTIGNORE=" :export *"

You do not need to export this variable.

Be warned however than ignoring an item from the history means that it cannot be recalled by any means not even with the up and down arrows.

Use Encrypted Storage for Sensitive Variables

For frequently used sensitive data, consider using tool like pass. This is a password manager for Linux systems.

To install pass on Ubuntu:

sudo apt-add-repository universesudo apt-get updatesudo apt-get install pass

For example, to store the API key for a service like GitHub:

pass init --path=github/api_key gpg-id="<GPG_ID>" # Initialize storepass insert github/api_key/my_pass # Store password

Retrieve the API key securely when needed:

API_KEY=$(pass github/api_key/my_pass)

Use the API key in your command:

curl -H "Authorization: token $API_KEY" https://api.github.com/user

Use GUI Tools to Manage Shell History

You can use the open-source shell manager tools like altuin to manage your shell history.

To delete specific entries in history, select specific command with arrow, type <CTR-O> and then <CTR-D>.

References:

]]>

Basics of Remote Procedure Call (RPC)

Maxat Akbanov — Fri, 25 Oct 2024 10:51:30 GMT

Remote Procedure Call (RPC) is a powerful technique used in computer networks and distributed systems to enable a program on one computer (the client) to execute code on a remote system (the server) as if it were a local procedure call. By abstracting the complexity of network communication, RPC simplifies the development of distributed applications by allowing developers to write code that looks like standard, local procedure calls.

How RPC works

Consider an example of online shop with the payment system for customers. When customer makes payment on online shop website, under the hood it sends request to a third-party service on another server that processes the payment card details.

When call is made to the payment service, the code that resides on the online shopping site makes an RPC call to the cardholders bank via the third-party payment processing service.

The pseudo code for such interaction might look like the following:

// Function to process payment during checkoutfunction processPayment(orderDetails, paymentInfo) {    // Prepare the payment request object with necessary details    paymentRequest = {        orderId: orderDetails.orderId,        amount: orderDetails.totalAmount,        currency: orderDetails.currency,        paymentMethod: paymentInfo.method,        // e.g., 'CreditCard', 'PayPal'        cardNumber: paymentInfo.cardNumber,        cardExpiry: paymentInfo.cardExpiry,        cardSVC: paymentInfo.cardSVC,        billingAddress: paymentInfo.billingAddress,        customerEmail: orderDetails.customerEmail,        // Additional fields as required...    }    // Create an RPC client proxy to the payment processing service    paymentService = createRPCProxy("https://payment-service.com/api")    // Make the remote procedure call to process the payment    try {        // Invoke the 'processPayment' method on the payment service        paymentResponse = paymentService.processPayment(paymentRequest)        // Check the response status        if (paymentResponse.status == "Success") {            // Payment was successful            updateOrderStatus(orderDetails.orderId, "Paid")            sendConfirmationEmail(orderDetails.customerEmail)            displayMessage("Your payment was successful. Thank you for your purchase!")        } else {            // Payment failed            displayMessage("Payment failed: " + paymentResponse.errorMessage)            // Optionally, prompt the user to try again or use a different payment method        }    } catch (rpcError) {        // Handle any errors that occurred during the RPC call        displayMessage("An error occurred while processing your payment. Please try again later.")        logError(rpcError)        // Additional error handling as needed...    }}

Making the RPC Call:

The call is made with processPayment(paymentRequest) as if it were a local function.
The RPC mechanism handles serialization of paymentRequest and communication over the network (using JSON, XML, etc formats for data representation).

In detail the communication with payment service involves:

Core Concepts

Transparency: RPC aims to make the remote call appear identical to a local call from the programmer's perspective. This means handling data serialization, network communication, and error detection behind the scenes.
Client-Server Model: The architecture typically involves a client making a request to a server, which performs the desired operation and returns the result to the client.
Stubs and Skeletons:
- Client Stub: Acts as a proxy for the client, responsible for packaging the procedure call into a message format suitable for transmission over the network (marshalling).
- Server Skeleton: Receives the message on the server side, unpacks it (unmarshalling), and invokes the appropriate procedure.
Marshalling and Unmarshalling:
- Marshalling: The process of converting procedure parameters into a standard format for transmission.
- Unmarshalling: The reverse process, converting the received data back into a format usable by the server's procedure.

RPC Procedure

Procedure Call: The client invokes a procedure as if it were local.
Client Stub Processing: The client stub intercepts the call, marshals the parameters, and sends a message to the server.
Network Communication: The message is transmitted over the network using a communication protocol (e.g., TCP/IP).
Server Skeleton Processing: The server skeleton receives the message, unmarshals the parameters, and invokes the corresponding procedure on the server.
Execution and Response: The server executes the procedure and sends the result back through the skeleton.
Result Handling: The client stub receives the response, unmarshals the data, and returns it to the client application.

Types of RPC

Synchronous RPC: The client waits (blocks) until the server responds. This is the traditional form of RPC.
Asynchronous RPC: The client continues processing and can handle the server's response at a later time, improving concurrency and resource utilization.

Protocols and Implementations

ONC RPC (Open Network Computing RPC): Developed by Sun Microsystems, commonly used in UNIX systems.
DCE RPC (Distributed Computing Environment RPC): Developed by the Open Software Foundation, provides additional features like authentication and directory services.
XML-RPC and JSON-RPC: Use XML or JSON over HTTP for communication, making them suitable for web-based applications.
gRPC: An open-source RPC framework developed by Google, which uses HTTP/2 and Protocol Buffers for efficient communication.

Applications of RPC

Distributed Systems: Facilitates communication between different components in a distributed architecture.
Microservices: Enables services to communicate in a microservices architecture.
File Systems: NFS (Network File System) uses RPC for remote file operations.
Web Services: Underlying mechanism for many SOAP-based web services.

Advantages

Simplifies Development: Abstracts the complexity of network programming.
Interoperability: Allows systems written in different languages to communicate.
Modularity: Encourages a modular approach to system design.

Disadvantages

Latency and Network Issues: Network communication introduces latency and potential for failures.
Security Risks: Exposing procedures over a network can introduce security vulnerabilities.
Complex Debugging: Errors may occur in the network layer, making debugging more challenging.

Security Considerations

Authentication and Authorization: Ensuring only authorized clients can make RPC calls.
Encryption: Protecting data transmitted over the network using SSL/TLS.
Input Validation: Preventing injection attacks by validating input parameters.

Practice: Making an RPC call to a Service Running on EC2 Instance

This example demonstrates how to make a Remote Procedure Call from your local laptop to a service running on a remote AWS EC2 instance. It uses Python's built-in xmlrpc library for simplicity. Example also provides Terraform code to provision the necessary AWS infrastructure, including the EC2 instance and networking components.

To learn more, checkout this repository:
https://github.com/Brain2life/terraform-cookbook/tree/main/example-rpc-call-to-ec2

References:

YouTube: Distributed Systems 1.3: RPC (Remote Procedure Call)

]]>

AWS Route 53: Latency-based Routing Policy

Maxat Akbanov — Fri, 18 Oct 2024 08:21:34 GMT

Latency-based routing (LBR) in AWS Route 53 is designed to route end-user requests to the AWS region that provides the lowest latency. This routing policy ensures that users are connected to the closest and fastest endpoint (in terms of network latency), enhancing performance by reducing delays in the communication path between the users and your application. Data about the latency between users and your resources is based entirely on traffic between users and AWS data centers.

How It Works

AWS measures network latency between different regions and end users.
When a DNS request is received, Route 53 determines which endpoint (from your configured set of endpoints) offers the lowest latency to the users location.
Route 53 returns the IP address (or CNAME) of the selected endpoint to the user's browser or application.

This helps ensure that users from different parts of the world are connected to the fastest AWS region, reducing response times and improving user experience.

Latency between hosts on the internet can change over time as a result of changes in network connectivity and routing. Latency-based routing is based on latency measurements taken over a period of time, and the measurements reflect these changes. For example a request that is routed to the Oregon Region this week might be routed to the Singapore Region next week.

Example Scenario

Assume your application is deployed in multiple regions:

US-East (N. Virginia)
EU-Central (Frankfurt)
AP-Southeast (Singapore)

A user from Canada sends a request. Route 53 detects that the US-East region provides the lowest latency and directs the traffic there.
A user from Spain sends a request. Route 53 determines the EU-Central endpoint has the lowest latency and routes the traffic there.
Similarly, users in Hong Kong will be directed to the AP-Southeast region.

Use Case

Latency-based routing is particularly useful when:

You have a multi-region architecture, with your resources (like EC2 instances, load balancers, or API Gateway) deployed across multiple AWS regions.
You want to minimize latency and improve response times for users located around the globe.

Steps to Set Up Latency-Based Routing in Console

Create your resources in multiple AWS regions (like an Application Load Balancer, EC2 instances, or an S3 static website).
Set up a hosted zone in Route 53 for your domain.
Create latency-based records in Route 53:
- Choose A or CNAME record type.
- Select the region associated with each endpoint.
AWS will automatically direct traffic based on the latency between the user and your endpoints.

Latency-Based Routing Example Code in Terraform

resource "aws_route53_record" "latency_us_east_1" {  zone_id = "Z03242191HGJ5WDONGNAD"  # Replace with your hosted zone ID  name    = "latency-us-east-1"  type    = "A"  ttl     = 60  set_identifier = "us-east-1"  records = ["192.0.2.1"]  # Replace with your IP address or DNS name  latency_routing_policy {    region = "us-east-1"  }}resource "aws_route53_record" "latency_us_west_2" {  zone_id = "Z03242191HGJ5WDONGNAD"  name    = "latency-us-west-2"  type    = "A"  ttl     = 60  set_identifier = "us-west-2"  records = ["192.0.2.2"]  # Replace with your IP address or DNS name  latency_routing_policy {    region = "us-west-2"  }}

This Terraform code creates two latency-based records for the Z03242191HGJ5WDONGNAD hosted zode in us-east-1 and us-west-2 regions:

For more information, see Terraform: aws_route53_record resource

Health Checks and Latency-Based Routing

Health checks can be integrated with latency-based routing to ensure that Route 53 sends requests only to healthy endpoints. If one of your regions is down, Route 53 will automatically exclude it from routing decisions.

Limitations

Latency-based routing only improves network performance, not application processing time.
Not suitable for use cases where traffic should be directed to a specific endpoint (like compliance or regulatory needs).
AWS regions may experience dynamic latency changes based on internet conditions, so results are not always guaranteed to be consistent.

Summary

AWS Route 53s Latency-Based Routing Policy is an excellent way to ensure low-latency experiences by directing traffic to the nearest and fastest AWS region. It is particularly useful for global applications with users distributed across different geographical locations, such as content delivery systems, real-time gaming platforms, and multi-region web applications.

]]>

Securing Secrets with SOPS: An Introduction

Maxat Akbanov — Thu, 17 Oct 2024 11:10:24 GMT

SOPS (Secrets OPerationS) is an open-source tool developed by Mozilla for managing secrets such as passwords, API keys, and confidential configuration data in a secure and convenient manner. Initially launched in 2015 as a Mozilla project, it has been donated and accepted to CNCF on May 17, 2023. It serves as an editor for encrypted files, supporting various formats including YAML, JSON, ENV, INI, and binary files. SOPS integrates seamlessly with key management systems like AWS KMS, GCP KMS, Azure Key Vault, age, and PGP to encrypt and decrypt data.

The main benefit of the tool is that you dont have to encrypt and decrypt the files with one tool and then read them with another tool. Sops allows you to edit and read the secret values right inside the sops text editor! Hence, you avoid using two tools and simplify the secret management. The secret values inside the sops editor are encrypted and decrypted automatically.

SOPS is a powerful tool that bridges the gap between security and usability in secret management. By integrating with industry-standard key management systems and supporting a variety of file formats, it provides a secure and efficient way to handle sensitive information within collaborative and automated environments.

Key Features

Multi-Format Support
- YAML, JSON, ENV, INI, and Binary Files: SOPS can handle a variety of file formats commonly used in configuration and environment settings.
Encryption Mechanisms
- AWS KMS, GCP KMS, Azure Key Vault: Leverage cloud-based key management services to encrypt and decrypt secrets.
- age: A modern, simple, and secure file encryption tool.
- PGP: Utilize Pretty Good Privacy keys for encryption, supporting both public and private key pairs.
Selective Encryption
- Encrypts Only the Values: SOPS encrypts the values within the file while keeping the keys and structure in plaintext. This approach allows the file to remain human-readable in terms of structure and facilitates easier version control and diffing.
Version Control Friendly
- Git Integration: Encrypted files can be safely stored in version control systems like Git, enabling collaborative workflows without exposing sensitive information.
Ease of Use
- Transparent Editing: SOPS allows you to edit encrypted files as if they were unencrypted. It handles the encryption and decryption transparently during the editing process.
- Command-Line Interface: Provides a CLI sops tool for easy integration into scripts and automation pipelines.

Installation on Linux

Binaries and packages of the latest stable release are available at https://github.com/getsops/sops/releases.

To install sops on Linux on an AMD64 architecture:

 # Download the binary curl -LO https://github.com/getsops/sops/releases/download/v3.9.1/sops-v3.9.1.linux.amd64 # Move the binary in to your PATH sudo mv sops-v3.9.1.linux.amd64 /usr/local/bin/sops # Make the binary executable sudo chmod +x /usr/local/bin/sops

Install cosign tool for checksum and signature verification:

 # Download the latest release file for your Linux architecutre (e.g. amd64) wget https://github.com/sigstore/cosign/releases/download/v2.4.1/cosign-linux-amd64 # Move the binary in to your PATH sudo mv cosign-linux-amd64 /usr/local/bin/cosign # Make the binary executable chmod +x /usr/local/bin/cosign

Verify the installation:

 # Download the checksums file, certificate and signature curl -LO https://github.com/getsops/sops/releases/download/v3.9.1/sops-v3.9.1.checksums.txt curl -LO https://github.com/getsops/sops/releases/download/v3.9.1/sops-v3.9.1.checksums.pem curl -LO https://github.com/getsops/sops/releases/download/v3.9.1/sops-v3.9.1.checksums.sig # Verify the checksums file cosign verify-blob sops-v3.9.1.checksums.txt \   --certificate sops-v3.9.1.checksums.pem \   --signature sops-v3.9.1.checksums.sig \   --certificate-identity-regexp=https://github.com/getsops \   --certificate-oidc-issuer=https://token.actions.githubusercontent.com

How SOPS Works

Data Encryption Key (DEK)
- SOPS generates a symmetric Data Encryption Key for encrypting the actual data within the file.
Master Key Encryption
- The DEK is encrypted using one or more master keys from your chosen key management services (e.g., AWS KMS, GCP KMS, Azure Key Vault, PGP).

File Structure Preservation
- By encrypting only the values and not the keys or structure, SOPS ensures that the file remains partially readable. This feature is particularly useful for understanding the configuration without exposing sensitive data.
Decryption Process
- When you open a file with SOPS, it decrypts the values on-the-fly using the appropriate master keys. After editing, it re-encrypts the values before saving.

Use Cases

Secure Configuration Management
- Store application configurations that contain secrets in a secure manner.
Infrastructure as Code (IaC)
- Embed encrypted secrets within your IaC templates for tools like Kubernetes, Terraform, or Ansible.
Collaborative Environments
- Allow multiple team members to access and edit secrets securely without sharing plaintext secrets.
Automated Deployment Pipelines
- Integrate SOPS into CI/CD pipelines to decrypt secrets during the deployment process securely.

Using SOPS with PGP and a text file

Below is an example of how to encrypt and decrypt a text file using SOPS with PGP.

Prerequisites

Install GPG (GNU Privacy Guard):
On Ubuntu:
```
 sudo apt install gnupg
```
Generate a PGP key pair if you don't already have one:
```
 gpg --full-generate-key
```
- Use RSA encryption (default).
- Provide your name, email, and password as requested.
List the generated key and keep the note of the key fingerprint:
```
 gpg --list-secret-keys "<your_key_name>"
```

Export the PGP public key to encrypt files:

 gpg --armor --export [key_id] > my-public-key.asc

Export the PGP private key to decrypt files:

 gpg --armor --export-secret-keys [key_id] > my-private-key.asc

Encrypting a Text File using SOPS and PGP

Create a sample text file to encrypt:
```
 echo "my-secret-password" > secret.txt
```
Configure the default encryption and decryption behavior of SOPS tool. Create a file named .sops.yaml under the $HOME directory with the Key ID from the previous steps:
```
 creation_rules:     - pgp: >-         C0B4391E80CDBF4FCE820212F9D999A91B838104
```
Encrypt the file using SOPS and your PGP key:
```
 sops --encrypt secret.txt > secret.enc.txt
```
This will encrypt the secret.txt file with the default PGP key specified in .sops.yaml file.
Verify the encrypted content:
```
 cat secret.enc.txt
```
You should see a YAML structure with encrypted content that looks like this:

Decrypting the Text File

Use SOPS to decrypt the encrypted file:

 sops --decrypt secret.enc.txt > decrypted.txt

View the decrypted content:
```
 cat decrypted.txt
```
You should see the original text:
```
 my-secret-password
```

Automating Decryption in Configuration Files

You can use SOPS-encrypted files in your configuration by decrypting them on the fly, such as:

export MY_SECRET=$(sops --decrypt --input-type binary secret.enc.txt)

This example shows how to use SOPS with PGP to encrypt and decrypt a text file. With PGP encryption, you can ensure that only users with the appropriate private key can decrypt the file, making it ideal for managing secrets in infrastructure code or configuration files.

References:

]]>

AWS Route 53: Weighted Routing Policy

Maxat Akbanov — Wed, 16 Oct 2024 10:25:54 GMT

The Weighted Routing Policy allows you to route DNS query traffic to multiple resources in proportions that you specify. By assigning a relative weight (a numerical value) to each DNS record, you can control the percentage of requests that is directed to various endpoints.

Amazon Route 53 sends traffic to a resource based on the weight that you assign to the record as a proportion of the total weight for all records in the group:

NOTE: Weights dont need to sum up to 100%. Each record in weighted policy can have a health checks.

How It Works:

Weights Assignment: Each DNS record (e.g., IP addresses, URLs) is assigned a weight between 0 and 255 and it must have the same name and type.
Traffic Distribution: Route 53 uses these weights to determine the probability of returning each record in response to DNS queries.
Proportional Routing: The proportion of traffic directed to each resource is calculated based on the weight of that resource relative to the total weight.

Example:

Suppose you have three servers:

Server A: Weight 70
Server B: Weight 20
Server C: Weight 10

Total weight = 70 + 20 + 10 = 100

Server A will receive 70% of the traffic.
Server B will receive 20% of the traffic.
Server C will receive 10% of the traffic.

💡

Assigning a weight of 0 to a record will result in stopping sending traffic to a resource. If all records have weight of 0, then all records will be returned equally.

Use Cases for Weighted Routing Policy

Load Balancing Across Resources
- Scenario: Distribute traffic between multiple servers or resources to balance the load.
- Benefit: Improves resource utilization and application performance.
- Implementation: Assign equal weights for equal distribution or proportional weights based on server capacity.
A/B Testing
- Scenario: Test new features or versions of an application by directing a portion of users to the new version while the rest use the stable version.
- Benefit: Collect user feedback and performance data with minimal risk.
- Implementation: Assign a smaller weight to the new version's DNS record.
Gradual Deployment (Canary Releases)
- Scenario: Gradually roll out updates to a small subset of users before a full-scale deployment.
- Benefit: Detect issues early and minimize impact on the user base.
- Implementation: Start with a low weight for the new release and incrementally increase it.
Testing Infrastructure Changes
- Scenario: Assess how infrastructure changes handle production traffic without affecting all users.
- Benefit: Validate performance and stability under real-world conditions.
- Implementation: Divert a controlled percentage of traffic to the updated infrastructure.
Resource Scaling Based on Capacity
- Scenario: Allocate more traffic to resources with higher capacity or performance.
- Benefit: Optimizes resource usage and cost-effectiveness.
- Implementation: Set weights proportional to each resource's capacity.
Disaster Recovery and Failover
- Scenario: Maintain backup resources that can receive traffic if primary resources fail.
- Benefit: Enhances application availability and resilience.
- Implementation: Assign a weight of 0 to backup resources, increasing it when needed.
Multi-Region Deployment
- Scenario: Distribute traffic across different geographic regions.
- Benefit: Reduces latency and improves user experience globally.
- Implementation: Assign weights based on desired traffic distribution between regions.

Implementing Weighted Routing in Route 53

Create Hosted Zone:
- Set up a hosted zone for your domain in Route 53.
Add DNS Records:
- For each resource (e.g., server, load balancer), create a DNS record with the same name and type.
- Example: Multiple A records for www.example.com.
Assign Weights:
- In each DNS record, specify the desired weight.
- The weights determine the traffic proportion.
Optional - Set Health Checks and Record ID:
- Associate health checks with DNS records to ensure traffic is only routed to healthy resources.
- Set alias names for your DNS record via Record ID

Save Configuration:
- Apply the changes to make them active.

Considerations When Using Weighted Routing

Health Checks:
- Purpose: Ensure traffic isn't routed to unhealthy resources.
- Implementation: Configure Route 53 health checks for each resource.
DNS Caching and TTL:
- Impact: Changes in routing may not be immediate due to DNS caching.
- Solution: Adjust the Time to Live (TTL) settings for quicker propagation.
  
  💡
  
  Not all recursive DNS servers strictly follow TTL values. You cannot force them to update their caches, some DNS resolvers dont respect TTLs at all and just implement their own caching policy however they wish. It may take considerable time, ranging from one to several days, for TTL updates to become effective. You may want to set the TTL to a lower value before making DNS changes such as 60 seconds, but in the real world, the common DNS TTL before making changes is to lower it to 300 seconds, and once the change is done you can set it back to 3600(1 hour) or 86,400 seconds(24 hours). [Source: Inside AWS Route53's Weighted Routing Policyby@devgrowth ]
Relative Weights:
- Understanding: The actual weight values are relative; it's the ratio that matters.
- Example: Weights of 1 and 3 have the same effect as 10 and 30.
Monitoring and Logging:
- Importance: Keep track of traffic distribution and resource performance.
- Tools: Use AWS CloudWatch and other monitoring services.

Limitations

Complexity: Managing multiple weights can become complex in large-scale environments.
DNS Resolver Behavior: Some resolvers may cache DNS responses longer than expected, affecting traffic distribution.
Not Real-Time: Changes in weights may take time to reflect due to DNS propagation delays.

Practical Example

Scenario: You run an e-commerce website with servers in the US and Europe.

Objective: Direct 70% of global traffic to the US servers and 30% to the European servers.

Implementation Steps:

US Server DNS Record:
- Name: www.example.com
- Type: A
- Value: IP address of US server
- Weight: 70
European Server DNS Record:
- Name: www.example.com
- Type: A
- Value: IP address of European server
- Weight: 30
Result:
- Route 53 routes approximately 70% of the traffic to the US server and 30% to the European server.

The AWS Route 53 Weighted Routing Policy is a powerful feature that provides granular control over how traffic is distributed to your resources. By adjusting weights, you can implement advanced traffic management strategies such as load balancing, A/B testing, and gradual deployments.

]]>

AWS Route 53: Simple Routing Policy

Maxat Akbanov — Tue, 15 Oct 2024 12:03:59 GMT

AWS Route 53 is Amazon's scalable and highly available Domain Name System (DNS) web service. It translates human-readable domain names (like www.example.com) into IP addresses (like 198.51.100.42) that computers use to connect to each other. Within Route 53, routing policies determine how DNS queries are answered. One of these policies is the Simple Routing policy, which is the default and most straightforward option.

What Is the Simple Routing Policy?

The Simple Routing policy is used when you want Route 53 to respond to DNS queries with a single record. It's ideal for straightforward configurations where multiple endpoints or complex traffic management aren't necessary.

Single Value Record: You create one DNS record for a domain or subdomain.
Multiple Values Record: While it's called "simple," you can include multiple IP addresses or endpoints in the same record.
No Advanced Features: This policy does not support health checks, weighted distribution, or geographic routing.

How Does It Work?

When a DNS query is made for your domain:

DNS Query Received: A user tries to access your domain (www.example.com), initiating a DNS query.
Route 53 Responds: Route 53 looks up the DNS record associated with that domain.
Returns Record Values: It returns all the IP addresses or endpoints specified in the record, typically in a random order.
Client Chooses Endpoint: The client's DNS resolver selects one of the returned IP addresses to establish a connection.

Key Characteristics

Client-Side Load Balancing: If multiple IP addresses are provided, the client's DNS resolver decides which one to use. This can lead to basic load balancing but isn't as controlled as other routing policies.
No Health Checks: Route 53 doesn't check whether the endpoints are healthy. If an endpoint is down, its IP address is still included in DNS responses.
Simplicity: Configuration is straightforward, making it suitable for simple use cases.
One AWS resource for Alias record: When Alias record enabled you can specify only one AWS resource.

When to Use the Simple Routing Policy

Single Endpoint: You have one server or resource serving all traffic.
Multiple Identical Endpoints: Multiple servers serve the same content, and you don't need to control traffic distribution.
Basic DNS Needs: You don't require advanced features like failover, latency optimization, or geolocation-based routing.

Limitations

No Traffic Control: You can't specify how traffic is distributed among multiple endpoints.
Potential Downtime: Without health checks, users may be directed to an unavailable endpoint.
Not Ideal for Complex Architectures: Lacks support for advanced routing strategies needed in sophisticated deployments.

Configuration Steps

Access Route 53 Console: Log in to your AWS Management Console and navigate to Route 53.
Create or Select a Hosted Zone: Choose the hosted zone corresponding to your domain.
Create a Record Set:
- Click on "Create Record."
- Name: Enter the domain or subdomain (e.g., www for www.example.com).
- Type: Choose the record type (e.g., A, AAAA, CNAME).
- Value(s): Enter the IP address(es) or endpoint(s).
- Routing Policy: Select "Simple."
- TTL (Time to Live): Set how long DNS resolvers cache the record.
Save: Confirm and save the record.

Summary

The Simple Routing policy in AWS Route 53 is ideal for basic DNS configurations where advanced routing mechanisms aren't required. It maps a domain directly to one or more IP addresses or endpoints without considering the health or performance of those endpoints. This simplicity makes it easy to set up and manage but comes with limitations that may not suit more complex or critical applications

]]>

AWS Route 53 Alias vs CNAME Records

Maxat Akbanov — Mon, 14 Oct 2024 18:19:14 GMT

A CNAME (Canonical Name) record is a type of DNS record used to alias one domain name to another. When a DNS resolver encounters a CNAME record, it will replace the original domain with the target domain and make a new DNS query for the target.

For example, suppose blog.example.com has a CNAME record with a value of "example.com" (without the "blog"). This means when a DNS server hits the DNS records for blog.example.com, it actually triggers another DNS lookup to example.com, returning example.coms IP address via its A record. In this case we would say that example.com is the canonical name (or true name) of blog.example.com.

Characteristics of CNAME:

Aliases one domain to another domain: The domain with the CNAME will inherit all records (e.g., A, MX, etc.) from the target domain.
Common Use Case: Typically used to map subdomains (e.g., www.example.com) to the main domain (example.com) or to an external service.
Limitations:
- MX and NS records cannot point to a CNAME record; they have to point to an A record (for IPv4) or an AAAA record (for IPv6). An MX record is a mail exchange record that directs email to a mail server. An NS record is a "name server" record and indicates which DNS server is authoritative for that domain.
- It cannot be used for the root domain (also known as the "apex" or "naked" domain, such as example.com).

Example:

Reference: Cloudflare

When a user requests blog.example.com, DNS will look for the records of example.com and return them, usually the A/AAAA record.

How to Create a CNAME Record in AWS Route 53

To create a CNAME record in AWS Route 53, follow these steps:

Log in to AWS Console:
- Go to AWS Management Console.
- Use your credentials to log in.
Open Route 53:
- In the search bar at the top, type Route 53 and select it from the results.
Select Hosted Zones:
- In the Route 53 dashboard, click on Hosted Zones.
- Select the hosted zone for the domain where you want to create the CNAME record.
Create a Record:
- In the hosted zone, click the Create record button.
Set Record Type and Details:
- Name: Enter the subdomain name you want to map (e.g., blog.example.com).
- Record Type: Select CNAME - Canonical name from the drop-down menu.
- Value: Enter the target domain (e.g., example.com or another domain like service.example.com).
- Leave the default TTL (Time to Live) or adjust if necessary.
- Routing Policy: Use the default Simple routing unless you need more advanced options like failover.
Create the Record:
- Review your settings and click Create records.

This will create a CNAME record mapping blog.example.com to example.com.

AWS Route 53 Alias Records

Alias records are AWS-specific and provide functionality similar to CNAME but with additional features tailored to AWS services.

Characteristics of AWS Alias Records:

Integration with AWS Services: Alias records allow you to map a domain name to AWS services like CloudFront distributions, S3 buckets (website mode), Elastic Load Balancers (ELBs), and API Gateways.
Root Domain Support: Unlike CNAME records, Alias records can be used at the root domain (e.g., example.com) to point to an AWS resource. This is because AWS manages the DNS lookup internally.
Free Queries for AWS Targets: Alias records do not incur additional DNS query charges when they point to AWS services.
Fast Failover: Since theyre integrated into Route 53, they can be used for Route 53-specific features like health checks and failover.

Example:

In this case, the root domain example.com can point to an AWS Application Load Balancer using an Alias record, something that isnt possible with a traditional CNAME.

When you use an alias record to route traffic to an AWS resource, Route 53 automatically recognizes changes in the resource. In the given example, an alias record for example.com points to an Elastic Load Balancing load balancer at my-load-balancer-1234567890.us-west-2.elb.amazonaws.com. If the IP address of the load balancer changes, Route 53 automatically starts to respond to DNS queries using the new IP address.

If an alias record points to an AWS resource, you can't set the time to live (TTL); Route53 uses the default TTL for the resource. If an alias record points to another record in the same hosted zone, Route53 uses the TTL of the record that the alias record points to.

Key Differences between CNAME and Alias Records

Feature	CNAME	AWS Alias
AWS Service Integration	No	Yes
Root Domain Support	No	Yes
DNS Query Cost	Standard DNS pricing applies	Free if pointing to AWS resources
Works with AWS Services	No	Yes
Use Outside AWS	Yes	No (AWS-specific)
Failover & Health Checks	Not supported	Supported (when using Route 53)

When to Use Each

CNAME Record:
- Use it when you want to alias one domain to another domain or subdomain outside AWS, or if you're mapping subdomains to external services (like app.example.com pointing to thirdparty.mywebsite.com).
- Avoid using it at the root domain (example.com) as DNS specifications do not allow CNAME records at the apex of a domain.
Alias Record:
- Use when integrating with AWS resources (like CloudFront, ELB, or S3) to take advantage of AWS-specific optimizations, especially when you need to map a root domain (example.com) to AWS services.
- Preferred when using Route 53 for advanced DNS features like failover, routing policies, and health checks.

In summary, CNAME records are more universal and suited for general aliasing, while Alias records are AWS-specific and offer extended functionality when dealing with AWS infrastructure and services.

How to Create Route 53 Alias Record by using Console

Sign in to the AWS Management Console and open the Route 53 console.
In the navigation pane, choose Hosted zones.
Choose the name of the hosted zone that you want to use to route traffic to your App Runner service.
Choose Create record.
Specify the following values:
- Routing policy: Choose the applicable routing policy. For more information, see Choosing a routing policy.
- Record name: Enter the domain name that you want to use to route traffic to your App Runner service. The default value is the name of the hosted zone. For example, if the name of the hosted zone is example.com and you want to use acme.example.com to route traffic to your environment, enter acme.
- Value/Route traffic to: Choose Alias to App Runner Application, then choose the Region that the endpoint is from. Choose the domain name of the application that you want to route traffic to.
- Record type: Accept the default, A IPv4 address.
- Evaluate target health: Accept the default value, Yes.
Choose Create records.

The Route 53 alias record that you created gets propagated on all Route 53 servers within 60 seconds. When the Route 53 servers are propagated with your alias record, you can route traffic to your App Runner service by using the name of the alias record that you created.

References:

]]>

Understanding the Risks of Terraform Code Refactoring

Maxat Akbanov — Sat, 05 Oct 2024 14:18:58 GMT

Refactoring - is the process of restructuring existing code without altering its external behavior or functionality. The goal is to improve the code's internal structure, readability, and maintainability.

Refactoring is an essential coding practice that should be done on regularly basis. It is true for general purpose programming languages. But when it comes to IaC like tools (Terraform), changing the internal structure of the code on the purpose of refactoring can be a risky operation. It can alter the external behavior and cause downtimes, crashes and many, many other scary things.

Let me show you how a common and simple refactoring technique such as renaming a resource name, can be a catastrophic in terms of Terraform.

Imagine that you have the following code that defines your EC2 instance:

resource "aws_instance" "web" {  ami           = "ami-0c55b159cbfafe1f0"  instance_type = "t2.micro"}

During refactoring process you come up with an idea to change the resource name to a more meaningful name such as web_server instead of web:

resource "aws_instance" "web_server" {  ami           = "ami-0c55b159cbfafe1f0"  instance_type = "t2.micro"}

What will happen when you try to apply this change?

Terraform will recreate the resource. Terraform sees aws_instance.web_server as a new resource and aws_instance.web as no longer needed. First Terraform destroys the existing one and then creates a new resource with identifier web_server. During this process your web server will be in a downtime. Thats not what you wanted.

The proper way of renaming a resource in the example above will be by using a terraform state mv command:

terraform state mv aws_instance.web aws_instance.web_server

NOTE: Staring from Terraform v1.1 and above you can use explicit refactoring declarations with moved blocks instead of terraform state mv command.

💡

Keep in mind that Terraform is not like a general-purpose programming language and the core nature of it is operating with the state of the infrastructure. The default behavior of Terraform is: destroy old then create new.

Safeguards:

So, how to keep yourself safeguarded? The main rule is to use terraform plan and inspect it thoroughly. Verify that your changes are not destroying or creating something new.

The general list of rules are:

Plan Before Applying: Always run terraform plan to see the implications of your changes before applying them.
Update All References: Ensure that all usages of renamed variables or resources are updated.
Manage State Carefully: Use Terraform state commands to reflect changes in resource names.
Use Version Control: Keep your Terraform code in a version control system to track changes and facilitate rollbacks if necessary.
Test Changes: Apply refactoring changes in a non-production environment first to observe any unintended effects.

By following these rules you can avoid the pitfalls associated with refactoring and maintain a stable infrastructure with Terraform.

References:

]]>

How 'copy' built-in function works in Go

Maxat Akbanov — Tue, 17 Sep 2024 20:02:59 GMT

The copy built-in function in Go is used to copy elements from one slice to another. It is particularly useful when you need to duplicate or copy portions of a slice into another slice.

Syntax

copy(dst, src []T) int

dst: The destination slice where elements are copied to.
src: The source slice from which elements are copied.
T: The type of elements in the slices.
Return value: The function returns the number of elements that were copied, which is the minimum of the lengths of src and dst.

As a special case, its legal to copy bytes from a string to a slice of bytes.

copy(dst []byte, src string) int

💡

Remember! You can use copy function to create independent slices. This allows to avoid the problem of sharing the same underlying array when using the slice of slices operations.

Rules:

Only the number of elements that both slices can accommodate will be copied (i.e., the minimum length of dst and src).
The copy function works with any slice type, as long as both slices have elements of the same type.

Example:

package mainimport "fmt"func main() {    // Source slice    src := []int{1, 2, 3, 4, 5}    // Destination slice with enough capacity    dst := make([]int, 3)    // Copying elements from src to dst    numCopied := copy(dst, src)    fmt.Println("Source slice:", src)                    // [1, 2, 3, 4, 5]    fmt.Println("Destination slice after copy:", dst)    // [1, 2, 3, 4, 5] --> [0, 0, 0] --> [1, 2, 3]    fmt.Println("Number of elements copied:", numCopied) // 3}

Output:

Source slice: [1 2 3 4 5]Destination slice after copy: [1 2 3]Number of elements copied: 3

Explanation:

The src slice has 5 elements: [1, 2, 3, 4, 5].
The dst slice has a capacity of 3, so only the first 3 elements from src are copied.
The copy function returns the number of elements copied, which is 3 in this case.

Edge Cases:

Destination shorter than source: Only as many elements as the destination can hold are copied.
Source shorter than destination: The copy will stop when all elements of the source have been copied.

package mainimport "fmt"func main() {    src := []int{1, 2}          // [1, 2]    dst := make([]int, 5)       // [0, 0, 0, 0, 0]    numCopied := copy(dst, src) // 2    fmt.Println("Dst:", dst)    // [1, 2, 0, 0, 0]    fmt.Println("Number of copied elements", numCopied)}

In this case, only 2 elements from src are copied into the first two positions of dst.

Benefits:

Efficient copying without needing a loop.
Works with slices of any type (integers, strings, structs, etc.).

Thats the basic idea of the copy function in Go!

References:

]]>

Get started with Kubernetes Annotations

Maxat Akbanov — Mon, 08 Jul 2024 18:31:16 GMT

Annotations in Kubernetes are a way to attach arbitrary non-identifying metadata to objects. They are key-value pairs, and unlike labels, they are not used to identify and select objects. Instead, annotations can be used to store additional information that can be useful for various purposes without affecting the behavior of Kubernetes itself.

The metadata in an annotation can be small or large, structured or unstructured, and can include characters not permitted by labels. It is possible to use labels as well as annotations in the metadata of the same object.

Annotations, like labels, are key/value maps:

"metadata": {  "annotations": {    "key1" : "value1",    "key2" : "value2"  }}

The keys and the values in the map must be strings. In other words, you cannot use numeric, boolean, list or other types for either the keys or the values.

Key Characteristics of Annotations:

Valid annotation keys have two segments: an optional prefix and name, separated by a slash (/). The name segment is required and must be 63 characters or less, beginning and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between. The prefix is optional. If specified, the prefix must be a DNS subdomain: a series of DNS labels separated by dots (.), not longer than 253 characters in total, followed by a slash (/).

The kubernetes.io/ and k8s.io/ prefixes are reserved for Kubernetes core components.

Arbitrary Key-Value Pairs: Annotations are free-form key-value pairs that can be used to attach any type of metadata to Kubernetes objects.
Not Used for Selection: Unlike labels, annotations are not used in selectors (e.g., for service selection, replication controller selection).
No Constraints on Values: The values of annotations can be any string, including structured data formats like JSON or XML.
Can Be Large: Annotations can store larger amounts of data compared to labels, which are typically limited to a few kilobytes.

Example of Pod manifest with annotation:

apiVersion: v1kind: Podmetadata:  name: annotations-demo  annotations:    imageregistry: "https://hub.docker.com/"spec:  containers:  - name: nginx    image: nginx:1.14.2    ports:    - containerPort: 80

Common Use Cases for Annotations:

Metadata for External Tools:

Annotations can be used to attach information that is used by external tools or systems. For example, a monitoring system might use annotations to store configuration details for collecting metrics from a pod.

apiVersion: v1kind: Podmetadata:  name: mypod  annotations:    monitoring.tool/config: '{"scrape_interval": "30s", "path": "/metrics"}'

Build and Deployment Information:

Annotations can be used to store build and deployment details such as the version of the application, build date, or git commit hash.

apiVersion: v1kind: Podmetadata:  name: mypod  annotations:    build.version: "1.0.0"    build.date: "2024-07-08T12:34:56Z"    build.commit: "abc123def"

Debugging and Auditing:

Annotations can be used to track the source of changes, reasons for deployments, or other debugging and auditing information.

apiVersion: v1kind: Podmetadata:  name: mypod  annotations:    lastUpdatedBy: "DevOps Team"    updateReason: "Security patch"

Configuration and Policy Management:

Annotations can be used to pass configuration data to controllers or admission webhooks that enforce policies or modify the behavior of resources.

apiVersion: v1kind: Podmetadata:  name: mypod  annotations:    policy.enforcer/ignore: "true"

Operational Information:

Annotations can store operational information such as priority data or custom handling instructions for specific environments.

apiVersion: v1kind: Podmetadata:  name: high-priority-pod  annotations:    environment: "production"    custom.scheduler/priority: "high"spec:  containers:  - name: nginx    image: nginx:1.17.4    ports:    - containerPort: 80

Dynamic annotation values

Keep in mind, that annotation values can be populated dynamically in CI/CD tools. In the following examples, you will see the hardcoded values. However, on practice we can generate the annotation value dynamically by using scripts, environment variables and CI/CD toolset.

For example here's .gitlab-ci.yml for GitLab CI/CD that annotates the deployment parameters by using environment variables:

stages:  - build  - docker  - deployvariables:  APP_VERSION: "1.2.3"build:  stage: build  script:    - echo "Building application version $APP_VERSION"    # Add your build commands heredocker-build:  stage: docker  script:    - docker build -t myapp:$APP_VERSION .    - docker tag myapp:$APP_VERSION myregistry/myapp:$APP_VERSION    - docker push myregistry/myapp:$APP_VERSIONdeploy:  stage: deploy  script:    - export BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ")    - export GIT_COMMIT=$(git rev-parse HEAD)    - |      cat <<EOF > deployment.yaml      apiVersion: apps/v1      kind: Deployment      metadata:        name: myapp-deployment        annotations:          build.version: "$APP_VERSION"          build.date: "$BUILD_DATE"          build.commit: "$GIT_COMMIT"          deployment.team: "devops"          deployment.reason: "feature release"      spec:        replicas: 3        selector:          matchLabels:            app: myapp        template:          metadata:            labels:              app: myapp            annotations:              monitoring.tool/config: '{"scrape_interval": "30s", "path": "/metrics"}'          spec:            containers:            - name: myapp-container              image: myregistry/myapp:$APP_VERSION              ports:              - containerPort: 80      EOF    - kubectl apply -f deployment.yaml

Managing Annotations:

Annotations can be managed using kubectl commands. For example:

Add an annotation:

  kubectl annotate pod example-pod example.com/owner="team-a"

Remove an annotation:

  kubectl annotate pod example-pod example.com/owner-

References:

]]>

Kubernetes Logging Basics: Tracking at Container and Pod Levels

Maxat Akbanov — Sun, 07 Jul 2024 10:10:23 GMT

In Kubernetes, logging at the container and pod levels is essential for monitoring and debugging applications. Kubernetes captures logs from each container in a running Pod.

Container-Level Logging

Container logs are generated by individual containers running within pods. These logs typically include application-specific logs, such as error messages, access logs, and other runtime information.

Image credits: Kubernetes

In general Kubernetes handles container logs in the following way:

Containers write their logs to standard output (stdout) and standard error (stderr) streams.
The container runtime, like Docker or containerd, intercepts these streams.
Kubernetes uses a standardized format (CRI logging) to interact with the container runtime and access the logs.
By default, logs are saved in files on the node, typically under /var/log/pods.
You can view container logs using the kubectl logs <pod_name> [container_name] command.
- The container_name is optional if the pod only has one container.
- You can use the -f flag to follow the logs in real-time.

Standard Output and Error Streams:

Containers in Kubernetes are designed to log information to the standard output (stdout) and standard error (stderr) streams. Container runtimes captures these streams and stores the logs for later access. Different container runtimes implement this in different ways; however, the integration with the kubelet is standardized as the Container Runtime Interface (CRI) logging format.

Kubernetes CRI defines a standard for container runtimes to follow, which also affects log format. A typical CRI log entry includes a timestamp, the output stream type, and the log message itself.

2023-10-06T00:17:09.669794202Z stdout F Your log message here2023-10-06T00:17:09.669794202Z stdout P Another log pt 12023-10-06T00:17:09.669794202Z stdout P Another log pt 22023-10-06T00:17:10.113242941Z stderr F Another log final

Log Storage:

The way that the kubelet and container runtime write logs depends on the operating system that the node uses (Linux, Windows).

On Linux nodes that use systemd, the kubelet and container runtime write to journald by default. You use journalctl to read the systemd journal; for example: journalctl -u kubelet.

If systemd is not present, the kubelet and container runtime write to .log files in the /var/log directory. If you want to have logs written elsewhere, you can indirectly run the kubelet via a helper tool, kube-log-runner, and use that tool to redirect kubelet logs to a directory that you choose.

By default, kubelet directs your container runtime to write logs into directories within /var/log/pods.

For more information on kube-log-runner, read System Logs.

Accessing Logs:

You can access container logs using the kubectl logs command:

kubectl logs <pod-name> -c <container-name>

If a pod has only one container, you can omit the -c <container-name> option.

You can use kubectl logs --previous to retrieve logs from a previous instantiation of a container.

Log Aggregation:

While accessing logs directly is useful for quick debugging, for large-scale applications, aggregating logs from all containers and storing them in a centralized location is more efficient. This is usually done using logging agents like Fluentd , Logstash, Loki or others, which collect logs from containers and forward them to a logging backend.

Pod-Level Logging

Pod logs encompass the logs of all containers within a pod. A pod is the smallest deployable unit in Kubernetes and can consist of one or more containers that share the same network namespace and storage volumes.

Multi-Container Pods:

When a pod contains multiple containers, each container logs independently. These logs need to be aggregated to get a complete picture of the pods activities.

In general Kubernetes manages pod level logging in the following way:

A pod might have multiple containers.
Kubernetes captures logs from each container within the pod.
By default, kubectl logs <pod_name> retrieves logs only from the first container.
To view logs from a specific container within a pod, use kubectl logs <pod_name> -c <container_name>.

Sidecar Containers:

Often, a sidecar container is used within the same pod to handle logging. The sidecar can read logs from shared volumes or directly from other containers and forward them to a centralized logging system.

You can use a sidecar container in one of the following ways:

The sidecar container streams application logs to its own stdout.
The sidecar container runs a logging agent, which is configured to pick up logs from an application container.

Image credits: Kubernetes

The sidecar containers read logs from a file, a socket, or journald. Each sidecar container prints a log to its own stdout or stderr stream.

This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr. The logic behind redirecting logs is minimal, so it's not a significant overhead. Additionally, because stdout and stderr are handled by the kubelet, you can use built-in tools like kubectl logs.

For example, a pod runs a single container, and the container writes to two different log files using two different formats. Here's a manifest for the Pod:

apiVersion: v1kind: Podmetadata:  name: counterspec:  containers:  - name: count    image: busybox:1.28    args:    - /bin/sh    - -c    - >      i=0;      while true;      do        echo "$i: $(date)" >> /var/log/1.log;        echo "$(date) INFO $i" >> /var/log/2.log;        i=$((i+1));        sleep 1;      done          volumeMounts:    - name: varlog      mountPath: /var/log  volumes:  - name: varlog    emptyDir: {}

It is not recommended to write log entries with different formats to the same log stream, even if you managed to redirect both components to the stdout stream of the container. Instead, you can create two sidecar containers. Each sidecar container could tail a particular log file from a shared volume and then redirect the logs to its own stdout stream.

Here's a manifest for a pod that has two sidecar containers:

apiVersion: v1kind: Podmetadata:  name: counterspec:  containers:  - name: count    image: busybox:1.28    args:    - /bin/sh    - -c    - >      i=0;      while true;      do        echo "$i: $(date)" >> /var/log/1.log;        echo "$(date) INFO $i" >> /var/log/2.log;        i=$((i+1));        sleep 1;      done          volumeMounts:    - name: varlog      mountPath: /var/log  - name: count-log-1    image: busybox:1.28    args: [/bin/sh, -c, 'tail -n+1 -F /var/log/1.log']    volumeMounts:    - name: varlog      mountPath: /var/log  - name: count-log-2    image: busybox:1.28    args: [/bin/sh, -c, 'tail -n+1 -F /var/log/2.log']    volumeMounts:    - name: varlog      mountPath: /var/log  volumes:  - name: varlog    emptyDir: {}

Now when you run this pod, you can access each log stream separately by running the following commands:

kubectl logs counter count-log-1kubectl logs counter count-log-2

Sidecar containers can also be used to rotate log files that cannot be rotated by the application itself. An example of this approach is a small container running logrotate periodically. However, it's more straightforward to use stdout and stderr directly, and leave rotation and retention policies to the kubelet.

Log Management Tools:

Using tools like Fluentd, Fluent Bit, or Promtail (from Loki), you can deploy them as DaemonSets or sidecars to collect logs from all pods across the cluster.

Example Fluentd DaemonSet configuration:

apiVersion: apps/v1kind: DaemonSetmetadata:  name: fluentd  namespace: kube-system  labels:    k8s-app: fluentd-loggingspec:  selector:    matchLabels:      name: fluentd  template:    metadata:      labels:        name: fluentd    spec:      containers:      - name: fluentd        image: fluent/fluentd:v1.17        env:        - name: FLUENTD_ARGS          value: "--no-supervisor -q"        resources:          limits:            memory: 200Mi            cpu: 100m          requests:            memory: 200Mi            cpu: 100m        volumeMounts:        - name: varlog          mountPath: /var/log        - name: varlibdockercontainers          mountPath: /var/lib/docker/containers          readOnly: true      terminationGracePeriodSeconds: 30      volumes:      - name: varlog        hostPath:          path: /var/log      - name: varlibdockercontainers        hostPath:          path: /var/lib/docker/containers

Centralized Logging

For large Kubernetes clusters, centralized logging is crucial. It involves collecting logs from all nodes, containers, and pods and storing them in a centralized logging system for easy access, search, and analysis. Common centralized logging solutions include:

EFK Stack: Elasticsearch, Fluentd, and Kibana.
ELK Stack: Elasticsearch, Logstash, and Kibana.
Loki and Grafana: Promtail for log collection, Loki for storage, and Grafana for visualization.
Third-Party Paid Services: Such as Datadog or Splunk.

]]>

Kubernetes Pod Anti-Affinity: A Beginner's Guide

Maxat Akbanov — Thu, 04 Jul 2024 14:04:56 GMT

Pod anti-affinity is a Kubernetes feature that allows you to specify that certain pods should not be scheduled on the same nodes as other specified pods. This is useful for ensuring high availability and resilience by spreading out workloads across different nodes.

💡

Pod anti-affinity is not only applied for nodes, but in case of using of cloud provider it can be applied for different availability zones and regions. This achieved by specifying the topologyKey value in the affinity specification.

Let's imagine a scenario where you have multiple applications running on your cluster in a microservice architecture, and your cluster consists of several nodes hosting your application pods. While the scheduler ensures that all your pods are placed on healthy nodes with adequate resources, it doesn't consider which specific applications are running in each pod when deciding on placement. This could lead to a situation where all the pods from a particular deployment are placed on the same node. The major drawback of this situation is that it exposes that application to the risk of failure if that node goes down. The Pod Anti-Affinity helps us to spread the pods evenly across the nodes in the cluster.

Difference between Node Affinity and Pod Affinity/Anti-Affinity

Node Affinity and Inter-Pod Affinity/Anti-Affinity are Kubernetes features that help control the placement of pods within a cluster. However, they serve different purposes and operate based on different criteria.

Node Affinity

Node Affinity is used to control which nodes a pod can be scheduled on based on node labels. It allows you to specify rules that must be satisfied for a pod to be scheduled onto a node.

Key Points:

Based on Node Labels: Node Affinity uses the labels assigned to nodes to determine where pods can be scheduled.
Types of Affinity:
- requiredDuringSchedulingIgnoredDuringExecution: The scheduler can't schedule the Pod unless the rule is met. This is a hard requirement.
- preferredDuringSchedulingIgnoredDuringExecution: The scheduler tries to find a node that meets the rule. If a matching node is not available, the scheduler still schedules the Pod.

💡

IgnoredDuringExecution means that if the node labels change after Kubernetes schedules the Pod, the Pod continues to run.

Example:

apiVersion: v1kind: Podmetadata:  name: with-node-affinityspec:  affinity:    nodeAffinity:      requiredDuringSchedulingIgnoredDuringExecution:        nodeSelectorTerms:        - matchExpressions:          - key: topology.kubernetes.io/zone            operator: In            values:            - antarctica-east1            - antarctica-west1      preferredDuringSchedulingIgnoredDuringExecution:      - weight: 1        preference:          matchExpressions:          - key: another-node-label-key            operator: In            values:            - another-node-label-value  containers:  - name: with-node-affinity    image: registry.k8s.io/pause:2.0

Inter-Pod Affinity/Anti-Affinity

Inter-Pod Affinity and Pod Anti-Affinity control the placement of pods relative to other pods based on their labels. These features allow you to specify rules for placing pods in relation to other pods.

Inter-pod affinity and anti-affinity require substantial amounts of processing which can slow down scheduling in large clusters significantly. It is not recommended using them in clusters larger than several hundred nodes.

Key Points:

Based on Pod Labels: Pod Affinity/Anti-Affinity uses labels assigned to pods to determine placement.
Types of Affinity:
- Inter-Pod Affinity: Ensures that pods are scheduled on the same node or close to other specified pods.
- Pod Anti-Affinity: Ensures that pods are not scheduled on the same node or close to other specified pods.
Topology Key: Specifies the failure domain within which the affinity or anti-affinity rule applies (e.g., kubernetes.io/hostname for nodes, topology.kubernetes.io/zone for zones in the cloud).

Pod anti-affinity requires nodes to be consistently labeled, in other words, every node in the cluster must have an appropriate label matching topologyKey. If some or all nodes are missing the specified topologyKey label, it can lead to unintended behavior.

Examples:

Inter-Pod Affinity:

apiVersion: v1kind: Podmetadata:  name: example-podspec:  affinity:    podAffinity:      requiredDuringSchedulingIgnoredDuringExecution:        - labelSelector:            matchExpressions:              - key: app                operator: In                values:                  - my-app          topologyKey: "kubernetes.io/hostname"

In this example, the pod will be scheduled on the same node as other pods with the label app=my-app.

Pod Anti-Affinity:

apiVersion: v1kind: Podmetadata:  name: example-podspec:  affinity:    podAntiAffinity:      requiredDuringSchedulingIgnoredDuringExecution:        - labelSelector:            matchExpressions:              - key: app                operator: In                values:                  - my-app          topologyKey: "kubernetes.io/hostname"

In this example, the pod will be scheduled on different nodes from pods with the label app=my-app.

Summary of Differences

Node Affinity:
- Focuses on the relationship between pods and nodes.
- Uses node labels.
- Controls which nodes a pod can be scheduled on.
Pod Affinity/Anti-Affinity:
- Focuses on the relationship between pods.
- Uses pod labels.
- Controls placement of pods relative to other pods.
- Can specify inter-affinity (co-location) or anti-affinity (separation).

Both Node Affinity and Pod Affinity/Anti-Affinity are powerful tools for managing pod placement and ensuring high availability, performance, and compliance with deployment policies in a Kubernetes cluster.

Use Cases for Pod Anti-Affinity

High Availability:
- Ensure that replicas of a critical application are not placed on the same node to prevent a single point of failure.
- Example: Deploying replicas of a web server across different nodes to ensure availability if one node goes down.
Resource Contention:
- Prevent pods from competing for the same resources by spreading them across different nodes.
- Example: Avoiding multiple CPU-intensive workloads on the same node to ensure they do not affect each other's performance.
Failure Isolation:
- Reduce the risk of correlated failures by ensuring that similar pods are not scheduled on the same node.
- Example: Distributing pods of a stateful application across different nodes to prevent losing data in case of a node failure.
Compliance and Security:
- Enforce organizational policies that require certain types of workloads to be isolated from each other.
- Example: Ensuring that test and production environments do not share the same nodes.

Example Scenario

Imagine you have a web application with multiple replicas (pods) and a database. You want to ensure that:

No two replicas of the web application are on the same node.
The database pod is not on the same node as any web application pod.

You would use pod anti-affinity rules to achieve this separation, thereby enhancing the resilience and performance of your application.

Workshop Exercise: Pod Anti-Affinity with Minikube

This exercise demonstrates how to set up and observe pod anti-affinity rules in a Kubernetes cluster using Minikube. By following these steps, you can see how Kubernetes schedules pods across different nodes to ensure high availability and fault tolerance.

Prerequisites:

Minikube installed on your local machine.
kubectl installed and configured to interact with Minikube.

Step 1: Start Minikube

Start a Minikube cluster with multiple nodes to see the anti-affinity rules in action.

minikube start --nodes 6

Verify that the nodes are up and running:

kubectl get nodes

Step 2: Create a Namespace

Create a new namespace for the exercise:

kubectl create namespace anti-affinity-demo

Step 3: Define a Deployment with Anti-Affinity

Create a deployment configuration file with pod anti-affinity rules. Save the following YAML content to a file named deployment.yaml.

apiVersion: apps/v1kind: Deploymentmetadata:  name: my-app  namespace: anti-affinity-demospec:  replicas: 3  selector:    matchLabels:      app: my-app  template:    metadata:      labels:        app: my-app    spec:      affinity:        podAntiAffinity:          requiredDuringSchedulingIgnoredDuringExecution:            - labelSelector:                matchExpressions:                  - key: app                    operator: In                    values:                      - my-app              topologyKey: "kubernetes.io/hostname"      containers:      - name: my-app-container        image: nginx        ports:        - containerPort: 80

Step 4: Apply the Deployment

Apply the deployment to your Minikube cluster:

kubectl apply -f deployment.yaml

Step 5: Verify Pod Distribution

Check the status of the pods and verify their node placement:

kubectl get pods -o wide -n anti-affinity-demo

You should see that the pods are distributed across different nodes, adhering to the anti-affinity rule specified.

Step 6: Inspect Node and Pod Details

Inspect the pods to verify the labels and affinity rules:

# Describe a podkubectl describe pod <pod-name> -n anti-affinity-demo# Get the yaml outputkubectl get pod <pod-name> -n anti-affinity-demo -o yaml

Replace <pod-name> with the name of one of your pods.

Step 7: Observe Anti-Affinity in Action

To observe the anti-affinity rules in action, try scaling the deployment up or down and see how Kubernetes schedules the pods.

# Scale up the deploymentkubectl scale deployment my-app --replicas=6 -n anti-affinity-demo# Verify pod distribution againkubectl get pods -o wide -n anti-affinity-demo

Scaling up should distribute the new pods across different nodes as much as possible, adhering to the anti-affinity rules.

Cleanup

After completing the exercise, clean up the resources:

# Delete the deploymentkubectl delete deployment my-app -n anti-affinity-demo# Delete the namespacekubectl delete namespace anti-affinity-demo# Stop Minikubeminikube stop

References

]]>

Understanding Ephemeral Storage in Kubernetes

Maxat Akbanov — Wed, 03 Jul 2024 09:18:39 GMT

Ephemeral storage in Kubernetes refers to the storage associated with a pod that exists only for the lifetime of that pod. It is typically used for temporary data that does not need to persist beyond the pod's lifecycle. Ephemeral storage is usually backed by the nodes local storage and is not designed for long-term data retention. It gets wiped clean once the pod is deleted or restarted.

Usage of Ephemeral Storage

Ephemeral storage in Kubernetes can be used in various scenarios, such as:

Scratch space for applications:
- Applications that require temporary space for intermediate data processing.
Caching:
- Temporary storage for cached data that can be regenerated if lost.
Logs:
- Storing log data that can be offloaded to persistent storage or log aggregation systems.
Temporary storage for build artifacts:
- During CI/CD processes, ephemeral storage can be used to store build artifacts that are needed only temporarily.

Types of Ephemeral Storage in Kubernetes

EmptyDir:
- A volume that is created when a pod is assigned to a node and exists as long as the pod is running. Empty at Pod startup, with storage coming locally from the kubelet base directory (usually the root disk /var/lib/kubelet) or RAM
ConfigMap and Secret:
- Used for storing configuration data and sensitive information that can be consumed by pods.
DownwardAPI:
- Allows pods to consume metadata about themselves or their environment.
CSI ephemeral volumes:
- Similar to the previous volume kinds, but provided by special CSI drivers which specifically support this feature.
Generic ephemeral volumes:
- Can be provided by all storage drivers that also support persistent volumes

emptyDir, configMap,downwardAPI, secret are provided as local ephemeral storage. They are managed by kubelet on each node.

Default Ephemeral Storage Type

When a Pod is started in Kubernetes, the default type of ephemeral storage used is the nodes local storage associated with the pod's lifecycle. Specifically, the primary forms of ephemeral storage that are used by default without any explicit configuration are:

Container Writable Layer

Each container in a pod has a writable layer provided by the container runtime (e.g., Docker, containerd). This writable layer is ephemeral and used by default for any file operations within the container that are not mapped to other volumes.

Automatic Ephemeral Storage Features

Container Writable Layer:
- Each container has its own writable layer where it can write files. This writable layer is ephemeral and lasts only for the duration of the container. Once the container is terminated or restarted, the data in this writable layer is lost.
EmptyDir (if specified):
- If an emptyDir volume is explicitly specified in the pod's configuration, it provides a dedicated ephemeral storage that is shared among all containers in the pod. This storage is tied to the pod's lifecycle and is created when the pod is scheduled and deleted when the pod is terminated.

💡

To get the list of all pods or deployments that are using emptyDir, use the following command: https://gist.github.com/Brain2life/173dbad51c112a2ff1b3f3add9312fb1

Here's an example of pod definition with emptyDir volume specified:

    apiVersion: v1    kind: Pod    metadata:      name: example-pod    spec:      containers:      - name: example-container        image: busybox        command: ["sh", "-c", "echo Hello Kubernetes! > /data/hello.txt && sleep 3600"]        volumeMounts:        - mountPath: /data          name: temp-storage      volumes:      - name: temp-storage        emptyDir: {}

Example

If you start a pod without specifying any volumes, the writable layer of each container acts as the ephemeral storage by default. Here is an example of a simple pod configuration without any explicit ephemeral storage:

apiVersion: v1kind: Podmetadata:  name: example-podspec:  containers:  - name: example-container    image: busybox    command: ["sh", "-c", "echo Hello Kubernetes! > /tmp/hello.txt && sleep 3600"]

In this example:

The container uses its writable layer to write the file /tmp/hello.txt.
This storage is ephemeral and will be lost if the container is terminated or restarted.

To see the writable layer of a container in Kubernetes, you typically need to access the container's filesystem. The writable layer is where all changes made to the filesystem of the container (such as creating or modifying files) are stored. Here's how you can inspect the writable layer:

Exec into the container:

kubectl exec -it example-pod -c example-container -- /bin/sh

Inspect filesystem. You should see hello.txt file:

cd /tmpls -l

By default, when a pod is started in Kubernetes, the containers writable layer is used as the ephemeral storage. If you need shared ephemeral storage among containers in the same pod, you can explicitly define an emptyDir volume. The writable layer is tied to the container lifecycle, while emptyDir (if used) is tied to the pod lifecycle.

Best Practices

Resource Requests and Limits

Set requests and limits for ephemeral storage to ensure that no single pod can consume all the node's storage. This helps in managing resource allocation and preventing resource exhaustion.

In the following example, the Pod has two containers. Each container has a request of 2GiB of local ephemeral storage. Each container has a limit of 4GiB of local ephemeral storage. Therefore, the Pod has a request of 4GiB of local ephemeral storage, and a limit of 8GiB of local ephemeral storage. 500Mi of that limit could be consumed by the emptyDir volume.

apiVersion: v1kind: Podmetadata:  name: frontendspec:  containers:  - name: app    image: images.my-company.example/app:v4    resources:      requests:        ephemeral-storage: "2Gi"      limits:        ephemeral-storage: "4Gi"    volumeMounts:    - name: ephemeral      mountPath: "/tmp"  - name: log-aggregator    image: images.my-company.example/log-aggregator:v6    resources:      requests:        ephemeral-storage: "2Gi"      limits:        ephemeral-storage: "4Gi"    volumeMounts:    - name: ephemeral      mountPath: "/tmp"  volumes:    - name: ephemeral      emptyDir:        sizeLimit: 500Mi

The sizeLimit field under emptyDir specifies the maximum size for the entire emptyDir volume. An emptyDir volume is a temporary directory that initially starts empty and is deleted when the Pod is removed.

In given example, the sizeLimit of 500Mi restricts the total size of the ephemeral volume that is mounted to /tmp in both containers.

Each container (app and log-aggregator) is allowed to request 2Gi and use up to 4Gi of ephemeral storage individually.

However, the shared ephemeral volume (emptyDir) has a sizeLimit of 500Mi, meaning the combined storage usage of both containers in /tmp cannot exceed 500Mi.

Log Management

Use log aggregation solutions (e.g., ELK Stack, Fluentd) to collect and store logs from pods to avoid losing critical log data when pods are deleted.

Monitoring and Alerts:

Implement monitoring for ephemeral storage usage. Tools like Prometheus and Grafana can be used to set up alerts for storage thresholds.

If the kubelet is managing local ephemeral storage as a resource, then the kubelet measures storage use in:

emptyDir volumes, except tmpfs emptyDir volumes
directories holding node-level logs
writeable container layers

If a Pod is using more ephemeral storage than you allow it to, the kubelet sets an eviction signal that triggers Pod eviction.

For container-level isolation, if a container's writable layer and log usage exceeds its storage limit, the kubelet marks the Pod for eviction.

For pod-level isolation the kubelet works out an overall Pod storage limit by summing the limits for the containers in that Pod. In this case, if the sum of the local ephemeral storage usage from all containers and also the Pod's emptyDir volumes exceeds the overall Pod storage limit, then the kubelet also marks the Pod for eviction.

By default ephemeral container data is located at:

/var/lib/kubelet
/var/lib/containers

on the Kubernetes Node.

To show the ephemeral storage usage on the node use:

df -h /var/lib

To show the ephemeral storage usage inside the container:

du -h .du -h [directory]

Data Backup

For critical temporary data, implement mechanisms to periodically back up data to persistent storage solutions.

Clean Up

Ensure that ephemeral storage is properly cleaned up when pods are terminated to avoid orphaned data and wasted storage space. Even terminated and failed pods take up the space. Therefore, it is important to clean up the cluster from hanging pods.

You can use the following Bash script in your CI/CD pipelines or clean your cluster manually by starting the script from your local machine: https://github.com/Brain2life/bash-cookbook/tree/k8s-cleanup-pods

Type of Issues with Ephemeral Storage

Data Loss:
- Since ephemeral storage is tied to the pod lifecycle, any data stored there will be lost if the pod is deleted or crashes.
Resource Contention:
- Without proper resource limits, pods might consume more storage than expected, leading to node resource contention and potential disruptions.
Node Disk Pressure:
- High usage of ephemeral storage can cause disk pressure on nodes, triggering eviction of pods or other resource management actions by the kubelet.
Limited Capacity:
- Nodes have finite storage capacity, and excessive usage of ephemeral storage can exhaust available space, affecting the overall cluster performance.
No Persistence:
- Ephemeral storage is not suitable for storing data that needs to be preserved across pod restarts or crashes. Applications requiring persistent storage should use Persistent Volumes (PVs) and Persistent Volume Claims (PVCs).

Understanding and effectively managing ephemeral storage in Kubernetes is crucial for ensuring the stability and performance of applications running in the cluster.

]]>

Get started with Gitleaks tool

Maxat Akbanov — Thu, 06 Jun 2024 13:49:34 GMT

Gitleaks is an open-source SAST tool designed to detect and prevent secrets (such as API keys, passwords, and other sensitive information) from being committed to version control repositories. It scans the entire repository history, including directories and files, to identify potential secrets. By configuring it properly and integrating it into your development workflow, you can significantly reduce the risk of exposing sensitive information.

Key Features

Detection:

Patterns and Rules: Gitleaks uses predefined patterns and rules to identify secrets. These patterns can include regular expressions for common types of secrets like AWS keys, Slack tokens, and more:

  [[rules]]  id = "aws-access-token"  description = "Identified a pattern that may indicate AWS credentials, risking unauthorized cloud resource access and data breaches on AWS platforms."  regex = '''(?:A3T[A-Z0-9]|AKIA|ASIA|ABIA|ACCA)[A-Z0-9]{16}'''  keywords = [      "akia","asia","abia","acca",  ]

Custom Rules: Users can define custom rules to detect specific types of secrets relevant to their projects.

Scanning:
- Repository Scanning: Gitleaks can scan entire Git repositories, including commit history, directories and files.
- Pre-commit Hook: It can be integrated as a pre-commit hook to scan changes before they are committed, preventing secrets from entering the repository.
Configuration:
- Config Files: Users can configure Gitleaks using a configuration file (gitleaks.toml) to customize the rules and behavior of the tool.
- Ignore Files: Gitleaks can be configured to ignore specific files or directories using an ignore file (.gitleaksignore).

Output:

Reports: Gitleaks generates detailed reports of identified secrets, including the location in the repository and the specific rule that triggered the detection.

  ~/code(master) gitleaks detect --source . -v  Finding:     "export BUNDLE_ENTERPRISE__CONTRIBSYS__COM=cafebabe:deadbeef",  Secret:      cafebabe:deadbeef  RuleID:      sidekiq-secret  Entropy:     2.609850  File:        cmd/generate/config/rules/sidekiq.go  Line:        23  Commit:      cd5226711335c68be1e720b318b7bc3135a30eb2  Author:      John  Email:       john@users.noreply.github.com  Date:        2022-08-03T12:31:40Z  Fingerprint: cd5226711335c68be1e720b318b7bc3135a30eb2:cmd/generate/config/rules/sidekiq.go:sidekiq-secret:23

Formats: Reports can be output in various formats such as JSON (default), CSV, junit, sarif or directly to the console.

Installation on Linux

Gitleaks can be installed as a binary from the GitHub releases page.

💡

Ensure that you have the Go installed properly on your system as a prerequisite.

For Go installation you can use the following settings in ~/.bashrc:

# Golang export GOPATH=$HOME/go export GOROOT=/usr/local/go PATH=$PATH:$GOROOT/bin/:$GOPATH/bin

To install the latest version of the gitleaks:

Compile the source code:

 # From Source git clone https://github.com/gitleaks/gitleaks.git cd gitleaks make build

After compiling the source code, you'll find the executable:

Now move the gitleaks executable into your /usr/local/bin/ :
```
 sudo mv gitleaks /usr/local/bin/
```
To verify the installation run:
```
 gitleaks version
```

Basic Usage

To scan a repository's git log history, navigate to the repository directory and run:

gitleaks detect .

The detect command is used to scan repos, directories, and files.

To scan files and directories while ignoring git log history use the --no-git option.
```
 gitleaks detect . --no-git
```

To save finding into report JSON (default) file:

 gitleaks detect --report-path gitleaks-report.json

To output current repository scan while ignoring the git log history:

 gitleaks detect . --no-git --report-format=json --report-path=gitleaks-report.json

If you need to process JSON report file for all found files with sensitive information (File field) use the following command:

jq -r '.[].File' gitleaks-report.json | sort | uniq > sensitive_files.txt

Gitleaks pre-commit

To use gitleaks as pre-commit hook, first install pre-commit tool:
```
 pip install pre-commit
```
Create .pre-commit-config.yaml file at the root of the project. Specify the configuration and the version of the gitleaks to use:
```
 # .pre-commit-config.yaml - repo: https://github.com/zricethezav/gitleaks   rev: v8.18.3   hooks:     - id: gitleaks
```
To install the hook run:
```
 pre-commit install
```
Optional. To update the gitleaks version used in the hook configuration to the latest:
```
 pre-commit autoupdate
```
Optional. To disable the gitleaks pre-commit hook you can prepend SKIP=gitleaks to the commit command and it will skip running gitleaks:
```
 SKIP=gitleaks git commit -m "skip gitleaks check"
```

Create custom configuration of rules

Gitleaks offers a configuration format you can follow to write your own secret detection rules. Create a gitleaks.toml file to define custom rules and settings:

title = "Gitleaks Configuration"[[rules]]description = "Generic API Key"regex = '''[a-zA-Z0-9]{32}'''tags = ["key", "api"]# Ignoring paths[[allowlist]]paths = ["path/to/ignore"]

For more information, see also Configuration section of gitleaks

]]>

Iterate over matching files: `with_fileglob` module in Ansible

Maxat Akbanov — Mon, 27 May 2024 08:19:15 GMT

The with_fileglob module in Ansible is used to iterate over files in a directory that match a specified pattern on the Ansible controller machine. This is useful when you need to perform tasks on multiple files without explicitly listing each one. The with_fileglob loop allows you to apply actions to all files that match a given globbing pattern, making your playbooks more dynamic and easier to maintain.

with_fileglob module use Pythons glob library to find all pathnames.

Use Cases

Deploying Configuration Files: You can use with_fileglob to copy multiple configuration files from a directory to a target host.
Template Rendering: Render multiple template files from a directory.
File Permissions: Change the permissions of multiple files in a directory.
Backup: Back up multiple files from a directory to another location.

Code Examples

Copying Configuration Files:

 - name: Copy all config files   hosts: webservers   tasks:     - name: Copy configuration files       copy:         src: "{{ item }}"         dest: /etc/myapp/config/       with_fileglob:         - /path/to/local/configs/*.conf

Rendering Templates:

 - name: Render multiple template files   hosts: appservers   tasks:     - name: Template files       template:         src: "{{ item }}"         dest: "/etc/myapp/templates/{{ item | basename }}"       with_fileglob:         - /path/to/templates/*.j2

Changing File Permissions:

 - name: Change permissions of log files   hosts: servers   tasks:     - name: Set permissions       file:         path: "{{ item }}"         mode: '0644'       with_fileglob:         - /var/log/myapp/*.log

Backing Up Files:

 - name: Backup important files   hosts: all   tasks:     - name: Copy files to backup directory       copy:         src: "{{ item }}"         dest: /backup/       with_fileglob:         - /etc/myapp/important_files/*

Explanation

with_fileglob: This is the loop directive that tells Ansible to iterate over files matching the given pattern.
src: Specifies the source file or directory.
dest: Specifies the destination file or directory.
item: The current item in the loop, representing each file that matches the pattern.

By using with_fileglob, you can simplify tasks that need to be performed on multiple files without hardcoding each file path, making your Ansible playbooks more efficient and easier to manage.

How with_fileglob works

The with_fileglob module in Ansible is used to match files on the Ansible control machine, not on the remote hosts. This is an important distinction because the file matching is performed locally on the machine where the playbook is being run, and then the actions specified in the tasks are applied to the matched files.

Key Points

Local to Control Machine: The file globbing operation (with_fileglob) is executed on the control machine where Ansible is running.
Remote Tasks: Once the files are matched, the specified tasks (such as copying files, changing permissions, etc.) are executed on the remote hosts.

Example Clarification

Let's clarify with an example:

Control Machine Setup:

Assume you have a directory /path/to/local/configs/ on your control machine with several .conf files.

Playbook:

- name: Copy all config files  hosts: webservers  tasks:    - name: Copy configuration files      copy:        src: "{{ item }}"        dest: /etc/myapp/config/      with_fileglob:        - /path/to/local/configs/*.conf

Execution Flow

Globbing on Control Machine: Ansible matches all files ending with .conf in the /path/to/local/configs/ directory on the control machine.
Copy Task: The copy module is then used to copy each matched file from the control machine to the specified destination (/etc/myapp/config/) on the remote hosts (e.g., webservers).

Workshop: Copy config files from one local directory to another

This example will demonstrate using with_fileglob to copy multiple configuration files from a local directory to another directory on the same machine.

Step 1: Prepare the Environment

Open a terminal and run the following commands:

mkdir -p ~/ansible_test/configsmkdir -p ~/ansible_test/target# Create some sample config filesecho "config1" > ~/ansible_test/configs/sample1.confecho "config2" > ~/ansible_test/configs/sample2.confecho "config3" > ~/ansible_test/configs/sample3.conf

Step 2: Create the Ansible Playbook

Create a file named copy_configs.yml and add the following content:

---- name: Copy configuration files using with_fileglob  hosts: localhost  connection: local  tasks:    - name: Copy configuration files to target directory      copy:        src: "{{ item }}"        dest: ~/ansible_test/target/      with_fileglob:        - "~/ansible_test/configs/*.conf"

Step 3: Run the Playbook

Ensure you have Ansible installed. If not, you can install it using:

sudo apt updatesudo apt install ansible -y

Run the playbook with the following command:

ansible-playbook copy_configs.yml

Step 4: Verify the Result

Check the contents of the target directory to ensure the files were copied correctly:

ls ~/ansible_test/target

You should see sample1.conf, sample2.conf, and sample3.conf in the ~/ansible_test/target directory.

Explanation

hosts: localhost: Specifies that the playbook will run on the local machine.
connection: local: Ensures Ansible runs the tasks locally.
with_fileglob: Matches all .conf files in the specified directory.
copy module: Copies each matched file from the source directory to the target directory.

References:

Ansible docs: ansible.builtin.fileglob lookup list files matching a pattern

]]>

Quick Startup with Ansible

Maxat Akbanov — Sun, 26 May 2024 14:38:44 GMT

Ansible is a powerful automation tool used for configuration management, application deployment, and task automation. It operates by connecting to your nodes and pushing out small programs, called Ansible modules, to perform tasks.

Image credits: Ansible.com

The main components of Ansible are:

Inventory: A list of hosts or nodes that Ansible manages.
Modules: Units of work that Ansible executes. Examples include tasks for installing packages, copying files, or restarting services.
Playbooks: YAML files that define a series of tasks to be executed on a set of hosts.
Roles: A way to group multiple tasks and configurations to be reusable.
Variables: Used to store values that can be reused in playbooks and templates.
Templates: Files that have placeholders for variables and can be dynamically populated during execution.
Handlers: Used to trigger actions at the end of a playbook run when certain conditions are met.

Ansible Components Overview

1. Inventory

Inventory File:

The default inventory file is usually named hosts or inventory.
By default, it is located at /etc/ansible/hosts.
It can be a simple INI-style file or YAML file, listing the managed nodes.

INI File:

An .ini file is a simple text file used for configuration, with sections, properties, and values.
In the context of Ansible, it defines groups of hosts and their variables.

Example (INI-style):

# Default inventory file location: /etc/ansible/hosts[webservers]web1.example.comweb2.example.com[databases]db1.example.com# Defining variables for groups[webservers:vars]ansible_user=adminansible_ssh_private_key_file=/path/to/key

Example (YAML-style):

all:  children:    webservers:      hosts:        web1.example.com:        web2.example.com:      vars:        ansible_user: admin        ansible_ssh_private_key_file: /path/to/key    databases:      hosts:        db1.example.com:

2. Modules

Modules:

Small programs that Ansible pushes out to nodes to perform tasks. After programs finish the execution, Ansible removes them.
Examples include modules for package management (apt, yum), file operations (copy, template), and service management (service).

Example:

- name: Install Nginx  apt:    name: nginx    state: present

Command to list available modules:

ansible-doc -l

3. Playbooks

Playbooks:

YAML files that define a series of tasks to be executed on hosts.
They contain plays, each mapping a group of hosts to roles and tasks.

Important Parts:

hosts: Specifies which hosts the playbook should run against.
tasks: Defines a list of tasks to be executed.
vars: Defines variables to be used within the playbook.
handlers: Defines handlers that can be notified by tasks.

Example:

---- name: Configure web servers  hosts: webservers  vars:    nginx_version: 1.18.0  tasks:    - name: Install Nginx      apt:        name: nginx={{ nginx_version }}        state: present    - name: Copy Nginx config      template:        src: templates/nginx.conf.j2        dest: /etc/nginx/nginx.conf      notify: Restart Nginx  handlers:    - name: Restart Nginx      service:        name: nginx        state: restarted

4. Roles

Roles:

Provide a structured way to group tasks, handlers, files, templates, and variables.
Allow for reuse and sharing of Ansible content.

Role Directory Structure:

roles/  webserver/    tasks/      main.yml    templates/      nginx.conf.j2    handlers/      main.yml    vars/      main.yml    defaults/      main.yml    files/    meta/      main.yml

Example Task File (roles/webserver/tasks/main.yml):

---- name: Install Nginx  apt:    name: nginx    state: present

The beauty of roles lies in their reusability. You can include the same role in multiple playbooks to automate the same configuration on different sets of servers. For example, like installing and configuring a web server (Apache or Nginx) or setting up a database (MySQL or PostgreSQL).

To include a role inside an Ansible playbook you can use include_role module.

This is the recommended approach for Ansible versions 2.4 and later. It offers more control and clarity.

- ansible.builtin.include_role:    name: myrole- name: Run tasks/other.yaml instead of 'main'  ansible.builtin.include_role:    name: myrole    tasks_from: other- name: Pass variables to role  ansible.builtin.include_role:    name: myrole  vars:    rolevar1: value from task- name: Use role in loop  ansible.builtin.include_role:    name: '{{ roleinputvar }}'  loop:    - '{{ roleinput1 }}'    - '{{ roleinput2 }}'  loop_control:    loop_var: roleinputvar- name: Conditional role  ansible.builtin.include_role:    name: myrole  when: not idontwanttorun- name: Apply tags to tasks within included file  ansible.builtin.include_role:    name: install    apply:      tags:        - install  tags:    - always

5. Variables

Variables:

Used to store values that can be reused in playbooks, roles, and templates.
Can be defined in playbooks, inventories, roles, or passed via command line.

Variable Precedence (highest to lowest):

Extra vars (command line -e)
Task vars (only in scope for the task)
Block vars (only in scope for the block)
Role and include vars
Play vars (for the entire play)
Inventory vars
Facts (gathered or set via set_fact)
Playbook group_vars
Playbook host_vars
Ansible configuration settings

To learn more about variables check out this article: How to Use Different Types of Ansible Variables (Examples)

1. Extra vars (command line `-e`)

Description:

Extra variables passed directly on the command line have the highest precedence.

Example:

ansible-playbook playbook.yml -e "var1=value1 var2=value2"

Playbook (playbook.yml):

---- hosts: all  tasks:    - debug:        msg: "var1={{ var1 }}, var2={{ var2 }}"

2. Task vars (only in scope for the task)

Description:

Variables defined within a specific task have a very high precedence.

Example:

---- hosts: all  tasks:    - name: Show task variable      debug:        msg: "Task variable var1={{ var1 }}"      vars:        var1: "task_value"

3. Block vars (only in scope for the block)

Description:

Variables defined within a block are in scope only for tasks within that block.

Example:

---- hosts: all  tasks:    - block:        - name: Show block variable          debug:            msg: "Block variable var1={{ var1 }}"      vars:        var1: "block_value"

To learn more, check out "Blocks"

4. Role and include vars

Description:

Variables defined within roles or included files.

Example:

# roles/webserver/vars/main.yml---var1: "role_value"# playbook.yml---- hosts: all  roles:    - role: webserver  tasks:    - debug:        msg: "Role variable var1={{ var1 }}"

5. Play vars (for the entire play)

Description:

Variables defined within a play are in scope for the entire play.

Example:

---- hosts: all  vars:    var1: "play_value"  tasks:    - name: Show play variable      debug:        msg: "Play variable var1={{ var1 }}"

6. Inventory vars

Description:

Variables defined in the inventory file.

Example (INI-style inventory):

# inventory[all:vars]var1=inventory_value

Playbook:

---- hosts: all  tasks:    - name: Show inventory variable      debug:        msg: "Inventory variable var1={{ var1 }}"

7. Facts (gathered or set via `set_fact`)

Description:

Facts are gathered by Ansible or set during playbook execution using set_fact.

Example:

---- hosts: all  tasks:    - name: Set fact variable      set_fact:        var1: "fact_value"    - name: Show fact variable      debug:        msg: "Fact variable var1={{ var1 }}"

8. Playbook group_vars

Description:

Variables defined for groups in the group_vars directory within the playbook directory.

Example (group_vars/all.yml):

# group_vars/all.yml---var1: "group_vars_value"

Playbook:

---- hosts: all  tasks:    - name: Show group_vars variable      debug:        msg: "Group_vars variable var1={{ var1 }}"

9. Playbook host_vars

Description:

Variables defined for individual hosts in the host_vars directory within the playbook directory.

Example (host_vars/hostname.yml):

# host_vars/hostname.yml---var1: "host_vars_value"

Playbook:

---- hosts: hostname  tasks:    - name: Show host_vars variable      debug:        msg: "Host_vars variable var1={{ var1 }}"

10. Ansible configuration settings

Description:

Variables set in the Ansible configuration files (e.g., ansible.cfg).

Example (ansible.cfg):

# ansible.cfg[defaults]var1 = config_value

Playbook:

---- hosts: all  tasks:    - name: Show config variable      debug:        msg: "Config variable var1={{ var1 }}"

6. Templates

Templates:

Jinja2 templates are used to create dynamic configuration files.
Variables can be embedded and will be replaced during execution.

Example Template (templates/nginx.conf.j2):

server {    listen 80;    server_name {{ server_name }};    location / {        proxy_pass http://{{ backend }};    }}

Example Task Using Template:

- name: Copy Nginx config  template:    src: templates/nginx.conf.j2    dest: /etc/nginx/nginx.conf

7. Handlers

Handlers:

Special tasks that are triggered by other tasks using the notify directive.
Run once at the end of the playbook execution if notified.

Example:

- name: Install Nginx  apt:    name: nginx    state: present  notify: Restart Nginxhandlers:  - name: Restart Nginx    service:      name: nginx      state: restarted

Understanding these components allows for efficient and effective use of Ansible for automation tasks. Each component plays a crucial role in defining the structure and functionality of your automation scripts.

References:

]]>

Kubernetes Troubleshooting: Disk Pressure Taint

Maxat Akbanov — Mon, 20 May 2024 05:30:44 GMT

In Kubernetes, a "taint" is a way to mark a node with a specific attribute that repels pods from being scheduled on it, unless those pods have a matching "toleration" for the taint. The term "disk pressure taint" refers to a specific condition where a node is experiencing low disk space. This taint is automatically added by Kubernetes to nodes that are nearing their storage capacity limits to prevent new pods from being scheduled on them, which helps in maintaining the node's performance and stability by avoiding additional strain on its disk resources.

This mechanism ensures that pods are scheduled on nodes with sufficient disk space, maintaining overall cluster health and performance. When disk space is freed or increased, the disk pressure condition may be resolved, and the taint can be removed, allowing pods to be scheduled on the node again.

Reasons

The disk pressure warning in Kubernetes signals that a node is running low on disk space or inode capacity. This condition can arise due to various factors, often related to resource usage and management within the cluster. Here are some common reasons for a disk pressure warning:

High Container Log Volume: Containers can generate substantial log output, especially if verbose logging is enabled. If not managed properly, these logs can fill up the nodes disk space.
Unused or Dangling Docker Images: Over time, Docker images and containers that are no longer in use can accumulate, especially if images are frequently updated. These unused images take up significant disk space.
Persistent Volume Claims (PVCs): Persistent volumes associated with PVCs can consume a large amount of disk space, particularly if many stateful applications are running within the cluster.
Evicted Pods: When pods are evicted due to resource constraints, their data might still reside on the node, consuming space. This is especially relevant if the pods were using ephemeral volumes or if the logs of these evicted pods are not cleaned up.
Node-Level Application Data: Applications might write data directly to the nodes filesystem, outside of what Kubernetes manages, which can accumulate over time.
Temporary Files: Applications or systems processes may create temporary files that arent cleaned up properly, leading to gradual space consumption.
System Logs and Backups: System operations, diagnostics, backups, and logs can take up significant space. These often grow over time unless log rotation and cleanup policies are in place.
Improper Resource Quotas: If resource quotas and limits arent set properly, a single namespace or application can end up using an undue portion of disk resources, leading to disk pressure.

To manage and mitigate these issues, administrators should implement monitoring and alerting for disk usage, enforce appropriate resource quotas, manage log lifecycles, and periodically clean up unused images and volumes. This proactive management helps maintain node health and prevents disruptions due to disk pressure.

Collecting Information About Disk Pressure

Kubernetes monitors and manages the state of nodes through a component called the kubelet, which runs on each node in the cluster. The kubelet is responsible for a variety of node management tasks, including monitoring the health and resource status of the node it runs on.

Image credits: kubesimplify

Heres how kubelet specifically handles disk space and detects disk pressure:

Resource Metrics: The kubelet collects metrics about the node's resources, including CPU, memory, and disk utilization. For disk resources, it monitors the filesystem that houses the node's Docker images and container writable layers, as well as the filesystem used for the nodes main processes.
Thresholds and Conditions: Administrators can set thresholds for what constitutes disk pressure in terms of available disk space and inode counts. These thresholds are part of the kubelet's configuration. When the disk usage exceeds these thresholds, the kubelet identifies this as a disk pressure condition.
Node Status: If the kubelet detects that disk usage exceeds the configured thresholds, it updates the node's status to reflect that it is under disk pressure. This status update includes adding a taint to the node that prevents new pods from being scheduled unless they tolerate this taint.
Reporting: The kubelet regularly reports the node status, including any resource pressure conditions, back to the Kubernetes API server. This information is then used by the scheduler and other control plane components to make decisions about pod placement and cluster management.
Automatic Eviction: In some cases, if the disk pressure persists, the kubelet may start to evict pods to free up disk space, prioritizing the eviction based on factors such as the quality of service (QoS) class of the pods.

This proactive monitoring and management help maintain the stability and performance of the Kubernetes cluster by ensuring nodes operate within their resource limits.

Get Information about Disk Pressure

To find out which node in a Kubernetes cluster is affected by disk pressure, you can use the kubectl command-line tool to inspect the nodes' status. Here's how you can identify nodes with disk pressure:

List Node Conditions: You can list all nodes along with their conditions by running:
```
 kubectl get nodes -o wide
```
However, for more detailed information about the conditions affecting each node, including disk pressure, you can use:
```
 kubectl get nodes -o jsonpath="{.items[*].status.conditions}"
```
This command displays all conditions for each node. Look for conditions where the type is DiskPressure and the status is True.
Specifically Check for Disk Pressure: To filter and show only nodes under disk pressure, you can use a more refined command:
```
 kubectl get nodes -o jsonpath="{.items[?(@.status.conditions[?(@.type=='DiskPressure' && @.status=='True')])].metadata.name}"
```
This command will output the names of nodes where the disk pressure condition is currently True.
Describe Nodes: For a more verbose and comprehensive look at a particular nodes status, including events and conditions like disk pressure, you can use the describe command:
```
 kubectl describe node <node-name>
```
Replace <node-name> with the name of the node you want to inspect. The output will include all conditions, events, and other details specific to that node, helping you understand why disk pressure is being reported.

These commands help you identify and diagnose nodes experiencing disk pressure, allowing you to take appropriate actions such as cleaning up disk space or adjusting resource allocations.

Clean-up Disk Space on the Node

Cleaning up disk space on a Kubernetes node involves several strategies, depending on what's consuming the space. Here are some common approaches:

Prune Unused Docker Images and Containers: Over time, Docker can accumulate unused images and containers, which can consume considerable disk space. Use the following Docker commands to clean them up:
```
 docker system prune -a
```
This command removes all unused containers, networks, images (both dangling and unreferenced), and optionally, volumes.

It's a good practice to ensure that you do not need the data in these containers or images before running this command.

Clear Kubernetes Evicted Pods: Sometimes, when a node is under resource pressure, Kubernetes may evict pods but their metadata might still remain. You can clear evicted pods with:

 kubectl get pods --all-namespaces -o json | jq '.items[] | select(.status.phase == "Failed") | "kubectl delete pods \(.metadata.name) --namespace \(.metadata.namespace)"' | xargs -n 1 bash -c

Clean Up Node Filesystem:
- Logs: System and application logs can consume significant disk space. You might want to clean or rotate logs managed under /var/log/, or set up a log rotation policy if not already in place.
- Temporary Files: The /tmp directory often contains temporary files which might not be necessary. Consider deleting older or unnecessary files.
- Orphaned Volumes: Check for any orphaned volumes in /var/lib/kubelet/pods/ and remove them if they are no longer linked to existing pods.
Manage Persistent Volume Claims (PVCs): Review persistent volume claims and their associated volumes. Sometimes, unused or old PVCs can be deleted to free up space. Use:
```
 kubectl get pvc --all-namespaces
```
to list all PVCs. Review and delete any that are unnecessary.
Use a Resource Quota: To avoid future disk space issues, consider setting resource quotas and limits for namespaces to better manage the space used by each project or team.
Node Monitoring and Alerts: Implement monitoring and alerting for disk usage to proactively manage disk space and prevent nodes from hitting disk pressure.

Before performing these cleanup tasks, especially in a production environment, ensure you have backups as needed and that the deletions will not affect running applications. It's also wise to test these actions in a staging environment before executing them in production.

How to Check the Log File Sizes for Docker and Podman on Nodes

When investigating disk pressure in a Kubernetes cluster and you suspect that logs from container runtimes like Docker and Podman are consuming a significant amount of disk space, you can check the log sizes directly on the nodes. Heres how to do it for both Docker and Podman:

For Docker

Docker typically stores its logs in the container's directory under /var/lib/docker/containers/. Each container has its own directory here, containing log files. To find out the size of these logs:

SSH into the Node: Log into the node where Docker is running.
Run a Command to Summarize Log Sizes: Use the following command to find the total size of all log files:
```
 sudo du -sh /var/lib/docker/containers/*/*-json.log
```
This command sums up the sizes of all Docker log files. It can be helpful to run this on each node to determine where the largest log files are.

For Podman

Podman stores its logs under /var/lib/containers/storage/overlay-containers/ in each container's directory. Similar to Docker, you can inspect the sizes of these logs:

SSH into the Node: Access the node where Podman containers are running.
Run a Command to Summarize Log Sizes: Execute the following to find out how much space the log files are using:
```
 sudo du -sh /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log
```
This will display the sizes of all Podman container log files.

References:

]]>

Get started with Taints and Tolerations in Kubernetes

Maxat Akbanov — Mon, 20 May 2024 02:10:18 GMT

In Kubernetes, taints and tolerations are two distinct concepts used together to control how pods are scheduled on nodes. They help ensure that pods are only scheduled on appropriate nodes.

Taints (Marking Nodes)

A taint is applied to a node to mark it as repelling pods, unless those pods explicitly tolerate the taint. Taints have three properties:

Key: The name of the taint.
Value: The value associated with the taint, providing additional specificity.
Effect: Describes what happens to pods that do not tolerate the taint. The effects can be:
- NoSchedule: Pods that do not tolerate this taint are not scheduled on the node. Pods currently running on the node are not evicted.
- PreferNoSchedule: Kubernetes will try to avoid placing a pod on the node but it's not guaranteed.
- NoExecute: Pods that do not tolerate this taint are evicted if they are already running on the node, and new pods will not be scheduled on the node.

Tolerations (Specifying Pod Compatibility)

A toleration is applied to a pod and allows (but does not require) the pod to be scheduled on a node with matching taints. Tolerations specify which taints they can accept. They consist of:

Key
Value
Effect
Operator: Determines how to match the taint's key and value. Commonly used operators include Equal (default) and Exists.
TolerationSeconds: Only relevant with the NoExecute effect, this specifies how long a pod should stay on the node after the taint appears.

A toleration "matches" a taint if the keys are the same and the effects are the same, and:

the operator is Exists (in which case no value should be specified), or
the operator is Equal and the values should be equal.

The default value for operator is Equal.

Differences

Where they apply: Taints are applied to nodes, while tolerations are applied to pods.
Purpose: Taints make a node repulsive to certain pods, while tolerations make a pod capable of resisting certain taints, thereby allowing it to be scheduled on or remain on a node with those taints.

How They Are Used Together

The combination of taints and tolerations works as a mechanism to ensure that pods are scheduled on suitable nodes. For example, you might have a set of nodes with special hardware like GPUs. You can taint these nodes to repel ordinary workloads and add tolerations to GPU-enabled pods so they can still be scheduled there.

Image credits: kubecost

Another example would be to use taints and tolerations to utilize spot nodes in AWS. If a workload is qualified to run on a spot node, the ideal way to configure Kubernetes to utilize these spot nodes is to add taints and tolerations so these workloads will tolerate the tainted spot instances.

These mechanisms can be particularly useful in multi-tenant environments, where certain nodes may be reserved for specific teams or types of workloads, helping in maintaining the desired level of security and performance isolation.

Example of Tainting a Node

First, you can taint a node using the kubectl command line tool. This command taints a node to prevent any pods from being scheduled on it unless they explicitly tolerate the taint:

kubectl taint nodes node1 key1=value1:NoSchedule

This command applies a taint with:

key = key1
value = value1
effect = NoSchedule

Example Pod Specification with Tolerations

Below is a YAML example of a pod specification that includes a toleration. This pod will tolerate the taint we added to node1:

apiVersion: v1kind: Podmetadata:  name: mypodspec:  containers:  - name: mycontainer    image: nginx  tolerations:  - key: "key1"    operator: "Equal"    value: "value1"    effect: "NoSchedule"

This configuration means the pod mypod can be scheduled on the node node1 despite its taint.

More Complex Taint and Toleration Scenario

If a node has a taint that should evict existing pods and prevent new ones from being scheduled unless they tolerate the taint, you can do the following:

Tainting the Node

kubectl taint nodes node2 key2=value2:NoExecute

This applies a taint that has the NoExecute effect, which will evict pods that do not tolerate this taint.

Pod with a Toleration Including `TolerationSeconds`

apiVersion: v1kind: Podmetadata:  name: mypod2spec:  containers:  - name: mycontainer2    image: nginx  tolerations:  - key: "key2"    operator: "Equal"    value: "value2"    effect: "NoExecute"    tolerationSeconds: 3600

This pod tolerates the NoExecute taint on node2 and will not be evicted for at least 3600 seconds (1 hour) if the taint is applied after the pod is already running on the node. This gives time to handle the pod appropriately, such as gracefully shutting down or relocating its workload.

Best practices

When using taints and tolerations in Kubernetes, its important to follow best practices to ensure efficient, secure, and reliable cluster operations. Here are some key best practices:

1. Use Specific Taints for Specialized Hardware

Taints are highly effective for nodes with specialized hardware such as GPUs or high-performance SSDs. Use taints to ensure that only workloads requiring such resources are scheduled on those nodes. This optimizes resource utilization and prevents general workloads from consuming resources that are expensive or scarce.

2. Minimize Use of `NoExecute` for Stability

The NoExecute taint effect can cause pods to be evicted from a node, which might lead to service disruption. Use this effect sparingly and mainly when its crucial to remove pods from a node due to compliance, security, or hardware decommissioning reasons. Always define tolerationSeconds to manage the eviction timing gracefully.

3. Balance Tolerance and Restriction

While taints can restrict pods from scheduling on certain nodes, excessive use of taints can lead to inefficient scheduling and resource fragmentation. Use taints judiciously and ensure there are enough tolerations in place to maintain a balanced and efficient workload distribution.

4. Integrate with Affinity/Anti-affinity

Taints and tolerations should be used in conjunction with pod affinity and anti-affinity specifications to fine-tune pod placement. For instance, while taints can prevent certain pods from running on a node, affinity rules can be used to co-locate pods that benefit from running in proximity to each other (e.g., for performance reasons).

5. Document and Manage Taints and Tolerations

As the number of taints and tolerations grows, its crucial to document why and where they are used. This avoids configuration errors and helps new team members understand the setup. Consider using infrastructure as code (IaC) tools to manage and version control taint and toleration configurations.

6. Security and Multi-tenancy

In multi-tenant clusters, use taints to isolate nodes dedicated to different tenants, ensuring that workloads from one tenant cannot be accidentally or maliciously scheduled on anothers nodes. This is particularly useful for enhancing security and compliance in shared environments.

7. Testing and Validation

Regularly test your taints and tolerations setup in a staging environment to ensure that they behave as expected, especially after modifications or updates to your cluster configuration. This can help avoid unexpected scheduling behavior in production.

8. Monitoring and Alerts

Set up monitoring and alerts for the scheduling process. If pods are unschedulable due to taints or if certain nodes are underutilized, you should have visibility into these events to make timely adjustments.

References:

]]>

Python 101: Special variables

Maxat Akbanov — Tue, 16 Apr 2024 12:07:08 GMT

Python uses several special variables, often referred to as "dunder" (double underscore) variables or "magic methods" in the context of OOP, that have specific meanings and are used by the Python interpreter for particular purposes. This double underscore naming distinguishes them from regular variables and avoid naming conflicts with your custom variables.

These variables are integral to Python's internal operations and often relate to object-oriented features and module management.

Here are some of the key special variables in Python:

__name__: This variable is perhaps the most well-known. It holds the name of the module in which it is used. If the module is the entry point to the program (i.e., the script that is run), __name__ is set to "__main__". This helps determine whether the module is being run directly or being imported.
```
 # example.py if __name__ == "__main__":     print("This script is running directly") else:     print("This script has been imported")
```
Run this file directly or import it in another Python script to see the difference.
__doc__: This variable contains the module's (script's) or function's documentation string (docstring), which is the first string after the definition. It's used to provide documentation about the module, class, method, or function.
```
 def example_function():     """     This is a docstring for example_function.     """     pass print(example_function.__doc__)
```
__file__: This variable contains the pathname of the file from which the module was loaded, if it was loaded from a file. This can be useful for modules that want to load data files located in the same directory as the module file.
```
 # Prints the pathname of the file from which the module was loaded print(__file__)
```
This will output the full path of the current script file.
__dict__: This dictionary contains the namespace supported by most objects in Python, including modules, classes, and instances. It maps attribute names to their values and can be used for introspection or dynamically manipulating objects.
```
 class Example:     def __init__(self):         self.attribute = "value" instance = Example() print(instance.__dict__)
```
This will print the dictionary containing the namespace of the instance of Example.
__package__: This variable is used to support relative imports. It holds the name of the package that the current module is a part of. If the module is not part of a package, __package__ is set to None.
```
 # example_module.py print(__package__)
```
When part of a package, this prints the package name. If not part of a package, it prints None.
__path__: This variable only exists on packages. It is used to define the list of paths where Python searches for sub-modules and sub-packages under the package. This is how Python determines what can be imported within a package.
__class__: Within a class method, this variable refers to the class object that the method is defined on. This can be useful for accessing class-level attributes and methods from instance methods.
```
 class MyClass:     def show_class(self):         print(self.__class__) obj = MyClass() obj.show_class()
```
This code will print the class MyClass of the object obj.
__bases__: Available only on class objects, this tuple contains the base classes, in the order of their occurrence in the base class list, of the class object.
```
 class BaseClass:     pass class DerivedClass(BaseClass):     pass print(DerivedClass.__bases__)
```
This will print the tuple of base classes of DerivedClass, here (BaseClass,).
__annotations__: This dictionary contains type hints for the module, class, or function. It maps parameter names to their type hints, and is available from Python 3.6 onwards.
```
 def func(a: int, b: str) -> float:     return 3.14 print(func.__annotations__)
```
This will print the type hints of the parameters and the return type of func.
__all__: This list controls what is exported when the module is imported using from module import *. Only names included in __all__ will be imported.
```
# mymodule.py__all__ = ['public_function']def public_function():    print("Accessible function")def _private_function():    print("Hidden function")
```
When you use from mymodule import *, only public_function will be imported due to __all__.

]]>