cookies and containers

Google Foobar Challenge

2023-03-12T00:00:00+00:00

A few weeks ago I was googling random things (more specifically how to rename a remote branch as my naming had impacted a colleagues’ local environment), when I got a pop-up.

A few years ago Google created this secret challenge as a hiring tool, I just didn’t know it was still around! The URL is https://foobar.withgoogle.com/, but without an invite you can’t access the content. It no longer seems to be used for hiring, but it’s still a fun challenge, especially if you enjoy sites like adventofcode or Project Euler.

How does it work?

After getting an invite pop-up (or a code from a friend), you get the option to request a challenge. There are five levels to complete and each level has a number of challenges.

Your overall goal is to defeat the evil Commander Lambda who is trying to destroy Bunny Planet. Each challenge helps you infiltrate the organization the Commander leads and rise in the ranks until you’ve defeated a doomsday device and saved all the bunnies. Nobody said the backstories for code challenges need to make sense.

The setup

Each level has up to three challenges, each of which comes with a time limit. There’s plenty of time for each challenge. Most of the ones I had allowed for multiple days to solve it.

Once you request a challenge you get an extra folder with a text file that introduces the challenge problem.

You’re restricted to using Python or Java. There are two stubs for each challenge, Solution.java and solution.py, which you can edit, run against a number of test cases and finally submit.

The challenges

Even though the challenges work based on some backstory in the overall “help destroy the evil corp”, there are underlying algorithmic concepts.

Knowing what these are and how they work is very helpful for solving the challenges. Some I came across needed dynamic programming, graphs (shortest path, negative cycles, DFS), cellular automata.

Sometime during the process Foobar does ask for your details, but you don’t have to submit them, or can choose to do so later. Keep in mind that (according to other blogs) the challenge hasn’t been used for hiring since 2020.

Success!

Finishing all challenges gives you an encrypted message. Decrypting that returns

b"{'success' : 'great', 'colleague' : 'esteemed', 'efforts' : 'incredible', 'achievement' : 'unlocked', 'rabbits' : 'safe', 'foo' : 'win!'}"

It’s definitely a fun one, and I learned a few new concepts in the process!

Unfortunately it’s quite easy to come across people posting solutions when researching the topics, so if you don’t want to see any solutions during your first attempt keep an eye out for spoilers on StackOverflow.

Useful kubectl commands

2023-01-14T00:00:00+00:00

Sometimes I find myself going through old messages, command line history or slack search to find that one random command for debugging or fixing something. This is the very random collection of commands like that, as well as some that I just think are neat and should be better known.

Finding all resources in a namespace

Namespace stuck in terminating? Sometimes not all the resources get cleaned up properly when deleting a namespace and you might end up with namespace that just won’t go away. This command get EVERYTHING in that namespace (contrary to kubectl get all which is limited to a subset of Kubernetes resources).

NAMESPACE=default # <- stuck namespace
kubectl api-resources --verbs=list --namespaced -o name  | \
  xargs -n 1 kubectl get --show-kind --ignore-not-found -n $NAMESPACE -oname | \
  xargs -n 1 kubectl delete -n $NAMESPACE

You can also add a dry-run flag to the delete or leave it out to just see the resources that are still there.

Delete resources stuck in terminating

Sometimes other resources can get stuck in terminating too (especially when you have custom controllers that might be offline or not functioning as expected), when something attaches a finalizers to a resource but doesn’t finish cleanup.

Patching the object by removing the finalizer will delete it. NOTE: This will not finish any steps that were supposed to be executed by whatever placed the finalizer, so run this only when you know what you’re doing.

kubectl get <resource e.g. pods> -l <common label> -n <ns> -oname | \
  xargs kubectl patch  --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]' -n <ns>

Get old logs

Sometimes a container goes away too quickly to capture its logs, luckily you can get the previous container logs from a pod that has restarted as long as it is still around.

kubectl logs <podname> --previous

K8s docs

Ephemeral debug pods

I was so glad when this feature was introduced. No more random debug pods, no more changing your minimal image to use debugging tools in dev.

kubectl debug -it <pod> --image=busybox:1.28 --target=<pod>

K8s docs

Create a Kubernetes event manually

Either by applying the manifest

kubectl apply -f - <<EOF
apiVersion: v1
kind: Event
metadata:
  name: myevent
  namespace: default
type: Normal
message: 'some message'
involvedObject:
  kind: someObject
EOF

or by passing the JSON:

jq --arg now "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" -n '{apiVersion: "v1", kind: "Event", metadata: { namespace: "default", generateName: "event-" }, involvedObject: {apiVersion: "v1", kind: "Pod", name: "pod", namespace: "default", uid: "6d496a1a-bbc5-a5bb-12a3-133eq7ca0383", fieldPath: "spec.containers{container}"}, message: "Hello world!", firstTimestamp: $now, lastTimestamp: $now, type: "Normal", reason: "JustBecause", source: {component: "event-client"} }' | kubectl create -f -

which creates an event that can then be seen or used to for example trigger an operator’s control loop manually.

$ kubectl get event
LAST SEEN   TYPE     REASON         OBJECT                            MESSAGE
4s          Normal   JustBecause    pod/pod                           Hello world!

Testing: Mocking the Kubernetes API in Go

2022-09-14T00:00:00+00:00

Whether you’re writing an operator just interacting with the Kubernetes API for other reasons, you’ll want to write unit tests for your code. Luckily the Kubernetes Go-client comes with it’s own mocking library to fake any reaction your API server might have.

Getting started

For a simple mock like creating a deployment we can start by creating a fake ClientSet and swapping out the dependency.

For a function that creates a deployment, for example

func createDeployment(ctx context.Context, client kubernetes.Interface, name string, namespace string) error { ... }

we can create a simple main_test.go (or whatever your package is called) that looks like this:

package main

import (
    "context"
    "testing"
    "time"

    "github.com/stretchr/testify/require"
    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes/fake"
)

func TestCreateDeployment(t *testing.T) {
    tests := []struct {
        name string
        err  error

        client *fake.Clientset
    }{
        {
            name:   "success - created deployment",
            err:    nil,
            client: fake.NewSimpleClientset(),
        },
    }
    for _, test := range tests {
        t.Run(test.name, func(t *testing.T) {
            ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
            defer cancel()

            err := createDeployment(ctx, test.client, "test", "default")
            if test.err == nil {
                require.NoError(t, err)
            } else {
                require.EqualError(t, err, test.err.Error())
            }
        })
    }
}

We’ve replaced the client with the fake client, so now calling the CreateDeployment function has a simple happy path. But what if we want to ensure our code works as intended if the deployment doesn’t get created?

Adding a Failure

We can add a test case for an error:

        {
            name:   "failure - creation errors",
            err:    fmt.Errof("error"),
            client: fake.NewSimpleClientset(),
        },

but that doesn’t actually tell the fake client to return anything else than before. What we need is something called a reactor, which (as the name implies) can add reactions to requests that are sent to the fake API server.

So we add a field to our tests struct, for adding a reactor function:

    reactor func(t *testing.T, client *fake.Clientset) func(action k8sTesting.Action) (bool, runtime.Object, error)

and change the failure case to:

        {
            name:    "failure - creation errors",
            err:     fmt.Errorf("fake error"),
            client:  fake.NewSimpleClientset(),
            reactor: createDeploymentFail,
        },

Now we can create a reaction function:

func createDeploymentFail(t *testing.T, client *fake.Clientset) func(action k8sTesting.Action) (bool, runtime.Object, error) {
    return func(action k8sTesting.Action) (bool, runtime.Object, error) {
        return true, nil, fmt.Errorf("fake error")
    }
}

and don’t forget to add the reactor to the client in the test run:

test.client.PrependReactor("create", "deployments", test.reactor(t, test.client))

This lets us control the return values of any request, but it can easily make you want to add functions for all the “common” behaviour you’d expect from the Kubernetes API. Like creating a deployment would mean there’s now a replicaset and pods and they’re up and running, right?

But since the point of this is not to emulate or rebuild Kubernetes functionality it comes down to figuring out what resources your own code is relying on and is reacting to. It’s also interesting to see if the reaction to your cluster misbehaving are as expected.

You can also add resources to the fake client immediately during initialization, e.g.

client: fake.NewSimpleClientset(
                &appsv1.Deployment{
                    ObjectMeta: metav1.ObjectMeta{
                        Name:      "test",
                        Namespace: "default",
                        Labels: map[string]string{
                            "app": "test",
                        },
                    },
                    Spec: appsv1.DeploymentSpec{
                        Replicas: Pointer(int32(2)),
                        Selector: &metav1.LabelSelector{
                            MatchLabels: map[string]string{
                                "app": "test",
                            },
                        },
                        Template: corev1.PodTemplateSpec{
                            ObjectMeta: metav1.ObjectMeta{
                                Labels: map[string]string{
                                    "app": "test",
                                },
                            },
                        },
                    },
                },
            ),

This can make it harder too read after adding a few resources, so focussing on only the absolute minimal examples or even reading a manifest file into an object might be preferable.

Creating a deployment is just a very simple example of what can be tested using the mocking feature of the go-client library, but it already requires some knowledge of Kubernetes internals like which resources exist and what they look like, what actions can and need to be mocked. Some simple tests for success and error responses are a good way to get started and see that unit testing your Go code that touches Kubernetes is not such a complicated task after all.

Trying out Scroll

2022-09-03T00:00:00+00:00

After running this blog with Jekyll and github-pages for over two years, I ran into Scroll and fell in love with the newspaper-y style and minimalist setup, so I wanted to try it out.

Getting started

Scroll is a easy to install as

npm install -g scroll-cli

git clone https://github.com/breck7/scroll
cd scroll
npm install -g .

and then running scroll init inside the blog directory. After that I converted all posts to .scroll files and pushed them to a separate branch.

Changes

The structure changes a bit, with all files in a flat structure (aside from images, those can stay in a subfolder). For the previous markdown in my posts I just added the markups.scroll to avoid too much rewriting. There are a few features that would go away with the switch, namely analytics and tagged pages. Though tagging seems to be replaceable with categories and I might have found a way to add the javascript to each page.

Running locally

For local rendering, it’s now scroll watch instead of jekyll -s, though you can also run scroll build and open the .html files in your browser directly.

Github Pages

Comparison

Comparing the Jekyll design to Scroll

It gives the site a different feel, but while I do really like the look of it I’m sticking with the Jekyll version for now, mostly for the Markdown and out of the box analytics support.

Tracing with Jaeger on EKS with Istio

2022-04-13T00:00:00+00:00

Understanding and debugging microservices communication can be a challenge in itself. Luckily there are a bunch of tools to make it easier. In addition to the observability you get from running visualization tooling like Kiali with your service mesh, tracing provides valuable information for troubleshooting. That’s where Jaeger comes in.

Installing Jaeger

Jaeger provides an Operator, and a Helm chart to get started. Using the Helm chart, installation is simple:

$ helm repo add jaegertracing https://jaegertracing.github.io/helm-charts

to use custom settings add your own values.yaml file (e.g. an existing ElasticSearch cluster as a datastore in this case)

storage:
  type: elasticsearch
  elasticsearch:
    host: elasticsearch.elk.svc.cluster.local  # using a local elasticsearch
    port: 9200
    # user: <USER>
    # password: <PASSWORD>
provisionDataStore:
  cassandra: false                             # disabling the default cassandra
  elasticsearch: false                         # disabling creating a ES cluster
collector:
  service:
    zipkin:                                    
      port: 9411                               # opening the port for istio to push

Install jaeger with

$ helm install jaeger jaegertracing/jaeger --values values.yaml --namespace istio-system

Resulting in something like this

NAME: jaeger
LAST DEPLOYED: Mon Apr 11 18:36:34 2022
NAMESPACE: istio-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
###################################################################
### IMPORTANT: Ensure that storage is explicitly configured     ###
### Default storage options are subject to change.              ###
###                                                             ###
### IMPORTANT: The use of <component>.env: {...} is deprecated. ###
### Please use <component>.extraEnv: [] instead.                ###
###################################################################

You can log into the Jaeger Query UI here:

  export POD_NAME=$(kubectl get pods --namespace istio-system -l "app.kubernetes.io/instance=jaeger,app.kubernetes.io/component=query" -o jsonpath="{.items[0].metadata.name}")
  echo http://127.0.0.1:8080/
  kubectl port-forward --namespace istio-system $POD_NAME 8080:16686

giving you the commands you need to locally check out the UI.

Adapting Istio

To see actual traces, you’ll need to edit Istio to forward data to Jaeger. Edit the istio ConfigMap in istio-system to point tracer.zipkin.address to

jaeger-collector.istio-system.svc.cluster.local:9411

You need to restart the istiod pod for the change to take effect.

Architecture

The traffic data goes from each istio-proxy sidecar to agents (running as DaemonSets on each host), to the collector, which validates, indexes and transforms the traces and either stores them or forwards them to Kafka. The default setup “samples” 1% of transactions/traces.

The github page has a great architecture diagram here.

Usage

After sending a few hundred requests to your application (or bookinfo as a standin), you can see the traces in the UI.

The System Architecture view won’t show anything, unless you’re using the all-in-one setup. For seeing the entire distributed architecture you can use Kiali or add Jaeger analytics to fill in the blank page.

Getting started with Chaos Engineering using Litmus

2022-01-06T00:00:00+00:00

The idea of using Chaos Engineering to improve systems is becoming more and more popular with the Kubernetes crowd, no doubt due to Netflix’s Simian Army and the experiences that followed. Simulating outages or different errors to test resilience, understanding your system better and battle-test your applications sounds like a good idea, here’s one way of getting started.

Understand the basics

Chaos Engineering is NOT about breaking things randomly and seeing what happens. It’s about making an assumption about your system, trying to understand what would happen if a certain error or outage happens and testing that assumption in a controlled environment. For real-life systems that involves a large amount of preparation, communication and learning from whatever results a chaos experiment may have.

If you’re just getting started and want to play around and learn, Litmus is a good tool to try on Kubernetes.

Install

Litmus describes itself as an end-to-end chaos engineering platform for cloud native infra and applications. It’s control plane comes as a Kubernetes deployment consisting of a frontend, a server and a MongoDB backend, all together called the Chaos Center.

kubectl apply -f https://litmuschaos.github.io/litmus/2.4.0/litmus-2.4.0.yaml

The above command is all it takes to install it on your cluster (see the docs for the latest version).

Once the pods are up and running, you can access the frontend via <IP>:<Nodeport> which you can see in the service that gets created (in my case it’s on a local cluster, so localhost:30711).

NAMESPACE     NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                       AGE
litmus        litmusportal-frontend-service   NodePort    10.98.60.168     <none>        9091:30711/TCP                                                66s
litmus        litmusportal-server-service     NodePort    10.109.185.162   <none>        9002:32132/TCP,8000:30885/TCP,9003:31270/TCP,3030:31009/TCP   65s

After logging in with the default credentials of user: admin and pw: litmus, you’re prompted to change the password. In the Chaos Center you can create workflows, add different hubs (like experiment libraries), agents or monitor experiment runs and results.

(The 9002 port on the server lets you access the internal GraphQL playground.)

Concepts

This post goes into more detail, but basically Litmus runs as an operator, using different CustomResources to define states and custom controllers to create Chaos Runners which execute the experiments as jobs.

CRs

Chaos Experiment: the actual test you want to run
Chaos Engine: the mapping of the test to the thing you want to test (infra/app)
Chaos Result: giving you the current state and outcome of a test

A Chaos Workflow can be used to add additional steps around an experiment, for example to add preparation, cleanup or just to execute multiple operations.

Running a test

With Litmus up and running you now have different options

Run a sample workflow using the UI
Run a workflow from the ChaosHub: The Hub contains different generic experiments like introducing network latency or node restarts, as well as application-specific experiments for a some cloud providers, Kafka, Cassandra.
Write your own (using Python or Golang)

For more resources, check out awesome chaos engineering on GitHub.

Using AWS Lambda to write from S3 to DynamoDB

2021-12-16T00:00:00+00:00

Creating a lambda function triggered by an S3 event can be done in different ways. Using a blueprint, the console or AWS CDK.

We’ll take a look at creating a Python Lambda function created from Typescript CDK code.

CDK Resources

We create a separate CDK Stack for an S3 bucket

    const bucket = new s3.Bucket(this, 'S3Bucket', {
      bucketName: 'somebucketnamethatsnottaken',
      autoDeleteObjects: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      encryption: s3.BucketEncryption.S3_MANAGED,
    });
    bucket.grantRead(new AccountRootPrincipal());

a DynamoDB table

    const dynamoTable = new Table(this, 'DynamoDb', {
      partitionKey: {
        name: 'id',
        type: AttributeType.STRING
      },
      tableName: 'sometable',
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

and finally a Lambda function.

    const lambdaFunction = new lambda.Function(this, "Lambda", {
      runtime: lambda.Runtime.PYTHON_3_9,
      code: new lambda.AssetCode("./lambda"),
      handler: "lambda_function.lambda_handler",
      vpc: vpc,
    });

    // this grants the Lambda function rights to read from the bucket
    bucket.grantRead(lambdaFunction);
    // this grants the Lambda function right to write to DynamoDB
    dynamoTable.grantReadWriteData(lambdaFunction);

    // this adds a trigger for any time an object is put into the bucket
    const s3PutEventSource = new lambdaEventSources.S3EventSource(bucket, {
      events: [
        s3.EventType.OBJECT_CREATED_PUT
      ]
    });
    lambdaFunction.addEventSource(s3PutEventSource);

The typescript files will usually be in the lib folder (easy to get starting using the AWS docs) and are added to the app in bin. The above Lambda function code is expected to be found in a file lambda/lambda_function.py in the same directory as bin and lib.

Python code

This function consists of two simple pieces, reading the CSV file from S3 and writing its contents to the DynamoDB table.

"""Imports CSV from S3 to DynamoDB"""
import uuid
import boto3

TABLE = 'sometable'

s3 = boto3.resource('s3')
db_table = boto3.resource('dynamodb').Table(TABLE)

def write_to_dynamodb(header, arr):
    """Assigns random id to csv line and writes to table"""
    items = dict(zip(header, arr))
    items['id'] = str(uuid.uuid4())
    return db_table.put_item(Item=items)

def lambda_handler(event, context):
    """Gets CSV and splits into lines"""
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    tmp_file = '/tmp/' + key

    s3.meta.client.download_file(bucket, key, tmp_file)
    with open(tmp_file, 'r') as f:
        header = next(f).rstrip().split(';')
        for line in f:
            if line != '\n':
                arr = line.rstrip().split(';')
                write_to_dynamodb(header, arr)
    return {'statusCode': 200}

Deployment

Now all you have to do is create your stack using cdk deploy and testing the function by adding a CCSV file to your S3 bucket.

User; Id; First Name; Age
spidey;123;Peter;22
mi6;007;James;60
McTestface;7;testy;4

Containerizing a simple Flask App

2021-11-29T00:00:00+00:00

It never hurts to have a basic example for a quick WebApp to deploy onto a container service. Here’s one for a Python app, using Flask, Docker and Gunicorn.

Folder structure

To get started we create the following structure (it’s ok to leave the files empty, we’ll fill them during the next few instructions):

.
├── Dockerfile                     # Instruction for building the docker image
├── app
│   ├── app.py                     # Actual python source
│   ├── templates                   
│   │   └── factorial-form.html    # Template for HTML used in the WebApp
│   └── tests
│       ├── __init__.py
│       └── test_app.py            # Simple tests for the webapp
└── requirements.txt               # Required packages

After this we can get started with the Python code.

Writing the Flask app

If you’ve never used Flask before, checkout the docs. It’s a micro web framework, making it super easy to build a WebApp.

Our app.py will look like this:

"""Main flask module."""
import math
from flask import Flask, render_template, request

app = Flask(__name__)


@app.route('/')
def factorial_form():
    """Renders main input page."""
    return render_template('factorial-form.html')

@app.route('/', methods=['POST'])
def form_post():
    """Calculates and returns factorial of input."""
    inp = int(request.form['Factorial'])
    fac = math.factorial(inp)
    return f"The factorial of {inp} is {fac}."


if __name__ == '__main__':
    app.run()

The first function creates the main page from the template file

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>FactorialApp</title>
</head>
<body>
    <h1>Calculate factorials</h1>
    <h4>Enter a number:<h4>
    <form method="POST">
        <input type="number" name="Factorial" min="0" required>
        <input type="submit" value="Calculate">
    </form>
</body>
</html>

which is basically just a form taking a single number as an input. The second function of app.py receives the input from the POST request, calculates the factorial, the product of all positive integers less than or equal to the input. E.g. the factorial of 5 would be 54321=120. The calculated number is then returned as part of a string, to create the output page.

To test this code you can simply run

flask run

and then you can see the App running on http://127.0.0.1:5000/.

Testing

We use pytest for testing, and add a few simple unit tests.

import app
import json
import pytest

@pytest.fixture
def client():
    client = app.app.test_client()
    return client

def test_home_page(client):
    """
    GIVEN an app configured for testing
    WHEN the '/' page is requested (GET)
    THEN check response is valid and correct
    """
    response = client.get('/')
    assert response.status_code == 200
    assert b"Enter a number:" in response.data

def test_home_page_post(client):
    """
    GIVEN an app configured for testing
    WHEN the '/' page is is posted to without num (POST)
    THEN check that an error is returned
    """
    response = client.post('/')
    print(response.data)
    assert response.status_code == 400
    assert b"The factorial of" not in response.data

def test_home_page_post_number(client):
    """
    GIVEN an app configured for testing
    WHEN the '/' page is is posted to with a number (POST)
    THEN check that its factorial is returned
    """
    headers={'Content-Type': 'application/x-www-form-urlencoded'}
    response = client.post(
                '/',
                data="Factorial=5",
                headers=headers,
            )
    assert response.data == b"The factorial of 5 is 120."

First we need a fixture to create the client to run the tests against (think of it like an instance of your app). Then we try to see if the webpage is up and contains some of the HTML of the template.

The second test simply checks if a false POST receives the error we want (no input, no result).

The third test checks if the calculation returns the expected result for a specific input.

To run the tests

pytest -v

and you should receive an output like

============================================ test session starts ============================================
platform darwin -- Python 3.9.9, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/opt/python@3.9/bin/python3.9
cachedir: .pytest_cache
rootdir: /Path/To/App
plugins: dash-2.0.0
collected 3 items

tests/test_app.py::test_home_page PASSED                                                              [ 33%]
tests/test_app.py::test_home_page_post PASSED                                                         [ 66%]
tests/test_app.py::test_home_page_post_number PASSED                                                  [100%]

============================================= 3 passed in 0.04s =============================================

Dockerize

After finishing our app, we can get started on creating the Dockerfile.

# base image to build
FROM python:3.7 as builder
RUN mkdir /install
WORKDIR /install
COPY requirements.txt /requirements.txt
RUN pip install -r /requirements.txt --target=/install

# smaller image to run
FROM python:3.7-alpine
COPY --from=builder /install /usr/local
COPY app /app
WORKDIR /app
ENV PYTHONPATH="${PYTHONPATH}:/usr/local"
EXPOSE 5000
CMD ["gunicorn", "-w 4", "-b 0.0.0.0:5000", "app:app"]

The first image create is the builder image, where we later copy our necessary python packages from. This is a multi-stage Dockerfile, so you’ll end up with a smaller image in the end. We’re using gunicorn as a production server for Python.

Then you simply build and run it

docker build -t factorialWebApp:v0.1 .
docker run -it -p 5000:5000 factorialWebApp:v0.1

And your WebApp in now running on http://127.0.0.1:5000!

Adding a secondary fluentd to Openshift Logging as a syslog receiver

2021-11-17T00:00:00+00:00

OpenShift comes with loads of handy operators managing various Kubernetes resources for you, some open source, some not. OpenShift Logging is the wrapper for an EFK stack running on your cluster. Like with other operators you don’t create your own daemonsets or deployments, instead you configure it with a CustomResource called ClusterLogging.

So if you’re thinking of using this Fluentd for anything other than logs from this cluster, you’re pretty much out of luck.

Note: You can set everything to Unmanaged and change the configs, but that’s not sustainable as you’re basically disabling any operator actions. Also editing what’s supposed to be an indented file represented as one long string is no fun.

You can however piggyback on the resources deployed by the operator by adding a secondary Fluentd in the same namespace with some modifications. We’ll create a Fluentd that acts as a syslog receiver, though you could in theory add any input you like, see the list of Fluentd input plugins. And we’ll re-use the ElasticSearch and Kibana pods to visualize those logs.

Secondary Fluentd

Assuming the logging operator is installed and one instance of the EFK pods is up and running:

Let’s start with a ConfigMap, where we add the Fluentd config file receiving logs on (the pods) port 5140, and sending everything as is to the local ElasticSearch. The exact paths might vary, but can be taken from the primary Fluentd daemonset.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: openshift-logging
data:
  fluent.conf: |-
    #syslog plugin
    <source>
      @type udp
      port 5140
      bind 0.0.0.0
      <parse>
        @type none
      </parse>
      tag system
    </source>
    <match **>
      @type copy
      <store>
        @type elasticsearch
        host elasticsearch.openshift-logging.svc
        port 9200
        logstash_format true
        scheme https
        ssl_version TLSv1_2
        client_key '/var/run/ocp-collector/secrets/fluentd/tls.key'
        client_cert '/var/run/ocp-collector/secrets/fluentd/tls.crt'
        ca_file '/var/run/ocp-collector/secrets/fluentd/ca-bundle.crt'
      </store>
    </match>

There is a syslog input plugin as well, but that might need specific settings depending on the syslog message format. After the config, we can create our deployment (a daemonset might be more useful for you).

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: fluentd-syslog
  name: fluentd-syslog
  namespace: openshift-logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fluentd-syslog
  template:
    metadata:
      labels:
        app: fluentd-syslog
    spec:
      containers:
      - image: quay.io/fluentd_elasticsearch/fluentd:v3.3.0
        imagePullPolicy: IfNotPresent
        name: fluentd
        env:
        - name: FLUENTD_ARGS
          value: --no-supervisor
        ports:
        - containerPort: 24224
          name: forward
          protocol: TCP
        - containerPort: 5140
          name: syslog
          protocol: TCP
        volumeMounts:
        - name: varlog
          mountPath: /var/log
          readOnly: false
        - name: config
          mountPath: /etc/fluent/config.d
        - mountPath: /var/run/ocp-collector/secrets/fluentd
          name: fluentd
          readOnly: true
        - mountPath: /etc/pki/ca-trust/extracted/pem/
          name: fluentd-trusted-ca-bundle
          readOnly: true
      volumes:
      - name: varlog
        emptydir: {}
      - configMap:
          defaultMode: 420
          name: fluentd-config
        name: config
      - name: fluentd
        secret:
          defaultMode: 420
          secretName: fluentd
      - configMap:
          defaultMode: 420
          items:
          - key: ca-bundle.crt
            path: tls-ca-bundle.pem
          name: fluentd-trusted-ca-bundle
        name: fluentd-trusted-ca-bundle

The only differences between this an a generic deployment are the additional mounted volumes for the certificates as well as an image that already contains the ElasticSearch plugin.

You can build your own image as well, see the README of the original Fluentd image for instructions.

After creating the deployment we only need to expose the port from inside the pod to the outside world. If you have a loadbalancer or ingress, use that. Otherwise a Nodeport works just as well.

---
apiVersion: v1
kind: Service
metadata:
  name: fluentd-syslog
  namespace: openshift-logging
  labels:
    app: fluentd-syslog
spec:
  type: NodePort
  ports:
  - port: 5140
    targetPort: 5140
    nodePort: 30514
    protocol: UDP
  selector:
    app: fluentd-syslog

This service exposed the deployment on the nodes port 30514.

Testing

Now you can test the deployment by sending a message yourself, for example

nc -w0 -u <Node IP> 30514 <<< "FAKE SYSLOG MESSAGE"

After a few seconds the log should show up inside your Kibana.

Running PowerDNS in docker-compose

2021-09-08T00:00:00+00:00

PowerDNS is easily installed and run on most servers, whether you’re using it as an authoritative server or a recursor. To better understand the difference, check out this blog.

Basically your authoritative server has the actual DNS info you’re looking for or providing (usually an IP), and your recursive server is the one finding the path to the right server and caching any previously checked information.

Checking out PowerDNS

While most other nameservers combine recursive and authoritative functions, PowerDNS or pdns comes in two flavours (pdns-auth and pdns-recursor). Trying it out by installing the binaries is easy enough with the installation docs. The default comes with a bind backend, but in this example we’re using sqlite3 instead.

So for the containerized version, all we need is a server with:

docker
docker-compose
port 53 (and potentially 8081/8082 if you want to contact pdns via API) open

We start with the docker-compose.yaml file:

# based on https://github.com/PowerDNS/pdns
---
version: '2.0'
services:
  recursor:
    image: powerdns/pdns-recursor-45:4.5.5 # the latest pdns-recursor from dockerhub
    # the forward-zones are used to make sure the recursor asks the local authoritative server 
    # about our own domains
    command: --forward-zones=<domain>=172.25.0.2:5300 --local-address=0.0.0.0 --local-port=53 
    environment:
      - PDNS_RECURSOR_API_KEY
    ports:
      - "53:53"
      - "53:53/udp"
      - "8082:8082" # HTTP API for the recursor

  auth:
    image: powerdns/pdns-auth-45:4.5.1 # latest pdns-authoritative nameserver image
    command: --local-address=0.0.0.0 --local-port=5300
    user: root
    environment:
      - PDNS_AUTH_API_KEY
    ports:
      - "5300:5300" # this is the port we redirect queries to above
      - "5300:5300/udp"
      - "8081:8081" # HTTP API of the server
    volumes:
      - /var/lib/powerdns/pdns.sqlite3:/var/lib/powerdns/pdns.sqlite3
    networks:
      default:
        ipv4_address: 172.25.0.2

networks:
  default:
    driver: bridge
    ipam:
      config:
        # defining the network range and IP for the authoritative server is only necessary in 
        # specific cases (e.g. to avoid breaking networking on some servers, most do fine without)
        - subnet: 172.25.0.0/16

Initializing the database

Before running pdns, the sqlite database needs to be created.

mkdir /var/lib/powerdns
sqlite3 -init schema.sql /var/lib/powerdns/pdns.sqlite3

Where schema.sql is taken from here.

Environment variables for API access

To enable the API access to pdns, make sure you have the following environment variables set:

export PDNS_RECURSOR_API_KEY=changeme
export PDNS_AUTH_API_KEY=changeme

Starting the containers is as simple as running docker-compose up -d in the folder where the above YAML file is.

Testing

Check if your containers are successfully running with docker-compose ps -a. If they’re up, you can test the service using

curl -v -H 'X-API-Key: changeme' http://<serverIP>:8081/api/v1/servers/localhost/ | jq

To add zones to the authoritative server you can use pdnsutil or the API.

To check if you’ve successfully added a zone or record, you can check with either

nslookup <domain> <serverIP>

dig <domain> @serverIP

Adding a --port 5300 to the dig command, let’s you test the authoritative server directly.