10/20/2024

Build a Fully-Automated DevOps Pipeline with GitOps


 


With the advancement in modern-day technologies, most teams are accelerating the software development lifecycle through automation. However, to attain total agility in a software development pipeline, DevOps teams require technologies to deploy software applications while maintaining IT infrastructure effectively continually. In this article, I will introduce an operational framework called GitOps that provides solutions to these demands. I will discuss its components and how GitOps works. I will highlight its principles, benefits and challenges. I’ll also discuss what distinguishes GitOps from DevOps and list the top GitOps tools.

Why GitOps?

Previously, managing IT infrastructure was a complex task. Most infrastructure configurations were manually managed, configured and distributed by system administrators(IAC). And, because different team members simultaneously collaborated on these infrastructures, this often resulted in multiple errors.

However, recent technologies are adapting new methods of designing, developing, documenting and maintaining IT Infrastructure to make it easier to edit and distribute configuration specifications among teams. DevOps and engineering teams can now create & store infrastructure files locally, execute and execute locally using a version control system.

Even though the introduction of IAC brought a lot of benefits to DevOps teams, this process also had its challenges. Since it allowed multiple teams to collaborate in the infrastructure configuration simultaneously, it required a review and approval process. It lacked tests to confirm changes from individual team members, and infrastructure updates were done manually. DevOps teams needed an automated, scalable way to document and provision application infrastructure; GitOps.

What is GitOps?

GitOps is an operational paradigm that extends DevOps best practices for application development and integrates them into infrastructure automation(IAC). GitOps is a blend of DevOps best practices such as version control, code review, and CI/CD pipelines to store and manage all data, documentation, and operational tasks for Kubernetes. It Allows you to manage architecture and rapidly reproduce the system’s cloud infrastructure by utilising git as the single source of truth for comprehensive automated testing and deployment. GitOps expands the capability of IaC by keeping the code, testing, staging, and production environments in sync. Similar to how application source code creates the same application binaries each time it is generated, GitOps configuration files provide the same infrastructure environment each time it is deployed.

Three components of GitOps

GitOps is no single tool or product; Instead, it combines IAC and complete DevOps pipeline practices. To get started with GitOps, you should consider these components.

  • Infrastructure as code (IaC)

GitOps used a version control system called git as the single source of truth for handling and storing application infrastructure as code. IAC defines infrastructure using configuration files(code) rather than manually using a graphical user interface. Infrastructure can be easily procured through configuration files that hold the infrastructure specifications, making it easy to edit and distribute configuration. DevOps teams use this to improve the infrastructure’s consistency, stability and scalability. Collaboratoration to the infrastructure configuration and provisioning takes the same flow as the application source code.

  • Merge Requests (MRs)

Merge requests (MRs) are the modification method used by GitOps for any updates to the infrastructure. Teams can collaborate on the infrastructure through reviews and comments before changes are officially approved in the main branch. A merge request acts as an audit log and helps keep track of the changes implemented by each team member. It is used to deploy and validate system infrastructure changes automatically. With an inbuilt roll-back feature, revert changes that are to the desired version.

  • Continuous integration and continuous deployment (CI/CD)

In a GitOps environment, infrastructure management is automated using a CI/CD(Continuous integration and Continuous deployment) pipeline. So, whenever a code is merged to the main branch, CI/CD acts as a reconciliation loop. It initiates the change from the git repository to the environment. GitOps automation and continuous deployment eliminate human errors that arise when infrastructure configuration is done manually.

How does GitOps work?

GitOps allows developers to manage infrastructure automation of various systems such as VMs and containers; it is especially popular among teams managing Kubernetes-based infrastructure. GitOps guarantees that a system’s cloud lives infrastructure is reconfigured and syncs with the state of a Git repository as soon as the merged request is approved.

Consider it this way, assume you developed a central repository containing all of the configuration files (YAML files), documentation, and code required for Kubernetes. You then automated it such that other system administrators responsible for Kubernetes deployment can simply clone the repository, modify the code, and submit a merge request to the central Git repository.




The workflow of a typical GitOps environment looks like this;

  • You make a pull request for a new feature to IAC, configuration, or application code on the Git repository.
  • The code will get reviewed and the changes approved.
  • After that, the code gets merged into the Git repository.
  • Immediately CI build pipelines are established and triggered from pull requests. It runs a few checks, and if all of them pass, it creates a new image and sends it to the image container.
  • The deployment Automator identifies changes to the image repository when the merge request is finished, grabs it from the registry, and updates the YAML configuration file.
  • This declares the changes operational, and the modification is automatically distributed to the cluster.

Benefits of GitOps

  • Allows team collaboration

Since it uses version control, respective members can simultaneously contribute to the configurations. Team members can create a merge request that suggests changes to the code. There would not be any cases of conflicting changes as every new configuration goes through the review process before it’s approved.

  • Version control

GitOps is a version control system; therefore, it has all of the capabilities of git, such as detailed audit records of all changes made. Details like the committer’s identity, the commit timestamp, and the commit ID are saved when a commit is made. As a result, you have total access to the system and can track all changes made to the infrastructure configuration.

  • Increases reliability

Manually configuring your application infrastructure can be unreliable, but with an automated system, you can achieve fewer errors and faster problem resolution. GitOps rescues you against configuration drift and snowflake environment by combining DevOps techniques and IAC. Also, since git is the single truth source, you can be sure of a reliable system.

  • Automation

Automating the entire deployment process saves time and increases productivity for a DevOps team. You can even increase the number of changes made to the configuration and spend more experimenting with new infrastructure configurations. The CI/CD methods are utilised to keep your infrastructure operational and lower downtime even when these modifications are being deployed.

GitOps vs DevOps: what’s the difference?

Although GitOps is a branch of DevOps, determining which approach is the best option depends on the project’s objectives and goals. DevOps is a philosophy based on collaborative efforts between developers and operators to accelerate the software development life cycle. While GitOps focuses on delivery, DevOps is significantly broader in scope since it addresses a broader set of issues such as CI, CD, visibility, and governance. In GitOps, the git repository is the source of truth for the deployment state, whereas, in DevOps, it is the application or server configuration files. Furthermore, DevOps may be applied to every process within an application, while GitOps is commonly used in combination with containerisation technologies such as Kubernetes.

GitOps Tools & Technologies

Some tools and technologies that can be sued to achieve GitOps procedures include:

Conclusions

GitOps is a methodology that combines DevOps practices with IAC to provide a comprehensive application development lifecycle and is quickly becoming a widespread practice in modern cloud infrastructure. GitOps employs git as its single source of truth, making it more trustworthy and straightforward for developers to get started. Adopting GitOps methods allows you to leverage the most recent DevOps principles and technologies to create a high-quality software development environment. Teams can be more productive and consistent in their application development while managing the current cloud infrastructure. Finally, while some businesses may take time to embrace GitOps and see the benefits, this methodology is still expanding in capabilities and popularity.

10/14/2024

Flink + Docker + Kafka

 Apache Flink is a powerful stream processing framework that enables real-time data processing. Docker provides an easy way to set up and experiment with Apache Flink locally. In this article, we'll guide you through running Apache Flink with Docker, demonstrate how to integrate Apache Kafka with Flink using a Dockerfile, and provide an example Flink script using Python for stream processing.

Setting Up Apache Flink with Docker

Step 1: Install Docker
If Docker is not installed on your system, you can follow the instructions in the [official documentation](https://docs.docker.com/get-docker/) to install it.

Step 2: Run Apache Flink Container
Run the following command in your terminal to start an Apache Flink container:

docker run -d -p 8081:8081 apache/flink:1.20.0

This will pull the Apache Flink image and start a container with the Flink web dashboard accessible at `http://localhost:8081`.

Dockerfile for Apache Kafka and Flink Integration

Step 1: Create Dockerfile
Create a `Dockerfile` in a directory of your choice with the following content:


FROM flink:1.20.0
# Install Kafka connector dependencies
RUN mkdir -p /opt/flink/usrlib
RUN wget -P /opt/flink/usrlib https://repo.maven.apache.org/maven2/org/apache/flink/flink-connector-kafka_2.12/1.14.0/flink-connector-kafka_2.12-1.14.0.jar

Step 2: Build and Run Docker Image
Navigate to the directory containing the `Dockerfile` and run the following commands:

docker build -t flink-kafka-integration .
docker run -it flink-kafka-integration

This will build the Docker image and start a container with Apache Flink and the Kafka connector integrated.


How I Dockerized Apache Flink, Kafka, and PostgreSQL for Real-Time Data Streaming | by Augusto de Nevrezé | Towards Data Science


Apache Flink with Apache Kafka | petitviolet blog



https://hub.docker.com/r/apache/kafka


How to getting started with Apache Kafka on Docker

 Apache Kafka, a distributed streaming platform, is a powerful tool for building real-time data pipelines and streaming applications. Docker simplifies the process of setting up Kafka locally, allowing developers to experiment and develop with ease. In this article, we’ll walk through the steps to download, install, and run Apache Kafka using Docker, providing a hands-on implementation for beginners.


Prerequisites:

Before we begin, ensure that you have Docker installed on your local machine. You can download and install Docker from the official Docker website.


Step 1: Create a Docker Compose File

Create a docker-compose.yml file in a new directory to define the Kafka and Zookeeper services. Copy and paste the following content:

version: '2'

services:
zookeeper:
image: wurstmeister/zookeeper:latest
ports:
- "2181:2181"

kafka:
image: wurstmeister/kafka:latest
ports:
- "9092:9092"
expose:
- "9093"
environment:
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CREATE_TOPICS: "telegram-topic:1:1"
volumes:
- /var/run/docker.sock:/var/run/docker.sock

This docker-compose.yml file defines two services: zookeeper and kafka. The Kafka service is configured to expose ports 9092 and 9093.

Step 2: Run Docker Compose

Open a terminal in the directory where the docker-compose.yml file is located and run the following command to start the Kafka and Zookeeper containers:

docker-compose up -d

This command will download the required Docker images and start the Kafka and Zookeeper services in detached mode (-d).

Step 3: Verify Kafka Container is Running

Check if the Kafka container is running by executing the following command:

docker ps

You should see containers for both Kafka and Zookeeper in the list.

Step 4: Create a Kafka Topic

Create a Kafka topic using the following command:

docker exec -it <kafka-container-id> /opt/kafka/bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic telegram-in-topic

and

kafka/bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic telegram-out-topic



Note: Replace <kafka-container-id> with the actual container ID of the Kafka container (you can find it using docker ps).

Step 5: Produce and Consume Messages

Use the Kafka console producer and consumer to test your Kafka setup:

Produce messages:

docker exec -it <kafka-container-id> /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic telegram-out-topic

Consume messages:

docker exec -it <kafka-container-id> /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic telegram-in-topic --from-beginning

Replace <kafka-container-id> with the actual container ID of the Kafka container.

Step 6: Stop and Remove Containers

To stop and remove the Kafka and Zookeeper containers, run:

docker-compose down

This will stop and remove the containers created by the docker-compose up command.

Conclusion:

Congratulations! You’ve successfully set up and run Apache Kafka on Docker locally. This hands-on guide provides a simple yet powerful environment for experimenting with Kafka. As you continue to explore Kafka, consider integrating it into your applications to harness its capabilities for building real-time data pipelines and streaming applications.










Database Persistence and Flyway in Quarkus

Database Persistence and Flyway in Quarkus  Building a Production-Ready Product Management System with PostgreSQL Introduction In ...