Using Avro with a Quarkus Application + Kafka + Schema Registry

 

Using Avro with a Quarkus Application + Kafka + Schema Registry (Product Example)

This guide walks you through building a Quarkus application that:

• Sends and receives Kafka messages

• Uses Avro for message serialization

• Integrates with a Schema Registry

• Works with a custom Product object

---

Prerequisites

You'll need:

• Java 17+

• Maven

• Docker (for Kafka & Schema Registry)

• Basic understanding of Quarkus, Kafka, and Avro

---

Architecture Overview

We’ll build a system with:

• A REST API to send Product data to a Kafka topic

• A Kafka consumer that reads Product messages

• A REST API to stream consumed products using SSE (Server-Sent Events)

• Avro used for data serialization

• Schema stored and retrieved from a Schema Registry

---

1. Create the Quarkus Project

Generate a project with required extensions:

quarkus create app org.acme:product-avro-kafka \
  --extension='rest-jackson,messaging-kafka,apicurio-registry-avro' \
  --no-code

cd product-avro-kafka

---

2. Define the Avro Schema

Create the Avro schema file product.avsc inside src/main/avro:

{
  "namespace": "org.acme.kafka.quarkus",
  "type": "record",
  "name": "Product",
  "fields": [
    { "name": "name", "type": "string" },
    { "name": "price", "type": "double" }
  ]
}

When you build the app, Quarkus automatically generates the Java class Product.java based on this schema.

---

3. Implement the Kafka Producer

Create ProductResource.java to expose a REST endpoint that sends product data to Kafka:

package org.acme.kafka;

import org.acme.kafka.quarkus.Product;
import org.eclipse.microprofile.reactive.messaging.Channel;
import org.eclipse.microprofile.reactive.messaging.Emitter;
import org.jboss.logging.Logger;

import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.core.Response;

@Path("/products")
public class ProductResource {

    private static final Logger LOGGER = Logger.getLogger(ProductResource.class);

    @Channel("products")
    Emitter<Product> emitter;

    @POST
    public Response sendProduct(Product product) {
        LOGGER.infof("Sending product: %s - $%.2f", product.getName(), product.getPrice());
        emitter.send(product);
        return Response.accepted().build();
    }
}

---

4. Kafka Configuration

In src/main/resources/application.properties, add:

# Kafka producer config
mp.messaging.outgoing.products.connector=smallrye-kafka
mp.messaging.outgoing.products.topic=products
mp.messaging.outgoing.products.apicurio.registry.auto-register=true

If using Confluent Schema Registry, use quarkus-confluent-registry-avro extension instead and replace the apicurio property with:

mp.messaging.outgoing.products.auto.register.schemas=true

---

5. Implement the Kafka Consumer and SSE Streaming

Create a class ConsumedProductResource.java:

package org.acme.kafka;

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

import org.acme.kafka.quarkus.Product;
import org.eclipse.microprofile.reactive.messaging.Channel;
import io.smallrye.mutiny.Multi;
import org.jboss.resteasy.reactive.RestStreamElementType;

@ApplicationScoped
@Path("/consumed-products")
public class ConsumedProductResource {

    @Channel("products-from-kafka")
    Multi<Product> products;

    @GET
    @Produces(MediaType.SERVER_SENT_EVENTS)
    @RestStreamElementType(MediaType.TEXT_PLAIN)
    public Multi<String> stream() {
        return products.map(p -> String.format("Product: %s - $%.2f", p.getName(), p.getPrice()));
    }
}

---

6. Kafka Consumer Configuration

Add to application.properties:

# Kafka consumer config
mp.messaging.incoming.products-from-kafka.connector=smallrye-kafka
mp.messaging.incoming.products-from-kafka.topic=products
mp.messaging.incoming.products-from-kafka.auto.offset.reset=earliest
mp.messaging.incoming.products-from-kafka.enable.auto.commit=false

---

7. Running the Application in Dev Mode

Run the application:

./mvnw quarkus:dev

Thanks to Dev Services, Kafka and Apicurio Schema Registry are started automatically in dev mode. No extra Docker setup needed.

Test the endpoints:

• Send a product:

curl -X POST -H "Content-Type: application/json" \
     -d '{"name":"Laptop","price":999.99}' \
     http://localhost:8080/products

• View the streamed products:

curl -N http://localhost:8080/consumed-products

You’ll see output like:

Product: Laptop - $999.99

---

8. Running in Production (JVM or Native)

In production, you'll need to run Kafka and Schema Registry yourself (e.g., via Docker). Here's a minimal docker-compose.yml:

version: '2'
services:
  zookeeper:
    image: quay.io/strimzi/kafka:0.41.0-kafka-3.7.0
    ...

  kafka:
    image: quay.io/strimzi/kafka:0.41.0-kafka-3.7.0
    ...

  schema-registry:
    image: apicurio/apicurio-registry-mem:2.4.2.Final
    ports:
      - "8081:8080"
    environment:
      - REGISTRY_STORAGE=mem

You will also need to configure these properties:

kafka.bootstrap.servers=localhost:9092
mp.messaging.connector.smallrye-kafka.schema.registry.url=http://localhost:8081/apis/registry/v2

---

9. Building the Application

For JVM build:

./mvnw package
java -jar target/quarkus-app/quarkus-run.jar

For native build (requires GraalVM):

./mvnw package -Pnative
./target/product-avro-kafka-1.0.0-SNAPSHOT-runner

---

10. Summary

You've just built a Quarkus application that:

✅ Defines a Product Avro schema
✅ Generates Java code from .avsc files
✅ Produces and consumes Avro messages using Kafka
✅ Uses Apicurio (or Confluent) Schema Registry
✅ Leverages Quarkus Dev Services for quick setup
✅ Streams data with Server-Sent Events (SSE)

AVRO

 

Updated Guide: Apache Avro™ with Java 

This is a short guide for getting started with Apache Avro using Java. This guide only covers using Avro for data serialization; 

---

📥 Download

Avro implementations for C, C++, C#, Java, PHP, Python, and Ruby can be downloaded from the Apache Avro™ Download page. This guide uses Avro 1.12.0, the latest version at the time of writing. For the examples in this guide, download avro-1.12.0.jar and avro-tools-1.12.0.jar.

Alternatively, if you are using Maven, add the following dependency to your pom.xml:

<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.12.0</version>
</dependency>

As well as the Avro Maven plugin (for performing code generation):

<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
  <version>1.12.0</version>
  <configuration>
    <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>
    <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
  </configuration>
  <executions>
    <execution>
      <phase>generate-sources</phase>
      <goals>
        <goal>schema</goal>
      </goals>
    </execution>
  </executions>
</plugin>
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-compiler-plugin</artifactId>
  <configuration>
    <source>1.8</source>
    <target>1.8</target>
  </configuration>
</plugin>

You may also build the required Avro jars from source. Building Avro is beyond the scope of this guide; 

---

🧾 Defining a Schema

Avro schemas are defined using JSON or IDL (the latter requires an extra dependency). Schemas are composed of primitive types and complex types.

Here’s a simple schema example, student.avsc:

{
 "namespace": "example.avro",
 "type": "record",
 "name": "Student",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}

This schema defines a record representing a hypothetical student. At minimum, a record definition must include its type ("record"), a name ("Student"), and its fields: name, favorite_number, and favorite_color.

---

⚙️ Serializing and Deserializing with Code Generation

Compiling the Schema

You can generate Java classes from the schema using avro-tools:

java -jar /path/to/avro-tools-1.12.0.jar compile schema student.avsc .

This will generate the appropriate source files in a package based on the schema’s namespace.

---

👩‍💻 Creating Students

Student student1 = new Student();
student1.setName("Alyssa");
student1.setFavoriteNumber(256);

// Alternate constructor
Student student2 = new Student("Ben", 7, "red");

// Construct via builder
Student student3 = Student.newBuilder()
             .setName("Charlie")
             .setFavoriteColor("blue")
             .setFavoriteNumber(null)
             .build();

Avro objects can be created by constructors or builders. Builders provide schema validation and handle default values, while constructors are faster.

---

💾 Serializing

DatumWriter<Student> studentDatumWriter = new SpecificDatumWriter<>(Student.class);
DataFileWriter<Student> dataFileWriter = new DataFileWriter<>(studentDatumWriter);

dataFileWriter.create(student1.getSchema(), new File("students.avro"));
dataFileWriter.append(student1);
dataFileWriter.append(student2);
dataFileWriter.append(student3);
dataFileWriter.close();

---

📂 Deserializing

DatumReader<Student> studentDatumReader = new SpecificDatumReader<>(Student.class);
DataFileReader<Student> dataFileReader = new DataFileReader<>(file, studentDatumReader);
Student student = null;
while (dataFileReader.hasNext()) {
    student = dataFileReader.next(student);
    System.out.println(student);
}

Output:

{"name": "Alyssa", "favorite_number": 256, "favorite_color": null}
{"name": "Ben", "favorite_number": 7, "favorite_color": "red"}
{"name": "Charlie", "favorite_number": null, "favorite_color": "blue"}

---

🧪 Serializing and Deserializing Without Code Generation

You can also use GenericRecords to avoid code generation.

Schema schema = new SchemaParser().parse(new File("student.avsc")).mainSchema();

GenericRecord student1 = new GenericData.Record(schema);
student1.put("name", "Alyssa");
student1.put("favorite_number", 256);

GenericRecord student2 = new GenericData.Record(schema);
student2.put("name", "Ben");
student2.put("favorite_number", 7);
student2.put("favorite_color", "red");

Serialize:

DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<>(schema);
DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<>(datumWriter);

dataFileWriter.create(schema, new File("students.avro"));
dataFileWriter.append(student1);
dataFileWriter.append(student2);
dataFileWriter.close();

Deserialize:

DatumReader<GenericRecord> datumReader = new GenericDatumReader<>(schema);
DataFileReader<GenericRecord> dataFileReader = new DataFileReader<>(file, datumReader);

GenericRecord student = null;
while (dataFileReader.hasNext()) {
    student = dataFileReader.next(student);
    System.out.println(student);
}

Output:

{"name": "Alyssa", "favorite_number": 256, "favorite_color": null}
{"name": "Ben", "favorite_number": 7, "favorite_color": "red"}

---

🚀 Compiling and Running the Example

Navigate to the project directory and run:

$ mvn compile
$ mvn -q exec:java -Dexec.mainClass=example.SpecificMain

For the generic version:

$ mvn compile
$ mvn -q exec:java -Dexec.mainClass=example.GenericMain

---

⚡ Beta Feature: Faster Code Generation

Enable faster encoding/decoding with:

$ mvn -q exec:java -Dexec.mainClass=example.SpecificMain \
  -Dorg.apache.avro.specific.use_custom_coders=true

No schema recompilation is required. This is a runtime toggle via a system property.

---

🧠 Summary

You’ve now learned how to:

• Define a schema for a Student

• Serialize/deserialize using Avro with and without code generation

• Use Avro tools and Maven integration

• Optimize performance with reuse patterns and optional beta features

QUARKUS & GraphQL

 QUARKUS & GraphQL

https://www.geeksforgeeks.org/graphql-tutorial/

https://quarkus.io/guides/smallrye-graphql-client

https://www.mastertheboss.com/soa-cloud/quarkus/how-to-build-graphql-applications-with-quarkus/

https://itnext.io/consuming-graphql-apis-with-quarkus-a2f482b6169d

https://dev.to/claudioaltamura/how-to-test-a-quarkus-graphql-api-2029

https://dev.to/claudioaltamura/graphql-apis-with-quarkus-3a6e

https://piotrminkowski.com/2021/04/14/advanced-graphql-with-quarkus/

https://jefrajames.wordpress.com/2020/04/20/developing-a-graphql-client-on-microprofile-with-quarkus/

https://hantsy.github.io/quarkus-sandbox/graphql-client.html

Jira CLI

 

Example Workflow to Create a Jira Ticket Using Jira CLI

To create a Jira ticket using the Jira CLI, you can use the jira-cli tool, which is a feature-rich command-line interface for interacting with Jira. Below is an example workflow:

Step 1: Install Jira CLI

First, ensure you have the jira-cli tool installed. You can install it using npm:

bash

npm install -g jira-cli

Step 2: Configure Jira CLI

Set up the Jira CLI by providing your Jira instance URL, email, and API token. Run the following command:

bash

jira configure

You will be prompted to enter:

Jira Base URL (e.g., https://yourcompany.atlassian.net)

Email address

API token (you can generate this from your Jira account settings).

Step 3: Create a Jira Ticket

Once configured, you can create a Jira ticket using the jira issue create command. Here's an example:

bash

jira issue create \

  --project "PROJECT_KEY" \

  --type "Task" \

  --summary "This is a sample ticket summary" \

  --description "Detailed description of the issue or task."

Replace PROJECT_KEY with the key of your Jira project (e.g., DEV, IT, etc.).

Replace "Task" with the issue type (e.g., Bug, Story, etc.).

Provide a meaningful summary and description for the ticket.

Step 4: Automate Ticket Creation (Optional)

You can integrate this command into a script or CI/CD pipeline (e.g., GitHub Actions) to automate ticket creation. For example, in a shell script:

bash

#!/bin/bash

jira issue create \

  --project "PROJECT_KEY" \

  --type "Bug" \

  --summary "Automated Bug Report" \

  --description "This bug was automatically reported by the CI/CD pipeline."

This script can be triggered whenever a specific event occurs, such as a failed build or test.

Additional Features

The jira-cli tool also supports other features like:

Transitioning tickets between statuses.

Adding comments to tickets.

Searching for issues.

For more advanced workflows, refer to the official documentation of  Jira Command Line Interface (CLI) | Atlassian Marketplace

More details: GitHub - ankitpokhrel/jira-cli: 🔥 Feature-rich interactive Jira command line.

Create issue based on template:

# Load description from template file

$ jira issue create --template /path/to/template.tmpl

# Get description from standard input

$ jira issue create --template -

# Or, use pipe to read input directly from standard input

$ echo "Description from stdin" | jira issue create -s"Summary" -tTask

kubectl/k8s Cheat Sheet

 

• Namespaces
  • List all namespaces: kubectl get namespace
  • Set a namespace: kubens <namespace-name>
  • See currently set namespace: kubens -c
• Pods
  • List all pods: kubectl get pods
  • List all pods in specific namespace: kubectl get pods -n <namespace>
  • Kill a pod: kubectl delete pod <pod-name>
  • Describe/get details of pod: kubectl describe pods <pod-name>
  • InitContainers
    • Get logs: First describe the pod and look for the name of the init container. Then run kubectl logs <pod-name> -c <init-container-name>
• Deployments
  • Get the manifest for a deployment: kubectl get deploy <deployment-name> -o yaml
  • Scaling a deployment: kubectl scale --replicas=<n> deployment/<deployment-name>
  • Gen env vars defined on a pod: kubectl exec <pod> -- env
• ConfigMaps
  • View data in a ConfigMap: kubectl describe configmaps <name>

Learning DevOps

 Learning DevOps

Terraform :  Course: Terraform for the Absolute Beginners with Labs | Udemy Business

Quarkus:  Building Microservices with Quarkus | Udemy Business

GitHub Actions: GitHub Actions - The Complete Guide | Udemy Business

Azure CLI : https://www.youtube.com/watch?v=GqpwiyYsNIw&ab_channel=AdamMarczak-AzureforEveryone

Kafka : Apache Kafka®: Basic Concepts, Architecture, and Examples

Kafka Stream : Kafka Streams: Basic Concepts, Architecture, and Examples

Kubernetes : Kubernetes for the Absolute Beginners - Hands-on | Udemy Business

Certified Kubernetes Administrator:  Certified Kubernetes Administrator (CKA) Practice Exam Tests | Udemy Business

ArgoCD Nana: ArgoCD Tutorial for Beginners | GitOps CD for Kubernetes

GitOps Nama: What is GitOps, How GitOps works and Why it's so useful

 

Other link: - https://cloud.folio3.com/blog/devops-pipeline/#elementor-toc__heading-anchor-5

Accessing ConfigMap Data in Quarkus

Accessing ConfigMap Data in Quarkus

To retrieve configuration data from a Kubernetes ConfigMap in a Quarkus application, you can utilize the quarkus-kubernetes-config extension, which simplifies the integration of Kubernetes ConfigMaps and Secrets into your application. 
Here’s how you can do it:

Step 1: Add the Extension

First, ensure that you have the quarkus-kubernetes-config extension added to your Quarkus project. You can add it using the following command:

bash

./mvnw quarkus:add-extension -Dextensions="kubernetes-config"

Step 2: Create a ConfigMap

Create a ConfigMap in your Kubernetes cluster that contains the configuration properties you want to use. For example, you can create a ConfigMap named app-config with the following YAML:

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  greeting: "Hello, Quarkus!"
  port: "8080"

Apply this ConfigMap to your Kubernetes cluster:

bash

kubectl apply -f configmap.yaml

Step 3: Configure Quarkus to Use the ConfigMap

In your application.properties file, you can specify that Quarkus should read configuration values from the ConfigMap. For example:

properties

quarkus.kubernetes-config.config-map=app-config

Step 4: Access the Configuration in Your Application

You can access the values stored in the ConfigMap using the @ConfigProperty annotation in your Quarkus application. Here’s an example of how to do this:

java

import io.quarkus.arc.properties.UnlessBuildProfile;
import io.quarkus.runtime.Startup;
import org.eclipse.microprofile.config.inject.ConfigProperty;

import javax.enterprise.context.ApplicationScoped;

@ApplicationScoped
@Startup
public class GreetingService {

    @ConfigProperty(name = "greeting")
    String greeting;

    @ConfigProperty(name = "port")
    int port;

    public String getGreeting() {
        return greeting;
    }

    public int getPort() {
        return port;
    }
}

Conclusion

By following these steps, you can easily access configuration data from a Kubernetes ConfigMap in your Quarkus application. This approach allows you to manage your application's configuration dynamically, making it more adaptable to different environments without changing the codebase. The use of ConfigMaps enhances the flexibility and maintainability of your applications deployed on Kubernetes.