How to Install Apache Flink On Local Windows

Requirements to Install Apache Flink on Windows

Installing and running Apache Flink on a local Windows machine requires the following things.

Windows 10

We need Windows 10 with the Windows Subsystem for Linux (WSL). But if your version of Flink binary has .bat files, then you don’t need these and can use another Windows version and run with cmd. But the version I am using only has the .sh files.

Java

The second requirement is Java. Java version 8 or 11 is recommended. It doesn’t matter if you use OpenJDK or Oracle-provided Java on the machine.

Verify your java with java -version. Also, make sure you set up your Environment Variable JAVA_HOME otherwise, the Flink won’t create any Job.

Download Latest Apache Flink on Local Windows

Now that the requirement is fulfilled, let’s jump to the next part. 

As I said earlier I will be installing Apache Flink version 1.20.0 for this post. 

To get the list of all Flink binaries available to download, head over to this link: Downloads | Apache Flink. 

Select the version you want to choose and download. You may be redirected to another page and in the 1st paragraph, you can see the mirror download link.

I will be selecting Flink Version 1.20. 0for download. and then select the -bin-scala_ file.

Start and Stop Apache Flink Cluster on Local Windows

Now that we have downloaded the Apache Flink binaries, let’s extract the tar file and copy it to a more appropriate location, any location would be fine, I will be putting it on c:\apache-flink\flink-1.20.0. 

But in WSL the path will be, /mnt/c/apache-flink/flink-1.20.0.

In the terminal, WSL, run the cluster with the following command.

./bin/start-cluster.sh

If it is successful then you will see a cluster start to message and the cluster will run in the background.

Then simply go to the localhost:8081 to access the Flink UI Dashboard.

To stop the cluster use the following command.

./bin/stop-cluster.sh

Run a Flink Job on Cluster

There are 2 ways you can run a Flink Job on the cluster. UI and via command.

Run Flink Job via Flink Dashboard

The simplest way is to use the UI. First, in the dashboard, go to the Submit New Job page and upload the Jar. Next, select the Jar from the list, and pass the Main class, the argument of the program. Then, submit the job and it will redirect you to the running job page, where you will see the pipelines and their status.

Run a Flink Job via Terminal

The second way is to run it via the Terminal or Command-Line. You can use the following command to run the Flink Job.

./bin/flink run     --detached     /path/to/jar/SampleFlinkJob.jar

This will run the Flink job in the background. You can then check the status in the Flink Dashboard, on localhost, if it runs successfully.

Understanding Flink’s key concepts

 

Understanding Flink’s key concepts:

• DataStream API: Flink's main tool for creating stream processing applications, providing operations to transform data streams.
• Windows: Defines a finite set of stream events for computations, based on count, time, or sessions.
• Transformations: Operations applied to data streams to produce new streams, including map, filter, flatMap, keyBy, reduce, aggregate, and window.
• Sinks: The endpoints of Flink applications where processed data ends up, such as a file, database, or message queue.
• Sources: The starting points of Flink applications that ingest data from external systems or generate data internally, such as a file or Kafka topic.
• Event Time vs. Processing Time: Flink supports different notions of time in stream processing. Event time is the time when an event occurred, while processing time is the time when the event is processed by the system. Flink excels at event time processing, which is crucial for correct results in many scenarios.
• CEP (Complex Event Processing): Flink supports CEP, which is the ability to detect patterns and complex conditions across multiple streams of events.
• Table API & SQL: Flink offers a Table API and SQL interface for batch and stream processing. This allows users to write complex data processing applications using a SQL-like expression language.
• Stateful Functions (StateFun): StateFun is a framework by Apache Flink designed to build distributed, stateful applications. It provides a way to define, manage, and interact with a dynamically evolving distributed state of functions.
• Operator Chain and Task: Flink operators (transformations) can be chained together into a task for efficient execution. This reduces the overhead of thread-to-thread handover and buffering.
• Savepoints: Savepoints are similar to checkpoints, but they are triggered manually and provide a way to version and manage the state of Flink applications. They are used for planned maintenance and application upgrades.
• State Management: Flink provides fault-tolerant state management, meaning it can keep track of the state of an application (e.g., the last processed event) and recover it if a failure occurs.
• Watermarks: These are a mechanism to denote progress in event time. Flink uses watermarks to handle late events in stream processing, ensuring the system can handle out-of-order events and provide accurate results.
• Checkpoints: Checkpoints are a snapshot of the state of a Flink application at a particular point in time. They provide fault tolerance by allowing an application to revert to a previous state in case of failures.