ksqlDB¶
ksqlDB is "the database purpose-built for stream processing applications."
How ksqlDB Works¶
ksqlDB can be used in the following execution modes:
The primary ksqlDB services are KsqlEngine and ServiceContext.
KSQL statements (e.g., DDL commands and queries) can be executed from the following:
- Query file (headless mode)
- ksql (REST mode)
KSQL statements are parsed by AstBuilder.Visitor to Statements.
Executing Queries¶
Query statements are handled by QueryExecutor (in REST mode).
In the end, EngineExecutor is requested to plan a query for execution (that gives an ExecutionStep and an OutputNode).
Executing DDL Commands¶
DdlCommands (e.g. CREATE STREAM) are planned for execution using EngineExecutor and executed by (per command-line options):
DistributingExecutor uses a transactional Kafka producer to enqueue the command (to the CommandQueue) that is then fetched by CommandRunner.
CommandRunner uses InteractiveStatementExecutor to execute commands.
When requested to handle a statement (as part of an enqueued command), InteractiveStatementExecutor uses KsqlEngine to execute a ksql plan.
For DdlCommands, EngineExecutor uses EngineContext to execute it (using DdlCommandExec).
Kafka Streams¶
ksqlDB uses Kafka Streams to build a physical query plan and for query implementation.
LogicalPlanner plans Query statements (after analysis) into an OutputNode (that can build a SchemaKStream that in turn can be requested for an ExecutionStep).
Vert.x¶
ksqlDB API server uses Vert.x for HTTP communication using ServerVerticle.
Libraries Used I Found Interesting¶
Run It Yourself¶
Get standalone ksqlDB first
The following assumes the standalone ksqlDB version installed per ksqlDB Quickstart.
curl http://ksqldb-packages.s3.amazonaws.com/archive/0.27/confluent-ksqldb-0.27.2.tar.gz \
--output confluent-ksqldb-0.27.2.tar.gz
$ cd $KAFKA_HOME
$ ./bin/zookeeper-server-start.sh config/zookeeper.properties
$ ./bin/kafka-server-start.sh config/server.properties
$ ./bin/ksql-server-start etc/ksqldb/ksql-server.properties
===========================================
= _ _ ____ ____ =
= | | _____ __ _| | _ \| __ ) =
= | |/ / __|/ _` | | | | | _ \ =
= | <\__ \ (_| | | |_| | |_) | =
= |_|\_\___/\__, |_|____/|____/ =
= |_| =
= The Database purpose-built =
= for stream processing apps =
===========================================
Copyright 2017-2022 Confluent Inc.
Server 0.27.2 listening on http://0.0.0.0:8088
To access the KSQL CLI, run:
ksql http://0.0.0.0:8088
ksqlDB uses ksql as the command-line interactive environment.
$ ./bin/ksql
===========================================
= _ _ ____ ____ =
= | | _____ __ _| | _ \| __ ) =
= | |/ / __|/ _` | | | | | _ \ =
= | <\__ \ (_| | | |_| | |_) | =
= |_|\_\___/\__, |_|____/|____/ =
= |_| =
= The Database purpose-built =
= for stream processing apps =
===========================================
Copyright 2017-2022 Confluent Inc.
CLI v0.27.2, Server v0.27.2 located at http://localhost:8088
Server Status: RUNNING
Having trouble? Type 'help' (case-insensitive) for a rundown of how things work!
ksql>