cnhaa.blogg.se - Using kafka tool create dc1 to dc2

This will cause a neverending replication loop between both Kafka clusters! Sep 21 11:28:37 dc2_kafka_workernode_01 systemd: Starting Apache Kafka - ~]$ĭo NOT start MirrorMaker on BOTH DCs. Sep 21 11:28:37 dc2_kafka_workernode_01 systemd: Started Apache Kafka - MirrorMaker. └─14189 java -Xmx256M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent =true .authenticate=false .ssl=false =/var/log/kafka nfiguration=file:/etc/kafka/tools-log4j.properties -cp /bin/./share/java/kafka/*:/bin/./share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/* nfig /etc/kafka/consumer_dc1_to_dc2.properties nfig /etc/kafka/producer_dc1_to_dc2.properties -num.streams 3 -abort.on.send.failure true -whitelist test test2 test3 test4 test5 Loaded: loaded (/usr/lib/systemd/system/rvice static vendor preset: disabled)Īctive: active (running) since Fri 11:28:37 UTC 2min 52s agoĬGroup: /system.slice/rvice Regarding port numbers that Kafka ~]$ sudo systemctl status confluent-mirrormaker -l

It will consume from DC1 and act as a producer for DC2. In this case the process will be started on DC2. Will provision a MirrorMaker instance on the TARGET DC.

Will only (re-) provision VMs belonging to either dc1 or dc2. After spinning up the VMs for a the DCs Ansible will be automatically called to provision the VMs. The most common options have been described below: This Makefile contains several options to control the environment. In the root of the project you will find a Makefile.

Kafka Schema Registry (and Schema registry API on port 8081)ġx MySQL node for testing MySQL source Connectorġx single node Elasticsearch for testing Elasticsearch sink Connector.

Kafka REST proxy (and API on port 8082).

Kafka Connect (and Connect REST API on port 8083).

The complete setup consists of 2 'data centers' to be able to test things like MirrorMaker and Replicator.Ī single DC consists of the following VMs:ģx Kafka brokers running the following components:Ģx Kafka worker nodes running the following components: Vagrant set of VMs to run Confluent Kafka on Oracle Linux.