본문 바로가기

리눅스

카프카 producer와 consumer 테스트

728x90

카프카 producer와 consumer 테스트

 

이미지 출처-https://www.tutorialspoint.com/apache_kafka/images/cluster_architecture.jpg

 

이미지 출처-https://www.researchgate.net/profile/Asad-Javed-9/publication/338516815/figure/fig2/AS:846484845125632@1578829122361/Example-of-Kafka-cluster-with-three-brokers.png

카프카 토픽 생성

  • topic list
kafka-topics.sh --list --bootstrap-server localhost:9092
  • topic create
    • replication-factor : 복제본 개수(2)
    • partitions : 파티션 개수(3)
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 2 --partitions 3 --topic helloworld
$ kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 2 --partitions 3 --topic helloworld
Created topic helloworld.
  • topic list
kafka-topics.sh --list --bootstrap-server localhost:9092
$ kafka-topics.sh --list --bootstrap-server localhost:9092
helloworld
  • topic delete
kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic helloworld
$ kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic helloworld
728x90

프로듀서(Producer), 컨슈머(Consumer) 실행하기

  • 프로듀서 실행 / 메시지 Write
kafka-console-producer.sh --broker-list localhost:9092 --topic helloworld
  • 컨슈머(Consumer) 실행 / 메시지 Read
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic helloworld

--from-beginning 옵션을 추가하면 토픽에 있는 모든 레코드를 확인 할수 있다.

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic helloworld --from-beginning
  • 컨슈머 그룹(Consumer Group) list
kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
$ kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
console-consumer-58810
  • 컨슈머(Consumer) status and Offset 확인
kafka-consumer-groups.sh --describe --group console-consumer-58810 --bootstrap-server localhost:9092
$ kafka-consumer-groups.sh --describe --group console-consumer-58810 --bootstrap-server localhost:9092

GROUP                  TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                            HOST            CLIENT-ID
console-consumer-58810 helloworld      0          -               1               -               consumer-console-consumer-58810-1-034dcce6-8cc4-46d9-a09d-dfcf32469460 /192.168.0.103   consumer-console-consumer-58810-1
  • 컨슈머 그룹(Consumer Group) delete
kafka-consumer-groups.sh --delete --group console-consumer-58810 --bootstrap-server localhost:9092
$ kafka-consumer-groups.sh --delete --group console-consumer-58810 --bootstrap-server localhost:9092
Deletion of requested consumer groups ('console-consumer-58810') was successful.

 

참고URL

- Kafka 문서 : https://kafka.apache.org/documentation/#introduction

- Kafka 시스템 구조 알아보기 : https://velog.io/@jwpark06/Kafka-%EC%8B%9C%EC%8A%A4%ED%85%9C-%EA%B5%AC%EC%A1%B0-%EC%95%8C%EC%95%84%EB%B3%B4%EA%B8%B0

 

728x90