Run StreamsFilterTweets.java and TwitterProducerOptimized.java

See result using:
kafka-console-consumer --bootstrap-server localhost:9092 --topic famous-tweets

Tweets that go into a topic named "twitter-tweets", will be filtered with a tweet that is posted by a user who has 10K followers and they will go into a topic named "famous-tweets"

+ Create app.config to load twitter api tokens

Safe + High throughput by using

compression = snappy,
linger ms = 20,
batch size = 32 * 1024,
acks = all,
idempotence = true,
retries = INT32_MAX
max.in.flight = 5

Create a topic named "twitter-tweets" with below command,

kafka-topics --zookeeper localhost:2181 --create --topic twitter-tweets --partitions 6 --replication-factor 1

and run this application.

Next we will build kafka producer to send tweets to kafka.

https://github.com/twitter/hbc

In this example, we read "first-topic", in partition 0, from offset 5 until offset becomes 10.

[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: null, Value: five
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 5
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: null, Value: Hello World!
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 6
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: id_1, Value: Hello 1
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 7
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: id_3, Value: Hello 3
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 8
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: key_1, Value: world
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 9
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Exiting...

+ python config updates for consumers

+ README updates

it reads all datas in three partitions from a topic named "first-topic".

I sent datas in String, like "hello world".

At line 34, we added key parameter.

Whenever we use same key when sending a data, same key always use same partition.

i.e)
i = 0, key = "truck_1" -> partition_0
i = 1, key = "truck_1" -> partition_0
...

for example,

whenever we send a data, this callback function will execute.

새 메타데이터
Topic: first-topic
Partition: 2
Offset: 13
Timestamp: 1606555474565

Send "Hello World!" string to topic named "first-topic".

Producer sends a data to Consumer asynchronous, so we have to flush to see results right away.

Execute below command to see "Hello World!"

>> kafka-console-consumer --bootstrap-server localhost:9092 --topic first-topic --group group-one