Kafka CLI

Apache Avro with fastavro

AjouSlackProducerAvro.py will produce a single record with schema

AjouSlackConsumerAvro.py will consume it with given schema

requests cause SSL Error.

For now, we change it to urlopen with unverifed context

Run StreamsFilterTweets.java and TwitterProducerOptimized.java

See result using:
kafka-console-consumer --bootstrap-server localhost:9092 --topic famous-tweets

Tweets that go into a topic named "twitter-tweets", will be filtered with a tweet that is posted by a user who has 10K followers and they will go into a topic named "famous-tweets"

read READM.md

using classes and optimizations.

more human readable

with class and optimized

After spending 4hours, managed to make it to work right.

I made, inner hyper-v nat, outer hyper-v nat

and outer nat shares with inner nat.

Setting advertised.listener to EXTERNAL://ubuntuIP:9093, I could finally get this server working.

HTMLParser + Slack API + MySQL in python

First, create a database with a table named "notices".

CREATE TABLE notices (
  id INT,
  title VARCHAR(100),
  date VARCHAR(30),
  link VARCHAR(101),
  writer VARCHAR(25)
)

go get -u "github.com/segmentio/kafka-go"

Added a test which contains filtering function (it'll filter notices within the array of strings)

Made producer more safe and highthroughput.

Added old notices remover in producer.

Added a little bit of resting time between consuer polling.

+ Create app.config to load twitter api tokens

Safe + High throughput by using

compression = snappy,
linger ms = 20,
batch size = 32 * 1024,
acks = all,
idempotence = true,
retries = INT32_MAX
max.in.flight = 5

Before: some notices have their name in notices

[writer]: [writer] title

After: remove writer in titles

[writer]: title

pip install faust

1시간마다 모든 공지를 불러 {"TITLE": "제목", "DATE": "올린 날", "LINK": "http 주소", "WRITER": "글쓴이"}를 json 형태로 저장한다.

이 json 파일이나 새로운 공지가 있으면 기존 json과 비교해서 새로운 데이터를 Consumer로 보내고, Consumer는 새로운 데이터를 받으면, Slack API를 이용해, "#아주대" 채널에 공지를 올려준다.

마지막 파싱 시간도 기록해 종료 후 다시 불러도 1시간이 지나지 않으면 파싱하지 않는다.

결과)
Last parsing: 1972-12-01 07:00:00
Trying to parse new posts...
Sending a new post...: 12179
...

Last parsing: 2020-12-04 19:11:42.839219
Trying to parse new posts...
No new posts yet...
Resting 1 hour...
...
Last parsing: 2020-12-06 11:55:35.386262
Wait for 3494 seconds to sync new posts.

this producer reads all messages from "#general", in every 5 seconds (this will cause ratelimited)

and if messages contain a word "bug", it will automatically sends "USERNAME" and "MESSAGE" to Kafka consumer.

And the consumer will leave a message in "#kafka" channel saying "USER found a bug ..."

modified version of an official confluent example.

topic id: SLACK-KAFKA

this will automatically posts a message to your specified channel in Slack, if someone leaves a bad review

(for now, you can just send a json data to test from CLI, see the README file.)

Create a topic named "twitter-tweets" with below command,

kafka-topics --zookeeper localhost:2181 --create --topic twitter-tweets --partitions 6 --replication-factor 1

and run this application.

Next we will build kafka producer to send tweets to kafka.

https://github.com/twitter/hbc

In this example, we read "first-topic", in partition 0, from offset 5 until offset becomes 10.

[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: null, Value: five
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 5
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: null, Value: Hello World!
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 6
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: id_1, Value: Hello 1
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 7
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: id_3, Value: Hello 3
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 8
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: key_1, Value: world
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 9
[main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Exiting...

+ python config updates for consumers

+ README updates

it reads all datas in three partitions from a topic named "first-topic".

I sent datas in String, like "hello world".