Skip to content
Snippets Groups Projects
  1. Dec 18, 2020
  2. Dec 17, 2020
  3. Dec 16, 2020
    • Seok Won's avatar
      Moved to Hyper-V Ubuntu kafka server · 1decb7e2
      Seok Won authored
      After spending 4hours, managed to make it to work right.
      
      I made, inner hyper-v nat, outer hyper-v nat
      
      and outer nat shares with inner nat.
      
      Setting advertised.listener to EXTERNAL://ubuntuIP:9093, I could finally get this server working.
      Unverified
      1decb7e2
  4. Dec 15, 2020
  5. Dec 14, 2020
  6. Dec 12, 2020
  7. Dec 11, 2020
    • Seok Won's avatar
      Update Ajou Notice Parser · ec2cff9a
      Seok Won authored
      Added a test which contains filtering function (it'll filter notices within the array of strings)
      
      Made producer more safe and highthroughput.
      
      Added old notices remover in producer.
      
      Added a little bit of resting time between consuer polling.
      Unverified
      ec2cff9a
  8. Dec 10, 2020
  9. Dec 09, 2020
  10. Dec 07, 2020
  11. Dec 06, 2020
    • Seok Won's avatar
      Create a Faust basic example · 8ef04d59
      Seok Won authored
      pip install faust
      8ef04d59
    • Seok Won's avatar
      Create pytest tests · ab8724c7
      Seok Won authored
      ab8724c7
    • defaultSouth's avatar
      Update README.md · 7e8b02de
      defaultSouth authored
      Unverified
      7e8b02de
    • Seok Won's avatar
      Slack + Kafka: 아주대학교 공지 봇 · df21baf7
      Seok Won authored
      1시간마다 모든 공지를 불러 {"TITLE": "제목", "DATE": "올린 날", "LINK": "http 주소", "WRITER": "글쓴이"}를 json 형태로 저장한다.
      
      이 json 파일이나 새로운 공지가 있으면 기존 json과 비교해서 새로운 데이터를 Consumer로 보내고, Consumer는 새로운 데이터를 받으면, Slack API를 이용해, "#아주대" 채널에 공지를 올려준다.
      
      마지막 파싱 시간도 기록해 종료 후 다시 불러도 1시간이 지나지 않으면 파싱하지 않는다.
      
      결과)
      Last parsing: 1972-12-01 07:00:00
      Trying to parse new posts...
      Sending a new post...: 12179
      ...
      
      Last parsing: 2020-12-04 19:11:42.839219
      Trying to parse new posts...
      No new posts yet...
      Resting 1 hour...
      ...
      Last parsing: 2020-12-06 11:55:35.386262
      Wait for 3494 seconds to sync new posts.
      df21baf7
  12. Dec 03, 2020
  13. Dec 01, 2020
  14. Nov 30, 2020
    • Seok Won's avatar
      Update TwitterProducer to send data to kafka · 9b667d5a
      Seok Won authored
      Create a topic named "twitter-tweets" with below command,
      
      kafka-topics --zookeeper localhost:2181 --create --topic twitter-tweets --partitions 6 --replication-factor 1
      
      and run this application.
      9b667d5a
    • Seok Won's avatar
      Create TwitterProducer with just TwitterAPI · cebbd832
      Seok Won authored
      Next we will build kafka producer to send tweets to kafka.
      
      https://github.com/twitter/hbc
      cebbd832
    • Seok Won's avatar
      Create Consumer with Assign&Seek · 768e01ea
      Seok Won authored
      In this example, we read "first-topic", in partition 0, from offset 5 until offset becomes 10.
      
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: null, Value: five
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 5
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: null, Value: Hello World!
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 6
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: id_1, Value: Hello 1
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 7
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: id_3, Value: Hello 3
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 8
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Key: key_1, Value: world
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Partition:  0, Offset: 9
      [main] INFO csw.kafka.study.lesson2.ConsumerDemoAssignSeek - Exiting...
      768e01ea
    • Seok Won's avatar
      Create Consumer with a thread · 047cbae2
      Seok Won authored
      + python config updates for consumers
      
      + README updates
      047cbae2
    • Seok Won's avatar
      Create Simple Producer in Python · 3886280c
      Seok Won authored
      3886280c
  15. Nov 29, 2020
    • Seok Won's avatar
      Create Simple Consumer · c998f7bf
      Seok Won authored
      it reads all datas in three partitions from a topic named "first-topic".
      
      I sent datas in String, like "hello world".
      c998f7bf
  16. Nov 28, 2020
Loading