[PYTHON] Creating a LINE BOT to notify you of additional AtCoder contests using AWS

Are you turning off notifications for apps other than LINE? I'm cutting. I wanted you to be notified on LINE when the schedule of the AtCoder contest was added, so I created a LINE BOT with Mr JJ.

What I created

We will search for contests scheduled at 00:00 every hour and BOT will notify you when new contests are added. image.png

There is also a function that returns when you send a message. (bonus) image.png

You can add friends to the created LINE BOT by clicking the button below.

Add friend

Backend configuration

LINE_BOT_structure.png All backend processing is done on AWS (Amazon Web Services). The operation on AWS can be divided into the following two.

-[Contest Notification Section](#Contest Notification Section) --Search for AtCoder contests and notify new contests -[LINE Response Section](LINE Response Section) --Respond when a message is sent to the BOT

Describes the features implemented for each section.

Contest notification section

Contest notification section flow

  1. Lambda function: ** AtCoder_contest_Search ** scrapes the AtCoder homepage to get a list of scheduled contests.
  2. ** AtCoder_contest_Search ** gets the notified contest list JSON file from S3.
  3. Calculate the difference set and extract unannounced contests. If the unannounced contest is empty, it ends.
  4. Send unannounced contest data to your Lambda function: ** LINE_contest_notify **.
  5. Notify the data sent ** LINE_contest_notify ** in broadcast format using line-bot-sdk.
  6. Upload the contest data obtained in [1] to S3

Scraping function

Python3


def get_contests():
    #Fetch data from URL
    url = "https://atcoder.jp/home?lang=ja"
    html_data = requests.get(url)

    #html perspective
    soup = BeautifulSoup(html_data.text, "html.parser")
    
    tags = ["upcoming", "recent"]
    ret_dic = {}
    for tag in tags:
        div = soup.find("div", id="contest-table-"+tag)
        table = div.find("table")
        times  = table.find_all("time")
        titles = table.find_all("a")

        ret_list = []
        for i in range(len(times)):
            dic = {}
            dic["start"] = times[i].text
            dic["title"] = titles[i*2+1].text
            dic["url"] = "https://atcoder.jp" + titles[i*2+1].get("href")
            ret_list.append(dic)
        ret_dic[tag] = ret_list
    return ret_dic
Data content
argument null
Return value Data for each contest(Start time / title / URL)List of dictionaries with

Only * ret_dic ["upcoming"] * is used in this BOT.

Get files from S3

Python3


def get_data_s3():
    #Notified contest json file on S3"contests.json"Loading
    
    client = boto3.client("s3")
    try:
        response = client.get_object(Bucket = "bucket_name", Key = "json_file_name")
    except Exception as e:
        print(e)
    return json.loads(response["Body"].read().decode())
Data content
argument null
Return value Dictionary data read JSON file on S3

The return value of the * get_object * method is a dictionary, and the key "Body" contains the contents of the file.

File upload to S3

Python3


def put_data_s3(save_dic):
    #New notified contest json file S3 uploader
    
    text = json.dumps(save_dic, ensure_ascii = False)
    s3 = boto3.resource("s3")
    s3.Object("bucket_name", "json_file_name").put(Body = text)
Data content
argument Dictionary data to be saved
Return value null

When converting the dictionary to JSON format text, if the value of * ensure_ascii * is set to False, The Japanese in the JSON file becomes easier to read and the file size can be reduced.

Broadcast message transmission

Python3


def DatetimeToString(date):
    if not isinstance(date, datetime.datetime):
        date = datetime.datetime.strptime(date, "%Y-%m-%d %H:%M:%S%z")
    week_list = "Monday, Tuesday, Wednesday, Thursday, Friday, Saturday and Sunday"
    return f"{date.strftime('%m/%d')}({week_list[date.weekday()]}){date.strftime('%H:%M')}"

def MakeMessageText(event):
    text = DatetimeToString(event["start"]) + "\n" + event["title"] + "is held!\n" + event["url"]
    return text

def main(event, context):    
    line_bot_api = LineBotApi("channel_access_token")
    message = MakeMessageText(event)
    line_bot_api.broadcast(TextSendMessage(text=message))

    # ***Processing to finish the main function***

Create a message format for the contest start time with the * DatetimeToString * function Created the format of the message to be notified by the * MakeMessageText * function.

You can notify all your BOT friends by simply calling the * LineBotApi * class * broadcast * method of line-bot-sdk. ..

LINE response section

Configuration diagram of this system ![LINE_BOT_structure.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/308399/d98a570f-3c81-a5e4-b71e-8ac8182c1a31.png)

LINE response section flow

  1. When the LINE BOT receives the LINE message, it uses the Webhook to POST the message content to the API Gateway.
  2. POST to API Gateway triggers Lambda function: ** LINE_reply_message ** to be executed.
  3. ** LINE_reply_message ** sends a LINE reply message using line-bot-sdk.

Send reply message

Python3


def main(event, context):
    handler = WebhookHandler("channel_secret")
    line_bot_api = LineBotApi("channel_access_token")
    body = event["body"]

    #Added reply function to handler
    @handler.add(MessageEvent)
    def SendReply(line_event):
        message = MakeReplyMessage(line_event.message.text)
        line_bot_api.reply_message(line_event.reply_token, TextSendMessage(text=message))

    try:
        handler.handle(body, event["headers"]["X-Line-Signature"])
    except #Error handling

    # ***Processing to finish the main function***

The * MakeReplyMessage * function returns a reply message as a character string according to the message content received by the BOT. We especially recommend reading the line-bot-sdk documentation for this part.

At the end

We introduced the main functions we created. Line-bot-sdk is very convenient because you can easily operate BOT. I have little knowledge about AWS and the design is quite rough, but if I feel like it, I will add and improve functions in the future. (maybe)

Recommended Posts

Creating a LINE BOT to notify you of additional AtCoder contests using AWS
Created a Discord bot to notify you of updates to become a novelist
Procedure for creating a Line Bot on AWS Lambda
[End of 2020] A memo to start using AWS CLI (Version 2)
Creating a LINE bot ~ Creating, deploying, and launching ~
I made a program to notify you by LINE when switches arrive
Allow Slack to notify you of the end of a time-consuming program process
Use AWS lambda to scrape the news and notify LINE of updates on a regular basis [python]
The story of creating a store search BOT (AI LINE BOT) for Go To EAT in Chiba Prefecture (2) [Overview]
Send a message to LINE with Python (LINE Notify)
Use AWS Lambda + LINE Notify to notify LINE not to forget your umbrella when you get home
The world's most easy-to-understand explanation of how to make a LINE BOT (1) [Account preparation]
Make a parrot return LINE Bot on AWS Cloud9
[Python] Using Line API [1st Creation of Beauty Bot]
LINE Bot that notifies you of the stocks of interest
Play to notify Slack of environmental data using AWS PaaS from SensorTag via Raspberry Pi3
How to make an interactive LINE BOT 004 (answer the closing date of a listed company)
[LINE Notify API, AWS] Regularly send buzzing tweets to group LINE
[Introduction to AWS] A memorandum of building a web server on AWS
I want to send a message from Python to LINE Bot
A quick explanation from creating AWS Lambda Layers to linking
The story of creating a database using the Google Analytics API
Let's make a LINE bot using various services [ngrok edition]
[AWS] I made a reminder BOT with LINE WORKS (implementation)
[Python] Let LINE notify you of the ranking of search results on your site on a daily basis.