[PYTHON] I want to make an SFC physical education reservation system monitoring system (β version)

Introduction

This article was written for Keio University SFC students. We will try to explain the technology in an easy-to-understand manner, but please note that the explanation of the physical education system itself is omitted.

`In addition, we are not responsible for any damage caused by the program created by referring to this article, so please create a similar system at your own risk. ``

Since it is a beta version, I am posting it with infinite improvements.

Purpose

I want to make a quick reservation when there is a vacancy in the class I want to take with the SFC physical education reservation system.

Read the terms of service carefully

There is no page in the physical education system that describes the terms of use, but [Keio University Site Terms](https://www.students.keio.ac.jp/com/other/contact/about-this-site. Note the following with reference to html). ** ・ Do not disclose information on the site to the outside public ** ** ・ You are free to link to the PE reservation system, but if you include the link in a site element such as an image, specify it. ** ** ・ Excessive access is subject to BAN, so request within the bounds of common sense **

environment

・ MacOS Catalina 10.15.5 ・ Python 3.8.0

Library used (link to official document)

requests A library for transferring data between servers. It's good to understand that the curl command is easy to use

bs4 python html parser

lxml Introduced because Xpath cannot be used with HTML, XML parser for python, and bs4 alone. Not required if you do not use Xpath

If you don't understand how to use Xpath, refer to this article → Summary of how to use Xpath

pyyaml A library for reading confidential information because it was saved in yaml format.

time Used to control the access interval.

• All of the above libraries can be installed with pip, so please install the uninstalled libraries with `pip ~` *

Other

LINE Notify: A convenient official API that allows you to send notifications to your LINE account

Preparing for LINE Notify

https://notify-bot.line.me/ja/ Log in to your LINE account from the above site and click the toggle on the upper right to go to My Page There is a button at the bottom of the screen as shown below, so click on it to get an access token.

Source code

file organization

・ Main.py (executable file) ・ Secrets.yaml (file that collects confidential information)

main.py

import requests
import yaml
from lxml import html
from bs4 import BeautifulSoup
import time

#Since insecureRequestWarning is noisy, delete it
import urllib3
from urllib3.exceptions import InsecureRequestWarning
urllib3.disable_warnings(InsecureRequestWarning)

#Read confidential information from another file
SECREATS = yaml.load(open('secrets.yaml').read(),Loader=yaml.SafeLoader)

def checkTargetClass(targetClass: str):
    '''
Log in and return the reservation link for the specified lesson if there is space available
    '''

    PE_SYSTEM_URL = 'https://wellness.sfc.keio.ac.jp/v3/' 

    #Start session
    session = requests.session()

    #Login
    login_info = {
        'login':SECREATS['user'],
        'password':SECREATS['pass'],
        'submit':'login',
        'page':'top',
        'lang':'ja',
        'mode':'login',
        'semester':'20205'
    }
    
    #Login execution
    res = session.post(PE_SYSTEM_URL, data=login_info, verify=False)
    res.raise_for_status() #If there is an error, raise an exception here

    #Parse in a format that can be searched by Xpath
    soup_parsed_data = BeautifulSoup(res.text, 'html.parser')
    lxml_coverted_data = html.fromstring(str(soup_parsed_data))

    #Reservation page link detection&Move
    reservePageTag = lxml_coverted_data.xpath('//*[@id="sidenav"]/ul/li[4]/a')
    reservePageLink = reservePageTag[0].get('href')
    res = session.get(PE_SYSTEM_URL+reservePageLink)
    res.raise_for_status()

    #Perth
    soup_parsed_data = BeautifulSoup(res.text, 'html.parser')
    lxml_coverted_data = html.fromstring(str(soup_parsed_data))

    #Get a list of classes that can be reserved and a link to the reservation confirmation page
    targetElemChunk = lxml_coverted_data.xpath('//*[@id="maincontents"]/div[6]/table/tbody/tr')
    classInfoList = []
    for classInfoRow in targetElemChunk:
        className = classInfoRow[7].text
        reseveConfirmPageLink = classInfoRow[9].find('a').get('href')
        if targetClass in className:
            notify(className+PE_SYSTEM_URL+reseveConfirmPageLink)
        classInfo = [className, reseveConfirmPageLink]
        classInfoList.append(classInfo)

def watch():
    while True:
        checkTargetClass('Dietary science')
        time.sleep(1800)
        

#Send notification
def notify(message):
    LINE_NOTIFY_URL = 'https://notify-api.line.me/api/notify'
    TOKEN = SECREATS['notify-token']
    headers = {'Authorization': f'Bearer {TOKEN}'}
    data = {'message': f'message: {message}'}
    requests.post(LINE_NOTIFY_URL, headers = headers, data = data)

def main():
    watch()

if __name__ == "__main__":
    main()

secrets.yaml

user: 'Login id'
pass: 'password'
notify-token: 'Token issued from LINE API'

How to use

Run python main.py from the command line and leave it forever

Improvement points

・ I want to move files to the server because it is inconvenient to leave them permanently. ・ I want to make it a web application and operate it with GUI ・ I set the access frequency every 1800 seconds, but it may be possible to shorten the interval a little. ・ ↑ Or if you can identify the timing when cancellations occur frequently, you may be able to endure it by accessing frequently only during that time.