[PYTHON] I tried to make Kana's handwriting recognition Part 3/3 Cooperation with GUI using Tkinter

Overview

Two times before (1/3): https://qiita.com/tfull_tf/items/6015bee4af7d48176736 Last time (2/3): https://qiita.com/tfull_tf/items/968bdb8f24f80d57617e

Whole code: https://github.com/tfull/character_recognition

Since the kana recognition model has reached the level where the operation of the system can be confirmed, we will generate an image with the GUI, load it into the model, and output characters.

Create a palette for kana input with Tkinter

I will use Tkinter, so install it.

# Mac +For Homebrew
$ brew install tcl-tk
#For Ubuntu Linux
$ sudo apt install python-tk

Create a canvas and left-click to draw a white line and right-click to draw a black line (a black background, so it's a real eraser). Two buttons are prepared, one for recognizing the written characters and the other for erasing (filling in black) the written ones.

import tkinter
# from PIL import Image, ImageDraw

class Board:
    def __init__(self):
        self.image_size = 256

        self.window = tkinter.Tk()
        self.window.title("Kana input")

        self.frame = tkinter.Frame(self.window, width = self.image_size + 2, height = self.image_size + 40)
        self.frame.pack()

        self.canvas = tkinter.Canvas(self.frame, bg = "black", width = self.image_size, height = self.image_size)
        self.canvas.place(x = 0, y = 0)

        self.canvas.bind("<ButtonPress-1>", self.click_left)
        self.canvas.bind("<B1-Motion>", self.drag_left)
        self.canvas.bind("<ButtonPress-3>", self.click_right)
        self.canvas.bind("<B3-Motion>", self.drag_right)

        self.button_detect = tkinter.Button(self.frame, bg = "blue", fg = "white", text = "recognition", width = 100, height = 40, command = self.press_detect)
        self.button_detect.place(x = 0, y = self.image_size)

        self.button_delete = tkinter.Button(self.frame, bg = "green", fg = "white", text = "Delete", width = 100, height = 40, command = self.press_delete)
        self.button_delete.place(x = self.image_size // 2, y = self.image_size)

        # self.image = Image.new("L", (self.image_size, self.image_size))
        # self.draw = ImageDraw.Draw(self.image)

    def press_detect(self):
        output = recognize(np.array(self.image).reshape(1, 1, self.image_size, self.image_size)) #recognize is a function that recognizes using machine learning
        sys.stdout.write(output)
        sys.stdout.flush()

    def press_delete(self):
        # self.canvas.delete("all")
        # self.draw.rectangle((0, 0, self.image_size, self.image_size), fill = 0)

    def click_left(self, event):
        ex = event.x
        ey = event.y

        self.canvas.create_oval(
            ex, ey, ex, ey,
            outline = "white",
            width = 8
        )

        # self.draw.ellipse((ex - 4, ey - 4, ex + 4, ey + 4), fill = 255)

        self.x = ex
        self.y = ey

    def drag_left(self, event):
        ex = event.x
        ey = event.y

        self.canvas.create_line(
            self.x, self.y, ex, ey,
            fill = "white",
            width = 8
        )

        # self.draw.line((self.x, self.y, ex, ey), fill = 255, width = 8)

        self.x = ex
        self.y = ey

    def click_right(self, event):
        ex = event.x
        ey = event.y

        self.canvas.create_oval(
            ex, ey, ex, ey,
            outline = "black",
            width = 8
        )

        # self.draw.ellipse((ex - 4, ey - 4, ex + 4, ey + 4), fill = 0)

        self.x = event.x
        self.y = event.y

    def drag_right(self, event):
        ex = event.x
        ey = event.y

        self.canvas.create_line(
            self.x, self.y, ex, ey,
            fill = "black",
            width = 8
        )

        # self.draw.line((self.x, self.y, ex, ey), fill = 0, width = 8)

        self.x = event.x
        self.y = event.y

With this, you can write and erase characters.

Use a pillow to follow inside

After some research, Tkinter's Canvas can draw lines, so it can be written, but it seems that it is not possible to read what you wrote and quantify it.

As I was looking for a solution, I found the argument that I should have the Pillow internally and draw the same on the Pillow's Image when drawing on the Canvas. (I couldn't think of that, so I thought it was a very smart idea.) The commented out part of the code above corresponds to that. Pillow's Image can be quickly turned into an array of numpy with numpy.array, making it easy to give to your machine learning model.

Completion of kana recognition system

We have completed a GUI for entering handwritten characters and a machine learning model for classifying kana images. The combination of these is next.

cr202011.gif

YouTube: https://www.youtube.com/watch?v=a0MBfVVp7mA

Impressions and consideration of operation check

It's not very accurate, but it recognized it fairly correctly. It doesn't recognize dirty characters, but when I typed "aiueo" carefully, it output the corresponding characters properly.

I tried the "Hello", if "blood" is recognized in the "La" and "filtration" was frequent. When I checked the image data of "Chi" because I was curious, I found that there was a slight difference between the shape of "Chi" I wrote and that of the image. Since I learned only 3 types of fonts, it seems that I can't handle the input with strange shapes. When I wrote the characters in a form close to the input data, they recognized it correctly.

The recognition time outputs the result in an instant without using the GPU for the machine learning model.

It would be ideal if we could recognize kanji, but the classification destinations would be 169 to thousands. Kana alone is not accurate enough, so I feel that performance is likely to drop at once.

It will take some time, but I think that the convenience as an interface will increase if you output the top n items with a high probability of classification.

After that, if you learn the certainty as a word and recognize it by the shape of the character + the likelihood as a word, the accuracy when entering a meaningful word or sentence may increase. (For example, when you enter "hello", add that "ha" is about to come next.)

Recommended Posts

I tried to make Kana's handwriting recognition Part 3/3 Cooperation with GUI using Tkinter
I tried to make Kana's handwriting recognition Part 1/3 First from MNIST
I tried to make Kana's handwriting recognition Part 2/3 Data creation and learning
I tried to make GUI tic-tac-toe with Python and Tkinter
I tried to make a stopwatch using tkinter in python
I tried handwriting recognition of runes with CNN using Keras
I tried to make a todo application using bottle with python
I tried to make an open / close sensor (Twitter cooperation) with TWE-Lite-2525A
I tried to make a calculator with Tkinter so I will write it
Upload images to S3 with GUI using tkinter
I tried to make a ○ ✕ game using TensorFlow
I tried to make a simple image recognition API with Fast API and Tensorflow
I tried to make an OCR application with PySimpleGUI
I tried to make various "dummy data" with Python faker
I tried to make a simple text editor using PyQt
[Python] I tried to make an application that calculates salary according to working hours with tkinter
I tried to make a motion detection surveillance camera with OpenCV using a WEB camera with Raspberry Pi
I tried face recognition with OpenCV
[5th] I tried to make a certain authenticator-like tool with python
[2nd] I tried to make a certain authenticator-like tool with python
I tried to make deep learning scalable with Spark × Keras × Docker
I tried to make a regular expression of "amount" using Python
I tried to make a regular expression of "date" using Python
I tried to make a periodical process with Selenium and Python
I tried to make a 2channel post notification application with Python
I tried using PyEZ and JSNAPy. Part 4: Automate ISP setup with PyEZ and JSNAPy
I tried to display GUI on Mac with X Window System
[4th] I tried to make a certain authenticator-like tool with python
[1st] I tried to make a certain authenticator-like tool with python
I tried to make a strange quote for Jojo with LSTM
I tried to make an image similarity function with Python + OpenCV
I tried to make a mechanism of exclusive control with Go
Python: I tried to make a flat / flat_map just right with a generator
I tried using Amazon SQS with django-celery
I tried using Azure Speech to Text.
GUI creation in python using tkinter part 1
I tried to implement Autoencoder with TensorFlow
I tried to get started with Hy
Make GUI apps super easy with tkinter
I tried using Selenium with Headless chrome
I tried using PyEZ and JSNAPy. Part 2: I tried using PyEZ
I tried to make "Sakurai-san" a LINE BOT with API Gateway + Lambda
I tried to classify text using TensorFlow
I tried to make a traffic light-like with Raspberry Pi 4 (Python edition)
I tried to convert datetime <-> string with tzinfo using strftime () and strptime ()
I tried implementing DeepPose with PyTorch PartⅡ
I tried to implement CVAE with PyTorch
I tried to make a Web API
I tried to solve TSP with QAOA
I tried simple image recognition with Jupyter
[Zaif] I tried to make it easy to trade virtual currencies with Python
I tried to make a url shortening service serverless with AWS CDK
I tried to make PyTorch model API in Azure environment using TorchServe
I tried to predict Covid-19 using Darts
I tried image recognition of "Moon and Soft-shelled Turtle" with Pytorch (using torchvision.datasets.ImageFolder which corresponds to from_from_directry of keras)
I tried to make a periodical process with CentOS7, Selenium, Python and Chrome
I tried to use Java with Termux using Termux Arch but it didn't work
When I tried to make a VPC with AWS CDK but couldn't make it
I tried to make a castle search API with Elasticsearch + Sudachi + Go + echo
I tried to make a translation BOT that works on Discord using googletrans
I tried to create an environment to check regularly using Selenium with AWS Fargate