[PYTHON] Let's play with Amedas data-Part 1

As a study of neural networks, I would like to handle some kind of data. Therefore, I decided to play with the data of AMeDAS, which seems to be easily available. It seems that the data can be downloaded from the following URL.

https://www.jma.go.jp/jma/menu/menureport.html

Meteorology → Download past weather data

So, it seems that you can get the appropriate data at the appropriate point. It seems that the amount of data that can be downloaded at one time is limited, so set it appropriately and download it. This time, I used the data from Matsumoto City, Nagano Prefecture, which I often jog.

matsu.jpg

The selected item is ... ■ Temperature ■ Wind direction / speed ■ Weather Three of. After selecting the appropriate period and clicking the "Download CSV file" button, the download was successful.

When I open it, it looks like a strange format. matsu_data.jpg

The header is included, but it may be a little broken. In order to analyze with Tensorflow, it is necessary to import data into Python, but this time I will use Pandas. Then, it seems that there are restrictions on the CSV file. Probably ...

Is it around? I can't use it as it is, and since the wind direction is Japanese, I decided to convert it to a numerical value.

I feel that it is the royal road to use Script such as Perl for this kind of thing, but since the environment I am using is Windows, I will write a program easily with VBS. How about something like the following? (Is it okay to publish this kind of thing ?? ... Those who use it are at their own risk)

transCSV.vbs


if WScript.Arguments.Count = 0 then
    WScript.echo("too few arguments.")
    WScript.Quit(-1)
end if

Set objFso = CreateObject("Scripting.FileSystemObject")
FileName = WScript.Arguments(0)

if objFso.FileExists(FileName) = 0 then
    WScript.echo("filename err")
    WScript.Quit(-1)
end if


tmpFileName = split(FileName,".")
outFileName = tmpFileName(0) & "_out." & tmpFileName(1)


Set objReadFile = objFso.OpenTextFile(FileName , 1, False)
Set objWriteFile = objFso.OpenTextFile(outFileName , 2, True)

If Err.Number > 0 Then
    WScript.Echo "Open Error"
Else
    ' write header
    objWriteFile.WriteLine "year,month,day,hour,temp,wind,angle,weather,"
    line_cnt = 0
    Do Until objReadFile.AtEndOfStream
        line_str = objReadFile.ReadLine
        If line_cnt > 5 then
            line_info = split(line_str,",")
            yearValue = split(split(line_info(0)," ")(0),"/")(0)
            monthValue = split(split(line_info(0)," ")(0),"/")(1)
            dayValue = split(split(line_info(0)," ")(0),"/")(2)
            hourValue = split(split(line_info(0)," ")(1),":")(0)
            temp = line_info(1)
            wind = line_info(4)
            
            strAngle = line_info(6)
            
            weather = line_info(9)

            If Not(len(temp)=0 or len(wind)=0 or len(strAngle)=0 or len(weather)=0) then
                If strcomp(strAngle,"North") = 0 then
                    angle = 0
                ElseIf strcomp(strAngle,"North-northwest") = 0 then
                    angle = 1
                ElseIf strcomp(strAngle,"North-northwest") = 0 then
                    angle = 2
                ElseIf strcomp(strAngle,"Northwest") = 0 then
                    angle = 3
                ElseIf strcomp(strAngle,"West-northwest") = 0 then
                    angle = 4
                ElseIf strcomp(strAngle,"West") = 0 then
                    angle = 5
                ElseIf strcomp(strAngle,"West-southwest") = 0 then
                    angle = 6
                ElseIf strcomp(strAngle,"Southwest") = 0 then
                    angle = 7
                ElseIf strcomp(strAngle,"South-southwest") = 0 then
                    angle = 8
                ElseIf strcomp(strAngle,"South") = 0 then
                    angle = 9
                ElseIf strcomp(strAngle,"South-southeast") = 0 then
                    angle = 10
                ElseIf strcomp(strAngle,"Southeast") = 0 then
                    angle = 11
                ElseIf strcomp(strAngle,"East-southeast") = 0 then
                    angle = 12
                ElseIf strcomp(strAngle,"east") = 0 then
                    angle = 13
                ElseIf strcomp(strAngle,"East-northeast") = 0 then
                    angle = 14
                ElseIf strcomp(strAngle,"Northeast") = 0 then
                    angle = 15
                ElseIf strcomp(strAngle,"Northeast") = 0 then
                    angle = 16
                End If
                line_out_str = yearValue
                line_out_str = line_out_str & "," & monthValue
                line_out_str = line_out_str & "," & dayValue
                line_out_str = line_out_str & "," & hourValue
                line_out_str = line_out_str & "," & temp
                line_out_str = line_out_str & "," & wind
                line_out_str = line_out_str & "," & angle
                line_out_str = line_out_str & "," & weather
                objWriteFile.WriteLine line_out_str
            End If
        End If
        line_cnt = line_cnt + 1
    Loop
End If


objReadFile.Close
objWriteFile.Close
Set objReadFile  = Nothing
Set objFso = Nothing

WScript.Echo FileName & " > " & outFileName & " transport end!"

To do this, run the following from the command line:

cscript transCSV.vbs data.csv

Then, data_out.csv was created in the same folder. (Display halfway)

data_out.csv


hour,temp,wind,angle,weather,
1,-3.0,1.8,0,2
2,-3.2,1.7,3,2
3,-4.1,1.1,0,2
4,-4.6,1.1,9,2
5,-4.6,1.0,12,2
6,-4.5,0.6,14,2
7,-5.1,1.0,0,2
8,-4.2,0.7,16,2
9,-2.7,0.5,13,2

Now you're ready to get it in Pandas. With reference to other samples, it seems that it can be imported with the following Python code.

import pandas as pd
import numpy as np
import tensorflow as tf

# deta making???
csv_input = pd.read_csv(filepath_or_buffer="data_out.csv", encoding="ms932", sep=",")

#Number of input items (number of lines)*The number of columns) will be returned.
print(csv_input.size)

#Returns the DataFrame object extracted only for the specified column.
x = np.array(csv_input[["hour","day","month"]])
y = np.array(csv_input[["wind"]])

So how do you play from here ??? First of all, I will try regression analysis using a neural network (predicting y from x) ... but it has become a little longer, so I will carry it over to the next time. I haven't had much time lately, but I plan to write more somewhere ... I'll do my best.

Recommended Posts

Let's play with Amedas data-Part 1
Let's play with Amedas data-Part 4
Let's play with Amedas data-Part 3
Let's play with Amedas data-Part 2
Let's play with 4D 4th
Let's play with Excel with Python [Beginner]
Play with Prophet
[Introduction to WordCloud] Let's play with scraping ♬
[Complement] [PySide] Let's play with Qt Designer
Play with PyTorch
Play with 2016-Python
Play with CentOS 8
Play with Pyramid
Play with Fathom
Python hand play (let's get started with AtCoder?)
[Piyopiyokai # 1] Let's play with Lambda: Creating a Lambda function
Play with Othello (Reversi)
[Let's play with Python] Make a household account book
Let's play with JNetHack 3.6.2 which is easier to compile!
[Piyopiyokai # 1] Let's play with Lambda: Get a Twitter account
[Piyopiyokai # 1] Let's play with Lambda: Creating a Python script
Play with reinforcement learning with MuZero
Play with push notifications with imap4lib
Let's run Excel with Python
Play around with Linux partitions
Let's make Othello with wxPython
Play with Jupyter Notebook (IPython Notebook)
[Python] Play with Discord's Webhook.
Let's make dice with tkinter
[Let's play with Python] Image processing to monochrome and dots
Let's write python with cinema4d.
Play RocketChat with API / Python
Let's do R-CNN with Sklearn-theano
Let's build git-cat with Python
Play with ASE MD module
Play with A3RT (Text Suggest)
[Let's play with Python] Aiming for automatic sentence generation ~ Completion of automatic sentence generation ~
Let's upload S3 files with CLI
Play with numerical calculation of magnetohydrodynamics
Play with a turtle with turtle graphics (Part 1)
Let's make a GUI with python.
Play with Poincare series and SymPy
HTTPS with Django and Let's Encrypt
Let's learn Deep SEA with Selene
Let's make a breakout with wxPython
Let's make Othello AI with Chainer-Part 1-
Play with Pythonista UI implementation [Action implementation]
Play with PIR sensor module [DSUN-PIR]
Play around with Linux partitions ~ Continued ~
Let's do image scraping with Python
Let's make a graph with python! !!
Let's make a supercomputer with xCAT
Let's make Othello AI with Chainer-Part 2-
Spark play with WSL anaconda jupyter (2)
Let's recognize emotions with Azure Face
Play with Turtle on Google Colab
Play with demons because it's setsubun
Let's analyze voice with Python # 1 FFT
Let's play with the corporate analysis data set "CoARiJ" created by TIS ①
Let's make an image recognition model with your own data and play!
[Let's play with Python] Aiming for automatic sentence generation ~ Perform morphological analysis ~