[PYTHON] Example of pytest environment to fix database with Docker

Purpose

I want to easily execute unit tests of Python applications using databases with pytest without being affected by the environment. (The one that starts up as a daemon instead of a file such as SQLite3)

What was made

Sample code: https://github.com/hkato/pytest-docker-mysql-example

Launch MySQL from pytest on Docker, input the schema and test data, and execute the test. If you set docker-compose and support SQL related docker-entrypoint-initdb.d, it should be basically the same for other databases such as PostgreSQL. (The initialization mechanism in docker-entrypoint-initdb.d should be the same for MySQL / PostgreSQL official containers, but I don't know anything else)

Required environment and pytest plugin

Regarding pytest, I started Docker and looked at plugins that handle databases, but pytest-docker is the maintenance status and my usage / environment. Looks good for. pytest-dotenv was used by Docker and pytest to describe environment variables in .env files and make them common.

Other considerations

As a result of some consideration other than pytest-docker,

-pytest-docker-compose: A little old. Since pytest-docker handles docker-compose, the impression is that pytest-docker with solid maintenance and documentation is better. -pytest-dbfixtures Series: Compatible with mainstream databases -pytest-mysql: I haven't seen it properly, but it behaves as a basic client, so handling Docker seems to be troublesome.

After all, pytest-docker seems to be the best because it can use the container of the web application as described in the official document or can be used for general purposes regardless of the database ([Start MinIO container and replace AWS S3](https:: //qiita.com/hkato/items/89e436300c50c46624b9) I'm thinking of doing it too).

Directory structure

.
├── src                 #Application code
│   └── *.py
├── tests               #Test code
│   ├── conftest.py
│   └── test_*.py
├── initdb.d            #Database schema & test data etc.
│   ├── *.sql
│   └── *.sh
├── pytest.ini          #pytest settings
├── .env                #Environment variable settings
└── docker-compose.yml  #Docker settings

Code overview

conftest.py

Docker-compose.yml placement support

By default pytest-docker references docker-compose.yml undertests /. By setting it directly under the project (specify another directory or file name in the sample in README.md of pytest-docker), in the case of Web application etc., run the database with docker-compose up -d You can run and debug the web app server with Gunicorn, uWSGI, Uvicorn, etc. (this is not the essence of this time, so this is excluded).

Supports waiting for MySQL startup

The pytest-docker documentation describes it as a sample that launches a httpbin REST API container and waits for 200 responses from port 80. In the case of MySQL, I connected with PyMySQL as shown below and decided that Exception did not come back, that is, the Docker container started up and the initial data input was completed.

def is_database_ready(docker_ip):
    try:
        pymysql.connect(
            host=docker_ip,
            user=os.getenv('MYSQL_USER', ''),
            password=os.getenv('MYSQL_PASSWORD', ''),
            db=os.getenv('MYSQL_DATABASE')
        )
        return True
    except:
        return False

Wait for the above with a 30 second timeout in a 0.1 second cycle.

@pytest.fixture(scope="session")
def database_service(docker_ip, docker_services):
    docker_services.wait_until_responsive(
        timeout=30.0, pause=0.1, check=lambda: is_database_ready(docker_ip)
    )
    return

Execution result

Initial setting of Python environment

$ #In actual operation, it may be better to use Pipenv or Poetry for the test environment and execution environment.
$ python -m venv .venv
$ source .venv/bin/activate
$ pip install -r requirements.txt

Environment variable settings

$ cat << 'EOF' > .env
MYSQL_ROOT_PASSWORD='p@ssw0rd'
MYSQL_DATABASE='mydb'
MYSQL_USER='foo'
MYSQL_PASSWORD='pa$$Word'
EOF

When the test is executed, the Docker container is started by Fixture, and the actual unit test is executed after SQL and shell scripts are applied by the mechanism of docker-entrypoint-initdb.d in the container.

$ pytest
============================= test session starts ==============================
platform darwin -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /Users/username/tmp/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/username/tmp, configfile: pytest.ini
plugins: dotenv-0.5.2, cov-2.10.1, docker-0.10.1
collected 1 item                                                               

tests/test_users.py::test_get_name_by_id PASSED                          [100%]

---------- coverage: platform darwin, python 3.8.6-final-0 -----------
Name         Stmts   Miss  Cover
--------------------------------
src/app.py       8      0   100%


============================== 1 passed in 16.35s ==============================

Summary

--pytest-docker is convenient because you can launch services that depend on tests with Docker, not limited to databases. --It may be good to change the location of docker-compose.yml from tests / to directly under the project. Can also be used for normal debugging --Implement start wait processing according to the contents of the container in docker_services.wait_until_responsive ()

Recommended Posts

Example of pytest environment to fix database with Docker
Prepare the execution environment of Python3 with Docker
Fix database with pytest-docker
From Kafka to KSQL --Easy environment construction with docker
Preparing the execution environment of PyTorch with Docker November 2019
Ssh to virtual environment with remote development of vscode
Prepare python3 environment with Docker
Pros and cons of converting Django's development environment to Docker
From environment construction to deployment for flask + Heroku with Docker
Build Mysql + Python environment with docker
How to use jupyter notebook without polluting your environment with Docker
I tried to fix "I tried stochastic simulation of bingo game with Python"
Build PyPy execution environment with Docker
Rebuild Django's development environment with Docker! !! !! !!
Data science environment construction with Docker
I wanted to use jupyter notebook with docker in pip environment (opticspy)
Prepare an environment to touch grib2 format files with python (Docker edition)
Setting to debug test by entering the contents of the library with pytest
Environment maintenance made with Docker (I want to post-process GrADS in Python
Summary of how to build a LAMP + Wordpress environment with Sakura VPS
Change Python 64bit environment to 32bit environment with Anaconda
How to create an NVIDIA Docker environment
Connect to MySQL with Python within Docker
Get a local DynamoDB environment with Docker
Script example to display BoundingBox with PIL
Create Python + uWSGI + Nginx environment with Docker
Try to prepare each environment of kivy
[Linux] Build a jenkins environment with Docker
Launch environment with LineBot + Heroku + Docker + Python
Introduction to docker Create ubuntu environment in ubuntu
Example of efficient data processing with PANDAS
Build NGINX + NGINX Unit + MySQL environment with Docker
How to install python3 with docker centos
[Linux] Build a Docker environment with Amazon Linux 2
I tried to build an environment of Ubuntu 20.04 LTS + ROS2 with Raspberry Pi 4
An example of cloudbuild.yaml when auto-deploying Django to App Engine with Cloud Build
Ubuntu 16.04 LTS, beginner memorandum of environment construction to switch anaconda version with pyenv
A simple example of how to use ArgumentParser
Hello World with gRPC / go in Docker environment
Build Django + NGINX + PostgreSQL development environment with Docker
Analytical environment construction with Docker (jupyter notebook + PostgreSQL)
Go (Echo) Go Modules × Build development environment with Docker
I want to mock datetime.datetime.now () even with pytest!
[Python] Build a Django development environment with Docker
Use multiple versions of python environment with pyenv
Example of getting mastodon access token with authorization_code
Create Nginx + uWSGI + Python (Django) environment with docker
Script to tweet with multiples of 3 and numbers with 3 !!
How to specify attributes with Mock of python
How to implement "named_scope" of RubyOnRails with Django
Story of trying to use tensorboard with pytorch
Poetry-virtualenv environment construction with python of centos-sclo-rh ~ Notes
Problems connecting to MySQL from Docker environment (Debian)
Flow of creating a virtual environment with Anaconda
Send msgpack with ajax to flask (werkzeug) environment
From Python environment construction to virtual environment construction with anaconda
Procedure to set hydrogen of atom (virtual environment)
Build PyPy and Python execution environment with Docker
Example of reading and writing CSV with Python
Use anaconda virtual environment with Zsh (problem fix)
How to get started with Visual Studio Online ~ The end of the environment construction era ~