[PYTHON] Row profiling with Cython

Overview

How to line profile inside the " cdef "function in the Cython code" * .pyx"

1. Library installation

pip install line_profiler

2. Modification of pyx file

2.1. Added line profile declaration at the beginning of the file

Set the profile declaration as a comment line at the beginning of the pyx file

Change before)

# -*- coding: utf-8 -*-
#
import math
...

After change)

# -*- coding: utf-8 -*-
#
# cython: profile=True
# cython: linetrace=True
# cython: binding=True
# distutils: define_macros=CYTHON_TRACE_NOGIL=1
import math
...

2.2. Calling the profile decorator

cdef does not have a function decorator, so enclose the function you want to call in profile

Change before)

a = funca(a1, a2)

funcb(a, b1)

After change)

pfa = profile(funca)
a = pfa(a1, a2)

pfb = profile(funcb)
pfb(a, b1)

2.3. Change the cdef function to the cpdef function

If you leave the cdef function, you will get an error with stringsource.wrap

Change before)

cdef funca(a1, a2):
    return a1 * a2

After change)

cpdef funca(a1, a2):
    return a1 * a2

3. Modify setup.py

3.1. Addition of Extention parameters

Change before)

       Extension("service.AService", sources=["service/AService.pyx"], include_dirs=['.']), 

After change)

       Extension("service.AService", sources=["service/AService.pyx"], include_dirs=['.'], define_macros=[('CYTHON_TRACE', '1')]), 

3.2. Changes to cythonize

setup(name="*", cmdclass={"build_ext": build_ext}, ext_modules=cythonize(setup_ext.ext, annotate=True, \
      compiler_directives={'language_level': "3"}))

After change)

setup(name="*", cmdclass={"build_ext": build_ext}, ext_modules=cythonize(setup_ext.ext, annotate=True, \
      compiler_directives={'language_level': "3", 'profile': True, 'linetrace': True, 'binding': True}))

4. Compile

Call kernprof at the time of compiling setup.py

kernprof -l setup.py build_ext --inplace

5. Profile execution

kernprof -l -v executor.py

result

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   503                                               cpdef dict prepare_avoidance_dataset(self, int data_set_idx, str direction):
   504         2      29445.0  14722.5      0.1          logger.info("Preparation for contact avoidance [No.%s - %s】", (data_set_idx + 1), direction)
   505
   506                                                   cdef int aidx, fno, from_fno, prev_block_fno, to_fno
   507                                                   cdef float block_x_distance, block_z_distance, x_distance, z_distance
   508                                                   cdef list all_avoidance_list, fnos, prev_collisions
   509                                                   cdef dict all_avoidance_axis, rep_avbone_global_3ds, rep_avbone_global_mats, rep_global_3ds, rep_matrixs, avoidance_list
   510                                                   cdef str avoidance_name, bone_name
   511                                                   cdef bint collision, near_collision
   512                                                   cdef BoneLinks arm_link, avodance_link
   513                                                   cdef ArmAvoidanceOption avoidance_options
   514                                                   cdef MOptionsDataSet data_set
   515                                                   cdef OBB obb
   516                                                   cdef RigidBody avoidance
   517                                                   cdef MVector3D rep_x_collision_vec, rep_z_collision_vec
   518
   519         2       1638.0    819.0      0.0          logger.copy(self.options)
   520                                                   #Data set to be processed
   521         2         29.0     14.5      0.0          data_set = self.options.data_set_list[data_set_idx]
   522

(Abbreviation)

Clean up

Put the code back after profiling