pydata

Keep Looking, Don't Settle

pickle error in multiprocssing

This is the useful links I used to solve the issue I got when using multiprocssing in python.

  1. python Multithreading does not work well on windows. multiprocess sometimes have issues too. One simple example is about importing the self-defined module with multiprocessing in another code in windows NT OS. It is necessary to distinguish between the parent and child process with __main__. Refer to This

  2. what can be pickled in python? Functions are only picklable if they are defined at the top level of the module. Refer to this

  3. When define the multiprocessing funtion inside the class, I got the error like Can't pickle when using multiprocessing Pool.map(). I try to solve it with copy_reg as mentioned in 2. Another way to solve it is to use pathos.multiprocessing. But finally I find this is the easiest way to do it: define the __call__ method in the same class to call the function. This is the easiest way for me to solve it.

Reference

  1. Python multiprocessing pickling error
  2. Can't pickle when using multiprocessing Pool.map()
  3. Why does Python's multiprocessing module import main when starting a new process on Windows?