‹‹‹ Blog Overview

Pure Python & Numpy openMP-style parallel for-loops!
POSIX shared memory and forks

I recently came across POSIX shared memory and thought like ... what can you do with this and numpy?

  • BSD
  • Linux
  • Python
  • Unix
  • fork
  • parallel computing
  • shared memory

Long story short: I searched for bits and pieces, found some tests by other people (good one here), but no complete, "clean" implementation. I would love to know if there is some Python package for this that I may have overlooked.

Otherwise, meet multi-process (bypassing the GIL) shared-memory prange-style (just like in numba and Cython) for-loops in virtually pure Python but very similar to what you can do with openMP & its threads in C & Fortran:

@parallel(processes = 2, shared = ('a',))
def task(a: np.ndarray, b: int):
    for idx in prange(a.shape[0]):
        print(f'[worker_id={worker_id:d} pid={os.getpid():d}] idx=={idx:d}')
        a[idx] += b

def demo():
    a = np.arange(1, 11, dtype = 'f8')
    task(a = a, b = 7)
Source code 1: Process-based parallelism with shared memory as if the processes where threads. Array a is shared between processes. prange executes parts of the for-loop in individual processes, similar to OpenMP from a code-design-perspective.

A complete implementation plus two demos is a available as a gist.

This text was posted and discussed on reddit, here and here. As it turns out, there is at least one established Python package in this field: shared-array.

‹‹‹ Blog Overview