Distributed Computing in Python

PySpark is the Python API for Apache Spark, which runs on the JVM. It is the mature choice for large batch jobs and SQL over data lakes, especially where a Spark platform already exists. Dask scales NumPy, pandas and custom task graphs from one machine to a cluster with an API that feels like the libraries you already use, and it is pure Python, so it is lighter to adopt. Ray is a general distributed runtime built on tasks and actors, with libraries on top for training, tuning and serving models, which makes it common in machine learning infrastructure.

joblib is the simple option for one machine: it parallelizes loops across processes and caches expensive function results to disk, and scikit-learn uses it internally. mpi4py brings MPI to Python for high-performance computing clusters where jobs already use MPI. Start with the smallest tool that works; many workloads that seem to need a cluster fit on one machine with joblib or Dask.