A hands-on collection of 25 Python scripts demonstrating MPI (Message Passing Interface) concepts — from basic process communication to parallel machine learning, scheduling algorithms, and computer vision.
A group of processes that can communicate with each other. MPI.COMM_WORLD is the default communicator containing all processes.
Each process has a unique integer ID called rank. size is the total number of processes in the communicator.
Point-to-point communication. comm.send() and comm.recv() let two processes exchange data directly.
comm.bcast() sends data from one root process to all other processes in the communicator simultaneously.
comm.scatter() distributes chunks of data to all processes; comm.gather() collects results back at root.
comm.reduce() applies a collective operation (SUM, MAX, etc.) across all processes and returns the result to root.
comm.Split(color, key) partitions processes into sub-groups, enabling hierarchical parallel designs.
In scheduling, makespan is the total time from start to completion of all jobs — a key metric for evaluating parallel efficiency.
On Ubuntu/Debian:
# Install OpenMPI sudo apt-get install openmpi-bin openmpi-common libopenmpi-dev # Install Python binding pip install mpi4py
Clone the repository and run any script:
# Clone the repo git clone https://github.com/ZiaUrRehman-bit/\ MPI--Message-Passing-Interface--\ Using-Python-for-Parallel-Programming # Run with N processes mpirun -n 4 python 01.SendMessage.py
Simplest MPI script:
from mpi4py import MPI comm = MPI.COMM_WORLD rank = comm.rank # This process ID size = comm.size # Total processes print(f"I am process {rank} of {size}")
Detect available CPU cores:
import multiprocessing from mpi4py import MPI cores = multiprocessing.cpu_count() print(f"CPU cores available: {cores}") # Run with: mpirun -n {cores} python script.py
Demonstrates point-to-point messaging using comm.send() and comm.recv(). Process 0 sends a message; Process 1 receives and prints it.
from mpi4py import MPI comm = MPI.COMM_WORLD rank = comm.rank if rank == 0: comm.send("Hello from P0", dest=1) elif rank == 1: msg = comm.recv(source=0) print(f"Received: {msg}")
Every parallel MPI program starts here. Each process reads its unique rank and the total number of processes (size) from the communicator.
from mpi4py import MPI comm = MPI.COMM_WORLD rank = comm.rank size = comm.size print(f"I am process {rank} of {size}")
Baseline sequential computation to measure execution time. Used as the reference point for comparing speedup achieved with MPI parallelization.
The parallel version of the same computation from Script 03. Distributes work across processes and measures total elapsed time to quantify speedup over sequential execution.
Uses comm.Split(color, key) to partition processes into sub-groups based on even/odd rank. Each sub-group gets its own communicator with local ranks.
color = rank % 2 # 0=even, 1=odd sub_comm = MPI.COMM_WORLD.Split(color, rank) print(f"Rank {rank}: color={color}, local_rank={sub_comm.rank}")
Uses Python's multiprocessing.cpu_count() to detect the number of available CPU cores — the practical upper bound for parallel processes.
Retrieves the number of logical CPU cores from the OS. Useful for dynamically sizing the MPI process pool to match hardware capacity.
Two-way handshake between processes. Process 0 sends a message, Process 1 receives it and sends back an acknowledgement — demonstrating bidirectional MPI communication.
if rank == 0: comm.send("Hello", dest=1) ack = comm.recv(source=1) print(f"Got ack: {ack}") elif rank == 1: msg = comm.recv(source=0) comm.send("ACK from P1", dest=0)
Root process distributes chunks of a data array to all processes via scatter(). Each process modifies its chunk locally, then results are collected at root via gather().
data = [i**2 for i in range(size)] local = comm.scatter(data, root=0) local *= 2 # local computation results = comm.gather(local, root=0)
Root process broadcasts a single value to all processes using comm.bcast(). All processes receive an identical copy — useful for sharing parameters or configuration.
data = "shared config" if rank == 0 else None data = comm.bcast(data, root=0) print(f"P{rank} got: {data}")
Splits matrix rows across MPI processes. Each process multiplies its assigned rows independently, then results are gathered at root — a classic parallel linear algebra pattern.
Divides an image into horizontal strips and distributes strips across processes. Each process applies a filter (e.g., Gaussian blur) in parallel via OpenCV, then strips are reassembled.
Benchmarks sequential vs parallel image processing. Measures and prints execution time for both approaches to quantify the speedup from MPI parallelization on image workloads.
Introduces MPI for ML workloads. Distributes training data across processes, trains local models in parallel, then aggregates predictions at root using gather().
Minimum Completion Time scheduling parallelized with MPI. Jobs are partitioned across processes; each computes start/completion/turnaround/waiting times, then all metrics are gathered at root.
Sequential baseline of the MCT scheduler. Jobs sorted by processing time, assigned to earliest-completion processor. Used to benchmark against the MPI parallel version.
Trains a classification model on the Pima Indians Diabetes dataset using Scikit-learn. Sequential training baseline for comparison with the MPI-distributed version in Script 18.
Full parallel ML pipeline on the Diabetes dataset. Data is scattered to processes, each trains a local model, accuracy scores are gathered at root and averaged — federated-style training.
local_data = comm.scatter(chunks, root=0) model = RandomForestClassifier() model.fit(local_data[X], local_data[y]) acc = model.score(X_test, y_test) all_acc = comm.gather(acc, root=0) if rank == 0: print(f"Avg accuracy: {sum(all_acc)/len(all_acc):.2f}")
Reads a live webcam feed using OpenCV and applies real-time frame processing. Sequential baseline for comparing with the MPI-parallelized webcam pipeline.
Parallelizes webcam frame processing. Frames are distributed to worker processes which apply filters concurrently, enabling higher throughput real-time computer vision pipelines.
Parallel Min-Min scheduler: each process handles a job subset, selecting the minimum-completion-time job iteratively. Results gathered via reduce(MPI.SUM) for aggregate metrics.
Sequential Min-Min scheduling: always selects the job with minimum expected completion time. Serves as the non-parallel baseline for measuring MPI overhead and speedup.
Implements the Sufferage heuristic: assigns each job to the resource where it would "suffer" most (largest difference between best and second-best completion time). Sequential version.
First Come First Served job scheduling. Jobs processed in arrival order. Computes makespan, start/completion/turnaround/waiting times. Baseline for FCFS-MPI comparison.
Parallelized FCFS scheduler using MPI. The 800-job GoCJ dataset is partitioned among processes; each computes its chunk's scheduling metrics, then all results are gathered at root.
chunk = jobs[start:end] for job_id, size in chunk: start_times[job_id] = current_time completion_times[job_id] = current_time + size current_time += size all_st = comm.gather(start_times, root=0) all_ct = comm.gather(completion_times, root=0)
Three classical scheduling algorithms are implemented both sequentially and with MPI parallelization. All use the GoCJ Dataset (800 jobs) for benchmarking.
| Algorithm | Strategy | Key Metric | MPI Pattern | Best For |
|---|---|---|---|---|
| FCFS | Process jobs in arrival order | Makespan, Waiting Time | Partition → Gather | Simple queues, fairness |
| Min-Min | Always assign job with minimum completion time | Minimized Makespan | Partition → Reduce (SUM) | Heterogeneous workloads |
| MCT | Assign each job to processor with earliest completion | Turnaround Time | Partition → Gather all metrics | Load balancing |
| Sufferage | Assign to resource with highest "sufferage" value | Reduced starvation | Sequential only | Preventing poor assignments |
Zia Ur Rehman — PhD Researcher in Computer Science at the University of Limerick, Ireland.
This repository was developed as part of research and teaching work at the Institute of Space Technology, Islamabad. It covers MPI-based parallel programming from fundamentals to advanced applications in machine learning and computer vision.
Current research focuses on Federated Learning, Explainable AI, Evolutionary Algorithms, and Green AI — with MPI parallelization as a core tool for scaling ML workloads.