Non-blocking Functions

So far in the MPI laboratories, you have worked with blocking communication functions (MPI_Send, MPI_Recv). In blocking communication, send and receive functions block until the buffer used for message transmission can be reused, specifically:

• For send (MPI_Send), a result is returned when the buffer where the data is placed for transmission can be reused (until then, sending is blocked).
• For receive (MPI_Recv), a result is returned when all data transmitted through the buffer can be processed (until then, receiving is blocked).

Message communication is divided into two main categories:

• Blocking, which is synchronous
• Non-blocking, which is asynchronous

Blocking sending comes in four modes:

• Standard (MPI_Send)
• Synchronized (MPI_Ssend)
• Buffered (MPI_Bsend)
• Ready (MPI_Rsend)

Regarding non-blocking communication, the data sending (MPI_Isend) and receiving (MPI_Irecv) functions immediately return a result. Thus, there are no guarantees regarding the completion of data transmission. To ensure that the data has been sent and received correctly and completely, we can use the MPI_Test and MPI_Wait functions, which we will discuss in detail below.

Non-blocking communication is useful for situations where deadlock may occur or when sending large amounts of data.

MPI_Isend

Function signature:

int MPI_Isend(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request *request)

The parameters of the MPI_Isend function behave the same as the parameters of the MPI_Send function, but what is added is the *MPI_Request request parameter, where MPI_Request is a structure used for testing the sending and receiving of data.

MPI_Irecv

Function signature:

int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request *request)

Unlike MPI_Recv, MPI_Irecv does not have an MPI_Status parameter, which is replaced by the MPI_Request parameter. Everything else remains identical to MPI_Recv.

MPI_Test

Function signature:

int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status)

Where:

• request is the one from the send or receive function whose result is being tested.
• flag indicates whether the send or receive operation has been successfully completed.
• status contains message-related data (relevant for the receive operation).

MPI_Wait

Function signature:

int MPI_Wait(MPI_Request *request, MPI_Status *status)

Where:

• request is the one from the send or receive function whose result is being tested.
• status contains message-related data (relevant for the receive operation).

First, it is tested using MPI_Test whether a function (send or receive) has finished sending/receiving the data. If the send/receive operation has not been completed, then MPI_Wait will be used.

Example of send-receive using non-blocking functions:

#include <mpi.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
 
#define MAX_LEN 10
 
int main(int argc, char *argv[])  {
    int numtasks, rank, dest, source, count, flag, tag = 1;
    char inmsg[MAX_LEN], outmsg[] = "Hello";
    MPI_Status status;
    MPI_Request request;
 
    MPI_Init(&argc,&argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numtasks);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
 
    if (rank == 0) {
        // process 0 sends to process 1
        dest = 1;
        source = 1;
 
        sleep(3);
 
        MPI_Isend(outmsg, strlen(outmsg) + 1, MPI_CHAR, dest, tag, MPI_COMM_WORLD, &request);
        MPI_Test(&request, &flag, &status);
 
        if (flag) {
            printf("[P0] The send operation is over\n");
        } else {
            printf("[P0] The send operation is not over yet\n");
            MPI_Wait(&request, &status);
        }
 
        printf("[P0] The send operation is definitely over\n");
    } else if (rank == 1) {
        // process 1 is waiting for message from process 0
        dest = 0;
        source = 0;
 
        sleep(1);
 
        MPI_Irecv(inmsg, MAX_LEN, MPI_CHAR, source, tag, MPI_COMM_WORLD, &request);
        MPI_Test(&request, &flag, &status);
 
        if (flag) {
            printf("[P1] The receive operation is over\n");
        } else {
            printf("[P1] The receive operation is not over yet\n");
            MPI_Wait(&request, &status);
        }
 
        /*
         the status variable is used to find out details about the data exchange
         MPI_Get_count - calculates how many elements have been received
         */
        MPI_Get_count(&status, MPI_CHAR, &count);
        printf("[P1] Received %d char(s) from process %d with tag %d: %s\n", count, status.MPI_SOURCE, status.MPI_TAG, inmsg);
    }
 
    MPI_Finalize();
}