Establishment of Communication Channels

The message passing layer has to establish communication channels between the processes communicating with each other. There are several ways of establishing the communication channels:

Static:
In this approach, the communication channels are established among all the processes before the application starts to communicate. Although this is not static at compile time, it is static because the channels are established before the application starts to communicate. The advantage of this approach is that, there is just a one-time effort involved in setting up of the communication channels. Since all the processes call the init() method, the message passing layer gets control in all the processes to perform any protocol specific activity that needs synchronization between the processes. The disadvantage is that some of the processes may not communicate between them throughout the execution of the parallel application. Consider a parallel application consisting of N processes. This static approach establishes N * (N-1) communication channels for the processes to communicate among themselves. In reality, the communication structure within these process can be cyclic or a subgraph of a complete graph of these processes.
Dynamic:
In this approach, the communication channels are not established with all the other processes before the processes start to communicate. This approach is usually not feasible, because once the processes start communicating, the message passing layer does not have control to perform any protocol specific activity to establish the communication channel between two particular process. Usually this approach is taken, when the message passing software consists of local daemons on each machine which participates in the parallel computation for routing the messages to the destination. The advantage with this approach is the minimal usage of resources, since all communication channels are not established a priori. There are a few disadvantages associated with this approach. When a particular process wants to communicate with another process, the destination process to which it wants to communicate might be executing some application specific code before giving control to the message passing layer. The establishment of communication channels involves some protocol specific activity which has to be synchronized at both the ends. Another disadvantage is that a process might just communicate with another process once throughout its execution. With this approach, the communication channel established for just one data transfer would not be used after that data transfer. The resources are wasted for these kinds of communication patterns too.
Spanning Tree:
In this approach, a spanning tree of N nodes is constructed. The nodes in the graph denote the processes. The edges of the spanning tree denote the bi-directional communication channels established between the nodes. When messages are sent by the user application, the message passing library has to route the messages to the appropriate destinations. This involves the maintenance of routing tables at each node to specify routes to the destinations. This approach increases the one way message latency of messages transmitted between two process as it involves the routing of messages through intermediate nodes. With this approach, one can form a complete graph with each edge having a weight proportional to the communication traffic between two nodes. If one can form the spanning tree with maximum total weight the one way message latency between the frequently communicating nodes will be reduced. This approach consumes minimal resources compared to the other approaches.

One of the main goals of TCPMPL is high-performance. WARPED requires the TCPMPL to have a minimum one way message latency. Therefore, TCPMPL uses the static approach. Static approach has the least message latency although there is a potential wastage of resources. TCPMPL establishes N * (N-1) communication channels, one for every process to communication with one other process. Each communication channel is assumed to be a uni-directional channel (data can be send only in one-direction).

The different communication channels established by TCPMPL is shown in Figure 1. The initiator establishes communication channels in both direction between all the processes started on the remote machine. The processes in turn establish communication channels between them for communication.

  
Figure 1: Communication architecture between the Initiator, master and the slaves.