The CommManager class

The CommManager class defines functions required for communication between LogicalProcesses over MPI. As previously mentioned, communication between simulation objects local to each other occurs by direct insertion into the input queue, so the CommManager is not involved. Objects sending events to non-local destinations use the CommManager to do so.

Like the simulation objects, the CommManager has a method called recvEvent(BasicEvent *) to receive events that simulation objects wish the CommManager to handle. In addition, the CommManager has several overloaded versions of a method called recvMsg which are used by the LP and the GVTManager to send messages to other objects on other LPs. The types of messages defined include the following (these definitions can be found in KernelMsgs.hh):

• BasicMsg: this is the base message type. This type defines the minimum information needed to pass a message from one LP to another. Every other message is derived from this type. The information required to be filled in at the time of sending includes:
  • destLP: this is the id of the destination LogicalProcess.
  • type: this is an enumerated type that gets filled in automatically when new is called to create any of the messages derived from the base type. The value of type is the same for a class as its name, but all capitalized. For instance, the value of type for an InitMsg is INITMSG.

  Additional information in the fields senderLP and sequence is filled in automatically before the message is given to MPI. senderLP is the id of the LP sending the message, and sequence is a monotonically increasing long long the counts the number of messages sent from the sending LP to the destination LP.

• InitMsg: these are the messages that are exchanged between LPs to communicate where each simulation object resides. In addition to the BasicMsg info, objId defines the id of the simulation object, and lpId contains the id of the LP that has that object.
• StartMsg: LP 0 sends these out when it the simulation is initialized and ready to begin. These contain no additional information from a BasicMsg, other than the type itself.
• TerminateMsg: when received, the LP will immediately exit. These are used for abnormal termination on error. In addition to the BasicMsg information, this message contains a string of up to 255 characters that contains an error message.
• CheckIdleMsg: these messages are circulated to all LPs to determine if there is an LP that isn't idle. In addition to the BasicMsg information, a bool flag cancel tells the receiving LP if the the sender wants to cancel this circulation or not, and the int numCirculations tells the receiver how many times this message has visited all of the LPs. Upon receiving this message, if the LP is idle, it passes it on to the LP with id = id + 1 (or id = 0, if this is the last LP). If it is not idle, then it is return to LP 0 with idle = false.
• EventMsg: these messages are inherited from both BasicMsg and BasicEvent. When the CommManager receives a BasicEvent for delivery, it copies the BasicEvent into an EventMsg, and fills in the destination from the LP's simArray. When received, the data is copied back out into a BasicEvent for delivery to the simulation object.

The above conclude the base message types. The following message types are also included in the base WARPED kernel to implement the PGVT algorithm. Future releases of WARPED will remove these message classes from the base kernel definition.

• LGVTMsg: these are messages passed between the GVTManagers to report changes in LGVT. The LGVT value being reported is stored in VTime lGVT. In addition, there is a bool lpIdle that tells if the LP was idle at the time of sending the message, to give the central GVTManager a hint as to which LPs are active.
• LowerLGVTMsg: this is the same as an LGVTMsg except that it tells the receiver to send back acknowledgment that the message has been processed. This is for handling valley messages in the PGVT algorithm, which will be discussed in the next section.
• NewGVTMsg: these messages inform the LPs that GVT has advanced to a new value. The new GVT value is a VTime called newGVT. Upon receiving this message, the LP performs garbage collection on the input, output and state queues of each simulation object it is responsible for.
• GVTAckMsg: this is the type of message generated when a GVTManager is acknowledging a LowerLGVT message from another GVTManager. The sequence number of the message being acknowledged is copied into the long long called sequenceMsgAcked, since each LP can have multiple LowerLGVTMsgs pending.
• LPAckMsg: this is the type of message sent from one LP to another to accept responsibility for a valley message as required by the PGVT algorithm. As with GVTAckMsgs, the sequence number of the message that is being acknowledged is copied into the response.
• DummyRequestMsg and DummyMsg: these messages are used to avoid circumstances which can cause GVT to quit advancing before the end of the simulation. They will be described in detail in the section on the GVTManager.

For each of the above message types, the method CommManager::recvMsg is overloaded to handle the message correctly. For instance, if the CommManager receives a TerminateMsg, it will deliver the message to all of the LPs in the simulation. In the case of an EventMsg, the message is simply delivered to the correct LP, and the receiving CommManager delivers the message to the correct simulation object. Similarly, the method remoteSend is overloaded for all of the mentioned message types to send messages over MPI to other LPs.

The CommManager's declaration follows:

class CommManager : public BasicTimeWarp {
friend class GVTManager;
public:
  CommManager();
  ~CommManager();
  void initialize(ObjectRecord*,  // pointer to simulation array
		  GVTManager*,    // pointer to the GVT Manager
		  int);           // The number of LPs

  int getID() { return id; };

  // These methods are for the _Commucation Manager_ to receive the types
  // of events passed in ( by local objects )
  void recvEvent( BasicEvent* );
  void recvMsg( TerminateMsg * );
  void recvMsg( LGVTMsg * );
  void recvMsg( LowerLGVTMsg * );
  void recvMsg( newGVTMsg * );
  void recvMsg( GVTAckMsg * );
  void recvMsg( LPAckMsg * );
  void recvMsg( InitMsg * );
  void recvMsg( StartMsg * );
  void recvMsg( CheckIdleMsg * );
  void recvMsg( DummyMsg * );
  void recvMsg( DummyRequestMsg * );

  // this is a method the LP calls to let everyone know that he's got a
  // certain simulation object in his posession.
  void iHaveObject(int);


  int recvMPI(int = 1); // wait for the number of messages specified (and
                    // deliver them). If there aren't that many 
                    // messages, stop receiving them.  In either case,
                    // return the number of messages actually received.

  BasicMsg * recvMPIMsg();  // grab ONE message from MPI and return it..
                              // return NULL if there are none...

  void terminateSimulation( TerminateMsg * );

  long long getSequence( int );  // returns the sequence number of the
                                 // next message to the LP with the id
                                 // passed in
  void waitForStart();
  void waitForInit( int numExpected );

  // since the general model of memory in this system is that whoever
  // news memory, deletes it, the commanager gets to garbage collect
  // too
  void gcollect(VTime gtime);

  // these are to please the compiler
  void executeProcess();
  BasicEvent* getEvent();
  void simulate();
  
  VTime calculateMin();
private:

  int numLPs;
  ObjectRecord *simArray;
  GVTManager *gVTHandle;

  // these will be filled in with arrays so that each communication channel
  // has a unique sequence running

  long long * sendSequenceArray;
  long long * recvSequenceArray;

  // we have to define this in order to please the compiler
  void sendEvent( BasicEvent*);

  // These methods are to allow the Communation Manager to deliver messages to
  // local objects only
  void deliver(BasicEvent *);
  void deliver(LGVTMsg *);
  void deliver(LowerLGVTMsg *);
  void deliver(newGVTMsg *);
  void deliver(LPAckMsg *);
  void deliver(GVTAckMsg *);
  void deliver(CheckIdleMsg *);
  void deliver(DummyMsg *);
  void deliver(DummyRequestMsg *);

  // this method takes an LP Msg, figures out what kind of derived type it is,
  // and then delivers it to the LOCAL entity that is receiving it.
  void routeMsg( BasicMsg *);

  void remoteSend( BasicMsg *, int );

};