One of the benefits of a system architecture like that shown in Figures 7 and 8 is the flexibility provided in choosing where to insert vendor-specific optimizations. One illustration of how this flexibility was used is given by the evolution of the Silicon Graphics version of MPICH.
Since Chameleon had been ported to p4 and p4 had been ported to SGI workstations long before the MPICH project began, MPICH ran on SGI machines from the very beginning. This is the box shown as SGI(0) in Figure 8 . This implementation used TCP/IP sockets between workstations and standard Unix System V shared memory operations for message passing within a multiprocessor like the SGI Onyx.
The SGI(1) box in Figure 8 illustrates an enhanced version achieved by using a simple, portable shared-memory interface we call p2 (half of p4). In this version, shared memory operations use special SGI operations for shared-memory functions instead of the less robust System V operations.
SGI(2) in Figure 8 is a direct implementation of the channel interface that we did in collaboration with SGI. It uses SGI-specific mechanisms for memory sharing that allow single-copy data movement between processes (as opposed to copying into and out of an intermediate shared buffer), and it uses lock-free shared queue management routines that take advantage of special assembler language instructions of the MIPS microprocessor.
SGI next developed a direct implementation of the ADI that did not use the channel interface model (SGI(3) in Figure 7 ), and then bypassed the ADI altogether to produce a very high-performance MPI implementation for the Power Challenge Array product, combining both shared-memory operations and message-passing over the HiPPI connections between shared-memory clusters. Even at this specialized level, it retains much of the upper levels of MPICH that are implemented either independently of, or completely on top of, the message-passing layer, such as the collective operations and topology functions.
At all times, SGI users had access to a complete MPI implementation, and their programs did not need to change in any way as the implementation improved.