Aggregation in Collective operations

Combining collective operations
- Much cheaper to do one 2-element allreduce than two 1-element allreduce
- Example - combine allreduce in algorithm with error or convergence values
Bulk synchronous programming model http://www.bsp-worldwide.org