GridFTP and Yoga
Here in the land of GridFTP developement we try to be as flexible and extensible as possible. We created an XIO library entirely around this idea called GlobusXIO . This has allowed us to much more easily add new protocols to GridFTP, like UDT and Phoebus. It has also allowed us to achieve greater than network speeds by use of compression.

While that has provided us with karate-man center splits type flexibility, it was not enough for us. We also created the DSI (Data Storage Interface) to GridFTP. Another dynamically loadable plug-in to GridFTP! This has allowed GridFTP servers to be front ends to complex filesystems like SRB, HPSS and further, allowed us to turn a single GridFTP server into a striped gridftp server as well as a load balancing dynamic backend server.

Ah, now we are starting to feel the burn. Yet even with 2 places for dynamically loadable modules, we did not feel pluged-in enough. We wanted to give our admins more knobs to turn, and excited contributing developers more levers to pull. So we purused our code for proepr places to carve out clean abstractions. After months of discussing this over many beers we ultimately found the right spot, the ECM (event callout mumble--the plugin formerly known as ACL). We had a few cups of coffee and then wrote the interface that allowed for NeST to control disk storage reservations and allowed for interaction with CAS and other AUTHZ services. Now we are starting to get our Bird of Paradise on!

But the problem here is, these are all supported features. Thats no fun. They will all just work and if they don't one only needs to send us email for help. What about the poor hacker out there looking to tinker? What have we left for the poor souls that are not happy unless subversively trick a web server into being a TCP router with the long unix pipe commands? Those poor people cant deal with know and easy to use configuration options! T-Rex wants to hunt!

Well, because we care about the hacker (and because we wanted to leverage some quality tools like ssh) we created the GlobusXIO popen driver.

The popen driver allows you open pipes to the standard IO of existing programs. This allows you to leverage programs much in the way that you can with UNIX pipes (actaully, EXACTLY in that way).

So say you want to tar up a directory on the fly and send it via GridFTP? No problem, pipe it through /bin/tar! Want to compress a file before sending it you say? Pipe that sucker through gzip. Whats that? you want your ASCII text files to appear as though they were written by The Swedish Chef ? We are here for you! Just popen /usr/games/chef and your speaking fluent muppet baby!
Examples here
Here we will show you how to do the following

    * tar a directory on the fly
    * Tar it up on the source side, and untar it on the dest side. translate english text to foreign languages at the source

How to do it
So first and foremost, none of this is really support. It is a neat trick you can do with Globus GridFTP servers. It will likely work, and work everytime, but it is a bit wacky to support on every system, and it could cause security headache (popen lets authenticated users run stuff remotely). For these reasons we leave this to ambitious fun loving sysadmins to play with, and will work with other users in more supported ways to achieve their needs.

Whitelist the popen Driver
For security reasons it is necessary that you explicitly allow any xio driver that will run inside the GridFTP server. You can't have remote clients inserting just any dynamcially loadable module, now can you? To allow the globus-gridftp-server to load the popen driver you use the -fs-whitelist option. This tell the server which globus xio modules it can load. For our purpuses here use:

    globus-gridftp-server -fs-whitelist popen,file,ordering

This will allow clients to use the default file driver (for doing regular stuff), and the popen driver for the fancy stuff here. You will also notice that the ordering driver is listed. This is there because when sending data to a pipe it needs to be in order. Often GridFTP data streams are not in order, we seek into the file to reorder them. Unfortunatey we cannot seek into popen'ed programs. But fortunately we have the ordering driver which will (up until memory contraints) re-order things for us.