CalTech demonstrated amazing data transfer speeds at the SuperComputing Conference 2011-Moving Towards Terabit/Sec Transfers. A petabyte of data was transferred from the demo booth in Seattle, Washington to the University of Victoria in Canada. That’s 212kms and 1,048,576 gigabytes per petabyte, or 250 million MP3 songs.

The team took advantage of finely tuned servers and the Fast Data Transport (FDT) transfer application developed by CalTech (California Institute of Technology) and Polytehnica Bucharest. FDT is an open source Java application that runs on all major platforms using the NIO libraries, streaming large files across an open TCP socket.

Dell Force 10 Z9000 Layer 3 Switch - 32 Slot
Dell Force 10 Z9000 Layer 3 Switch – 32 Slot Fiber Optic connectors for near-unlimited bandwidth

Mellanox ConnectX PCIe Gen3 100GbE chips were key components for linking workstations to the switchesThe researchers were using unproven, engineering samples. SuperMicro, TYAN and other vendors provided Sandy Bridge-based Server motherboards with PCIe Gen3. Mellanox offered the PCIe Gen3 compliant, dual port 40GE NIC, ConnectX-3.

Not unexpectedly, there were crashes, framing errors, lost traffic. However, the researchers reported that the Brocade MLXe-4 100G switch-router and Dell’s Force10 Z9000 switch provided 120Gbps port channel among them with no packet drops. They were able to write at the speed of 60Gbps on several SuperMicro and Dell Servers on the showroom floor. Dell provided R710 servers to drive the network from Victoria.

The exhibitors were able to use SCinet, a powerful and sophisticated network to demonstrate their project. For the event, SCinet connected multiple wide area circuits to the exhibit floor with a capacity of more than 450 gigabits per second using multiple brand new 100G connections to the major national research networks. They used one OTU4 (Optical Transport Network) link between a Ciena OME6500 in SCinet and another OME6500 in BCNet in Canada. They transmitted one petabyte of particle physics data in 24 hours at sustained speeds of 95 gigabits per second, over a single optical channel.

A senior director for external research at Ciena, Rod Wilson commented on the importance of the project: "High-capacity, high-performance networks play a critical role in global research and education efforts, enabling organizations to collaborate on research, development and leading-edge technology discoveries."

PetaByte Demo at SC11 Overview
100GE Wide Area Network Demonstration Model

Jim Roche, president and CEO of CANARIE, Canada’s Advanced Research and Innovation Network, explains the need for the CalTech team project. "Demand for bandwidth continues to grow as scientists embark on research that is increasingly data intensive and collaborative."

CANARIE is a dedicated network of high-speed, fiber optic cable that stretches 19,000 km across Canada and connects researchers in schools, hospitals and labs worldwide. Roche says: "CANARIE has documented a 284 per cent increase in the amount of data transmitted on our network since 2007." This trend is expected to accelerate as scientists at global research sites take part in more data-intensive research. It is hoped that ease and speed of collaboration will lead to enhanced productivity and accelerated discoveries.