A first look at scalable I/O in Linux commands

Abstract

Data created from and used by terascale and petascale applications continues to increase, but our ability to handle and manage these files is still limited by the capabilities of the standard serialized Linux command set. This paper introduces the Center for Computational Sciences (NCCS) at Oak Ridge National Laboratory (ORNL) efforts towards providing parallelized and more efficient versions of the commonly used Linux commands. The design and implementation details as well as performance analysis of an in-house developed distributed parallelized version of the cp tool, spdcp is presented. Tests show that our spdcp utility can achieve 73 times more performance than its serialized counterpart. In addition, we introduce current work to extend this approach to other tools.