Learning

Linux clusters: High-performance computing

 


Linux clusters are not a one-trick pony, so don't dismiss them as scientific lab tools only. This SearchEnterpriseLinux.com guide explores the many uses of

Continue Reading This Article

Enjoy this article as well as all of our content, including E-Guides, news, tips and more.

Linux clusters in the enterprise. Find out why Linux clusters can be your company's best bet for handling storage, applications, high-volume transactions and more. Take a spin though these articles, tutorials, tips, tools, white papers and expert Q&As. Drop me an e-mail to let me know what other learning guides you'd like to see on SearchEnterpriseLinux.com. -- Jan Stafford, Editor.

 

TABLE OF CONTENTS
   Basics
   High-performance computing clusters
   Application clusters
   Storage clusters
   Load-balancing clusters
   High availability clusters
   Clustering on blade servers
   Cluster configuration, management and administration
   Virtualization and clusters

   Other open source cluster projects

 

  Clustering basics  Return to Table of Contents

Glossary (SearchEnterpriseLinux.com, powered by Whatis.com)

Other basic resources

 

  High-performance computing clusters  Return to Table of Contents

General information

  • Cluster how-to
    How to set up a high performance Linux cluster.

  • Linux Clusters Institute
    The LCI site provides links to education and advanced technical training for the deployment and use of Linux-based computing clusters to the high-performance computing community worldwide.

  • Article: Driving Linux cluster performance (Bio-IT World)
    Planning early and choosing the right tools can boost performance by 27%, according to experts cited here.

  • Linux HPC cluster how-to
    This is a specific description of how a lab's cluster is set up. It details the data center aspects and the desktop, laptop and public server aspects of building a cluster.

  • Article: The Linux Business Case: Clusters (CIO Information Network)
    This article discusses the characteristics of high-performance applications that make them a good fit for Linux clusters.

  • Q&A: Tips for maximizing HPC cluster productivity (HPC Wire)
    Tom Quinn, Director of Government Business Development at Linux Networx, provides tips on maximizing cluster productivity.

  • Q&A: Linux cluster breakthrough: The big deal is big bandwidth (HPC Wire)
    Kent Koeninger, product manager in HP's High Performance Technical Computing Division, talks about HP's two-year tuning of the Lustre protocol and how the software solves I/O bottleneck issues in Linux cluster computing enterprises.

  • OpenMosix open source project site
    OpenMosix is a Linux kernel extension for single-system image clustering which turns a network of ordinary computers into a supercomputer.

  • Build a heterogeneous cluster with coLinux and openMosix (IBM.com)
    An experimental case study combines coLinux with an openMosix-enabled kernel to build a hybrid cluster.

  • SGI Partners to Launch 'Bright' Linux Clusters (IT Jungle)
    Clusters that behave like, are preconfigured like, and sold like monolithic systems are called "bright clusters," and supercomputer maker Silicon Graphics is partnering to target this fast-growing market with its Linux-based Altix machines.

  • Article: Dell inks HPC cluster deal with Cray
    This story describes how Dell plans to bolster its HPC Linux cluster business by partnering with Cray Inc., the Seattle-based supercomputer company.

Beowulf

  • Beowulf.org
    Project page for the open source project. Beowulf.org is a collection of resources for the expanding universe of users and designers of Beowulf class cluster computers. These enterprise systems are built on commodity hardware deploying Linux OS and open source software.

  • Beowulf cluster FAQ (Aspen Systems)

  • Beowulf how-to
    This document describes step by step instructions on building a Beowulf cluster.

  • Q&A: Beowulf founder: Linux holds its own in high-performance computing
    Beowulf founder Don Becker reflects on the legacy of Beowulf and its future in this interview. He also takes a hard look at Cray Canada CTO Paul Terry's claims that Linux clusters can't hold their own in HPC situations.

  • Article: Scyld's new cluster has Linux distro, Ganglia inside (SearchEnterpriseLinux.com)
    Scyld Software has added performance monitoring capabilities and hardware and customer support in the newest version of its Scyld Beowulf Series 29.

Scientific applications

Whether or not to use HPC clusters

  • Q&A: Linux clusters for HPC – Why and why not?
    What's keeping high-performance computing users away from Linux clusters? Should they reconsider?

  • Expert advice: What are the advantages of moving to a Linux HPC cluster? (Ken Milberg, SearchEnterpriseLinux.com expert)

  • Q&A: Cray CTO: Supercomputers outshine Linux clusters in HPC, Part 1 (SearchEnterpriseLinux.com)
    Linux clusters can not offer the same price-performance as supercomputers, according to Paul Terry, chief technology officer of Burnaby, British Columbia-based Cray Canada. In this interview, Terry explains that assertion and describes Cray's new Linux-based XD1 system, which will be priced competitively with other types of high-end Linux clusters.

  • Q&A: Cray Canada CTO answers IT pros' HPC questions (SearchEnterpriseLinux.com)
    Is high-performance computing only used in scientific labs? In this interview, Cray Canada chief technology officer Paul Terry answers this question, offers advice on how to choose the right HPC system responds to questions from SearchEnterpriseLinux.com's readers.

  • Q&A: Linux clusters give HPC price-performance (SearchEnterpriseLinux.com)
    High-performance computing (HPC) users are switching from supercomputers to Linux clusters, and enterprise IT shops are getting keen on that idea, too. In this interview, PathScale exec Alex Rublowsky explained why HPC on Linux makes sense. Also, find out which types of enterprise IT shops are checking out and what's ahead for HPC Linux clusters.

  • Linux clusters can have their day vs. supercomputers (SearchEnterpriseLinux.com)
    Linux Networx marketing executive Eric Pitcher makes the case for Linux clusters against supercomputers.

  • Filling in the gaps among Linux clusters (SearchEnterpriseLinux.com)
    Is the price right for a Linux cluster? Would it meet all of your company's needs? Get the scoop in part two of this interview with Linux Networx executive Eric Pitcher.

 

  Application clusters  Return to Table of Contents

 

  Storage clusters  Return to Table of Contents
  • Article: Lowering storage and file-serving costs with network attached storage (NAS) clusters (Source)
    NAS clustering can enable a scalable, fault-tolerant, file-serving cluster for thousands of users, Web clients or computing nodes.

  • Article: NAS clustering beats 1GB barrier (SearchStorage.com)
    Clustering low-cost Linux servers together with a network file system saves significant costs over purpose-built NAS filers; that much is a no-brainer. Now, advocates of NAS clustering argue that the performance of these offerings matches or even outperforms traditional NAS appliances.

  • Case study: Windows flops in online retail project (SearchEnterpriseLinux.com)
    Nowadays, more and more Java-based retail apps are being deployed on Linux. So, when Datavantage had to choose a platform for its new Stored Value gift card transaction-processing solution, Windows wasn't the only option.

  • Article: NetApp to reshape its storage architecture (SearchStorage.com)
    NetApp's Storage Grids will parallel an important trend in the server world: the move away from large symmetric multi-processing (SMP) systems, and toward large server farms made up of commodity Linux servers. That trend is quite evident today in the technical computing space, where NetApp intends to find its first customers for the Storage Grids technology.

  • Article: Bye-bye to bottlenecks (IDM.net)
    Storage software provider Ibrix has introduced its Fusion software suite, an enterprise-class, parallel file system for cluster computing environments that promises to eradicate system bottlenecks.

  • Article: PolyServe thinks outside the box with NAS cluster (SearchEnterpriseLinux.com)

  • Case study: Company bypasses NAS, chooses Linux cluster (SearchEnterpriseLinux.com)

  • Expert advice: How do I back up Linux clustering file systems? (Sam Greenblatt, SearchEnterpriseLinux.com expert)

 

   Load-balancing clusters  Return to Table of Contents
  • Home page: Linux Virtual Server Project
    The Linux Virtual Server is a highly scalable and highly available server built on a cluster of real servers, with the load balancer running on the Linux operating system. The architecture of the server cluster is fully transparent to end users, and the users interact as if it were a single high-performance virtual server.

  • Home page: Ultra Monkey
    Ultra Monkey is a project to create load-balanced and highly available network services, such as a cluster of Web servers that appears as a single Web server to end users.

  • White paper: Craft a load-balancing cluster with ClusterKnoppix (IBM)
    Using Knoppix-based LiveCDs, build your own supercomputing Linux cluster.
  • Article: Load balancing gives Web traffic a green light (SearchCIO.com)

  • Resource: Load balancing Linux clusters (LCIC.org)

 

  High availability clusters  Return to Table of Contents
  • Home page: High-Availability Linux Project (Linux-ha.org)
    The basic goal of the High Availability Linux project is to provide a high-availability (clustering) solution for Linux which promotes reliability, availability and serviceability through a community development effort.

  • White Paper: Introduction to clustering on IBM eServer OpenPower 710 (IBM)
    This article first introduces some general clustering concepts and software and then provides examples of how the OpenPower 710 can be used in the two most common types of clusters: high performance clusters and high availability clusters.

  • Home page: Kimberlite HA cluster project site
    The collaboration site for Kimberlite cluster development. Kimberlite is an open source project for high availability clustering on Linux. The page offers links to documentation and more information.
  • Resource: SGI Linux FailSafe FAQ (SGI)
    Linux FailSafe provides a clustered HA application platform in software.

  • Webcasts: SteelEye LifeKeeper HA clusters (Steeleye)
    Links to Web seminars on this company's high availability solutions on Linux. Requires registration.

  • New competition heats up iSeries HA (iSeries Network)
    New software companies are moving into the HA cluster market. Joining large company solution providers Vision Solutions, Lakeview Technology and DataMirror are SMB product vendors iTera, Maximum Availability, OS Solutions and Trader's. The new companies offer smaller, cheaper HA software that builds upon IBM's remote journaling software.

 

  Clustering on blade servers  Return to Table of Contents

 

   Cluster configuration, management and administration  Return to Table of Contents
  • Q&A: Beowulf founder: Virtualization hot on Linux clusters
    Virtualization is creating a buzz in the IT industry today, but that buzz fizzles when virtualization on Linux clusters is mentioned. That's a shame, says Beowulf Project co-founder Donald Becker, because clusters offer a no-clutter virtualization option.

  • White paper: What is CSM? (IBM)
    Find out more about Cluster Systems Management (CSM) software, which provides a distributed system management solution that allows a system administrator to set up and maintain a cluster of nodes that run the Linux operating system.

  • Tutorial: Installing and configuring rsh, XMPI and more (Microway)
    Here you'll find tips for installing and configuring Microway clusters.

  • Home page: System Imager
    This is the open source project site for SystemImager, software that automates Linux installs, software distribution, and production deployment for Internet server farms, database server farms, high performance clusters, computer labs, and corporate desktop environments.

  • Tutorial: Installing and configuring Condor (Microway)
    This tutorial discusses Condor, a software system that creates a high¬-throughput computing (HTC) environment. It effectively utilizes the computing power of workstations that communicate over a network. Condor can manage a dedicated cluster of workstations.

  • White paper: Introducing General Parallel File System (IBM)
    IBM's GPFS allows users shared file access within a single GPFS cluster and across multiple GPFS clusters.

  • Article: Scali releases new products for multi-platform clusters (Linux HPC.org)
    This is an update on Scali's high performance clustering management product, called Scali Manage.

  • FAQ: Scali Manage
    Answers to questions about Scali Manage node management and other issues can be found here.

  • FAQ: Cluster administration via SSH (ClusterSSH)
    ClusterSSH controls a number of xterm windows via a single graphical console window to allow commands to be interactively run on multiple servers over an ssh connection.

  • Home page: Maui Scheduler
    The Maui Scheduler helps admins manage large-system resources. Maui was designed to batch schedule and launch MPI-based parallel jobs, and can also schedule and run your own custom message-passing jobs.

  • Home page: The Cluster Command and Control (C3) tool suite
    This is the project page for The Cluster Command and Control (C3) tool suite, which implements a number of command line based tools that have been shown to increase system manager scalability by reducing time and effort to operate and manage the cluster.

  • Tip: Clustering in MySQL
    MySQL expert Scott Noyes recommends buying a subscription for MySQL Network if you plan to use MySQL clusters for mission-critical apps.

  • Tip: Distributed computing, clusters and high-availability solutions
    In this tip, clusters and servers expert Don Becker describes basic rules for making network boot servers more reliable -- like avoiding multicasts -- and explains the Mosix approach to cluster architecture. He also suggests ways to avoid data loss when consolidating servers.

  • Expert Response: Consolidation and common sense on the server side
    A Unix-to-Linux expert describes educational resources about distributed computing, clusters and high-availability solutions for a user who is considering a move to Linux from Unix.

  • Tip: Mastering MySQL: Drivers, clustering and remote access
    MySQL expert Scott Noyes describes the tricky aspects of clustering with MySQL and upgrading from MySQL 4.0 to MySQL 5.0.

  • Tip: Setting up MySQL replication for Linux server clusters
    MySQL facilitates Linux server clustering for load balancing and fault tolerance. Create a master/slave configuration with full data replication using the commands in this tip.

 

  Virtualization and clusters  Return to Table of Contents

 

  Other open source cluster projects  Return to Table of Contents
  • Home page: OpenSSI (Single System Image) Clusters for Linux
    The OpenSSI project is a comprehensive clustering solution offering a full, highly available SSI environment for Linux.

  • Home page: Cluster infrastructure for Linux
    This project is developing a common infrastructure for Linux clustering by extending the Cluster Membership and Internode Communication Subsystems from HP's NonStop Clusters for Unixware code base.

  • Resource: Cluster Project Page
    This page provides links to a number of Linux cluster development projects.

  • Home page: Open Cluster Group
    The Open Cluster Group is an informal group of people dedicated to making cluster-computing practical for high performance computing.

  • Home page: OSCAR: Open Source Cluster Application Resources
    OSCAR is an open source project for the development of a fully integrated, easy to install, small-to-midsized high performance clustering solution.


 

This was first published in February 2008