Summer 2003, Vol. 1, Issue 2
in this issue:
We will continue
to build upon the fundamental values that have driven SHARCNET’s
The SHARCNET Board of Directors looks forward to the challenges and successes ahead.
Nils O. Petersen, Chair
|Executive Director's Message|
|Message from the Chair of the Board|
|At the speed of light: New optical connection could position SHARCNET among top 50 in the world|
|World-class researcher finds home as SHARCNET Chair|
When SHARCNET was formally launched in June 2001, our vision was to establish a world-leading, multi-university and college, interdisciplinary institute with an active academic-industry partnership enabling forefront computation research in critical areas of science, engineering, and business. We are achieving that vision, and SHARCNET is having a significant impact on the speed at which research is conducted in Ontario, and the quality of said research.
SHARCNET provides a cost-effective platform for the creation and sustained growth of HPC in Ontario, and is part of a larger strategy to establish Canada as a world-class HPC centre. What makes SHARCNET unique is its consortium of equal institutional partners, who alone, could not afford to establish the technical and human infrastructure provided by SHARCNET.
I would like to take this opportunity to thank the University of Waterloo, Brock University, and the University of Ontario Institute of Technology, who participated in our March proposal to the Ontario Research and Development Challenge Fund (ORDCF). These institutions, along with York University, are also part of our recent proposal to the Canada Foundation for Innovation (CFI). We look forward to welcoming these institutions as new SHARCNET consortium members in the near future.
As we move ahead, SHARCNET will increase its computational capacity, expand the services we provide to researchers, incorporate new partners (both academic and industrial), and realize our vision of establishing of a world-leading computational grid in Ontario.
With our connection to the ORION network, SHARCNET researchers will have the opportunity to use the combined power of our three main computational sites to enhance their already innovative, globally-competitive research.
I am proud to be a part of this team as we move ahead into the future of high performance computing.
|A clear advantage: Researcher uses SHARCNET for speed and efficiency|
On June 18, SHARCNET officially launched its dedicated 1 Gigabit/second connection to the Optical Research and Innovation Optical Network (ORION), which links SHARCNET’s three main computing sites at The University of Western Ontario, McMaster University and the University of Guelph.
This high speed optical connection places SHARCNET’s computational power among the top 50 HPC sites in the world. Combined, the three sites will have the processing power of over one Teraflop. The term “flop” refers to “floating operation per second”, a recognized measure of processing power used by Top500.org, the project that tracks the world’s most powerful computer installations. One Teraflop is a trillion operations per second.
SHARCNET Western System Administrator, Gary Molenkamp, expressed his enthusiasm about the globally-competitive power of this innovative connection.
"The addition of the ORION network will allow us to take the power of SHARCNET’s geographically distributed sites and combine them to produce a computational infrastructure equivalent to those of the top 50. It is an extremely exciting prospect."
ORION, which will be officially launched on June 26, connects more than 50 Ontario institutions and organizations with other local and global networks. A unique partnership with Bell and Nortel, this network joins 21 communities through more than 3,700 kilometres of optic fibre province wide – one of the largest and most powerful research networks ever built.
The SHARCNET connection to ORION is an exclusive network dedicated to
Carmen Gicante, SHARCNET Executive Director, stresses the network’s important role in facilitating academic-industrial partnerships.
“We are extremely pleased to participate in this innovative provincial initiative with other leading organizations and institutions. It is a strong indication of our collective commitment to the development of Ontario’s knowledge-based economy.”
As part of this strategic partnership, ORANO , the not-for-profit company that owns and operates the ORION network, and SHARCNET will be collaborating to provide the opportunity for SHARCNET and ORANO researchers and staff to gain knowledge and experience in the testing, development and management of an advanced optical-based R&D network.
"ORION was created to bring leading-edge network capability to Ontario's publicly funded research and education community," explains Phil Baker, President and CEO of ORANO. "This partnership with SHARCNET is the first of many innovative initiatives that ORION will now make possible throughout Ontario."
SHARCNET’s recent proposal to the Ontario Research and Development Challenge Fund (ORDCF) proposed to augment this connection to 10Gigabit/second between the four large clusters at Western, Waterloo, Guelph and McMaster, and extend 1 Gigabit/ second connectivity to the computational clusters at the University of Windsor, Wilfrid Laurier University, Brock University, York University, University of Ontario Institute of Technology and Sheridan College.
Until then, SHARCNET researchers will be busy with the existing Teraflop of power.
must be created from knowledge.
This quotation by Galileo can be found on the web site of Dr. Martin Müser, University of Western Ontario Assistant Professor and SHARCNET Chair.
Dr. Müser has worked in world-class research groups at Columbia and John Hopkins Universities, and in Germany at the University of Mainz, where he obtained his Ph.D. However, he admits that he is impressed with SHARCNET’s unique approach to researcher support.
According to Müser, whose field of research is a triumvirate of Materials Science, Statistical Physics, and Condensed Matter Physics, SHARCNET’s technical and human infrastructure is world-class.
“This is really the way it should be,” he says. “High Performance Computing Consultants and on-site System Administrators make SHARCNET a much more efficient facility than any I am acquainted with.”
Müser was attracted to Canada and to Western by the SHARCNET Chairs program, through which he was appointed Chair of Applied Mathematics in July 2002. This initiative was designed to attract highly qualified personnel to SHARCNET institutions and has provided a foundation upon which many of SHARCNET partners are building world-class research expertise. The program provides bridge funding for new faculty in four key areas: computational materials, computational finance, biocomputing and HPC tools. Currently, 15 Chair positions have been allocated, and ten have already been filled.
Müser’s research focuses on three main areas: tribology, quantum simulations and the simulation of silica and related materials.
“I play with numbers,” he quips, referring to his work in tribology, the science and technology of controlling friction and wear. Müser’s tribological research has applications in the design of nanomechanical devices
His work with quantum simulations focuses on the way materials properties are affected by quantum fluctuation, such as such as electronic waves in disordered materials.
Working with silica and related materials, Müser studies the effects of extreme conditions, like pressure, on minerals. This work may lead to the discovery of new types of materials, previously unknown to modern materials researchers.
Müser’s work in all of these areas is 90% computational, and therefore largely dependent on SHARCNET. Though he has worked in world-class North America and European research facilities, Müser is enthusiastic about the potential for research and advancement afforded by SHARCNET.
“Thus far, the SHARCNET system has been extremely helpful, and continues to be promising in my own work, as well as that of my students and post-docs.”
Müser has plans to collaborate with members of Western’s growing pool of highly qualified computing experts, including Professor Tom Woo, an internationally recognized leader in computational chemistry who has also recently relocated to Ontario to leverage SHARCNET’s research power.
“Western is well on its way to becoming a strong computational centre,” predicts Müser.
“But,” he adds, “Their success is largely dependent on SHARCNET. The infrastructure and support this facility provides is a tremendous asset.”
In August, Müser will be the host of a SHARCNET-sponsored conference on Computer Simulations of Soft and Nano/Meso Scale Materials at the University of Western Ontario. The conference aims to bring together Canadian and international experts to exchange information on the subject of atomistic computer simulations and the role they play in improving the understanding of condensed matter systems at nano- and mesoscopic levels.
For more information, please see: www.sharcnet.ca/CMSC03
Richard Feynman, the renowned Nobel Prize-winning physicist, believed that turbulence was "the last great unsolved problem of classical physics."
And so does Maurizio Quadrio.
Quadrio, a Ph.D. in aerospace engineering, is visiting The University of Western Ontario from Politecnico di Milano, Italy, sponsored by the SHARCNET Senior Visiting Fellowship program. The aim of the program is to attract and support world-class, "distinguished" researchers who wish to visit a SHARCNET institution for an extended period.
Quadrio’s fellowship sponsor is Dr. J.M. Floryan, a North American specialist in Fluid Dynamics and an Engineering Professor at the University of Western. The two collaborated to explore new mechanisms for turbulence control and on adapting Dr. Quadrio’s algorithm to grid computing.
Says Floryan, “Dr. Quadrio is a recognized expert in the theory of turbulence and in direct numerical simulations of turbulent motions. He has developed a unique concept in parallel computing, where both the data and the computing are distributed, resulting in a significant reduction of data transfer.”
Quadrio’s research at SHARCNET has focused on the study of turbulence through Direct Numerical Simulations (DNS). According to Quadrio, DNS of turbulence is currently one of the most demanding problems in the field of engineering in terms of computational resources.
“When working with DNS, more computational resources equal more possibilities,” he states.
Quadrio, who began his fellowship at Western in January of this year, has been studying turbulence and turbulence control for the last fifteen years. He completed his Ph.D. in a joint program between Politecnico di Milano and the von Karman Institute for Fluid Dynamics, a NATO-supported institution specializing exclusively in aeronautics and aerospace, environmental and applied fluid dynamics, and turbomachinery and propulsion.
Applications of turbulence research include the reduction of aerodynamic drag on an automobile or a commercial airliner, or improvement of fuel efficiency in an engine. An understanding of turbulence can also be applied to physiological mechanisms, such as an understanding of the flow of blood in the heart.
This world-renowned turbulence expert has good things to say about SHARCNET and its enabling computational power.
“In six months I have completed research that would have otherwise taken much more time, and with good results, in some cases, resulting in new projects,” says Quadrio.
He is also optimistic about Canada’s computing future, to be led, he predicts, by SHARCNET.
“There is tremendous potential for growth within these facilities,” says Quadrio. “SHARCNET will ensure a positive future for HPC in Canada.
And of the SHARCNET Fellowship program, “I could not study in
Canada without SHARCNET resources – they have played a key role.
Without funding, there is no opportunity, and without resources, there
is no research.”
Lasers and CD Players:
Photonics and its related applications may not be at the top of your dinner table conversation list, but if you have used a laser printer or CD player recently, perhaps they should be!
The research group of Dr. Marek S. Wartak at Wilfrid Laurier University studies photonics, specifically the modeling, design, and simulation of semi-conductor lasers. Semi conductor laser diodes (a light-emitting device designed to use stimulated emission to form a coherent light output) are essential components in CD players and laser printers, as well as in fiber optic transmitters.
According to Wartak, a leading researcher in the field of simulations of multiple quantum well semiconductor lasers, photonics are and will continue to be key elements of the science, telecommunications and information technology sectors of the next decade.
“The fabrication process of these devices is a long and costly one,” says Dr. Wartak. “Modeling plays an important role in these developments by shortening the design time for new products. Thus, the advancement of design and simulation tools for those devices plays an increasingly important role today.”
In order to facilitate this advancement, new methods and approaches are needed.
With support from SHARCNET, Wartak’s group has
recently developed the first version of a new model to describe semiconductor
and are in the process of applying it to study quantum well semiconductor
lasers. A quantum well is the region of a laser where the density of
electrons is very high, resulting in increased lasing efficiency and
reduced generation of heat. The model is the first of its kind in the
An approach such as Wartak’s shortens the development time of new devices and, therefore, reduces the overall price of finished products.
Wartak collaborates with world-leading experts, including Dr. P. Weetman, a SHARCNET fellow, Dr. W. Beres of the National Research Council, and Dr. P. Rusek of Wroclaw University of Technology, Poland. He has published over 100 papers, has served as a lightwave technology consultant and is credited for significant contributions in multiple quantum well laser research.
Along with colleague Shaowen Song, Wartak led the development of a four-year degree program in optical science and photonics at Wilfrid Laurier University, the first of its kind in Canada.
Wartak has plans to commercialize the results of his research with the
Waterloo-based company Lasma Inc within a year.
When Gordon Drake began his research on the theory of atomic structure in 1968, he was limited by the available computational technology.
In 2003, Drake, now Chair of the Department of Physics at the University of Windsor, is combining high performance computing with high precision theory of fundamental atomic systems.
“An atom is like a miniature solar system with a nucleus and planetary electrons,” explains Drake. “Our goal is to find essentially exact solutions to the Schroedinger equation for atoms containing just two or three electrons.
The Schroedinger equation basically states that the kinetic energy plus the potential energy is equal to the total energy.
According to his peers, Drake has set a new standard for high precision
atomic theory. His results provide tools for performing more accurate
measurement of short-lived atomic nuclei that do not live long enough
to be studied by other more traditional means.
“Ultimately,” says Drake, “this will contribute to better standards of time and efficiency; to the accuracy of technologies like GPS, for example.”
Drake’s work on SHARCNET does not focus on its traditional parallel computing application, but rather makes use of its speed and efficiency to perform multiple precision arithmetic functions.
“Our research encompasses the simultaneous progress of theory and computational capability,” says Drake. “The architecture of the SHARCNET machines gives them a clear advantage over most other machines.”
Drake holds a B.Sc from McGill University, a M.Sc from the University of Western Ontario and a Ph.D from York University. He was a post-doctoral fellow at Harvard before joining the faculty in Windsor in 1969. His work is done in close collaboration with many experimental groups at laboratories around the world, which include York, Harvard and Florence, as well as with the TITAN project at TRIUMF, GSI in Germany and Argonne National Laboratory in the U.S.
To augment his enormously-successful research, Drake is in the process of creating parallel versions of his precision code that will further enhance the unique advantages of SHARCNET for this kind of work.
“SHARCNET is really creating interesting new opportunities for development and expansion of research capabilities,” says Drake. “It is revitalizing the research community.”
He also stresses the opportunities offered to SHARCNET researchers through initiatives like the Fellowship Program.
“It is not just computational hardware,” he confirms. “SHARCNET
programs, such as the Fellowship Program, provide alternative sources
to support students and researchers at all levels. It is the people that
are so important. They enable the research.”
Schedulers, Statistics and SHARCNET:
I am one of the SHARCNET system administrators. My duties include the day-to-day operations of the computer systems located at the University of Guelph. Some of my tasks include the management of the hardware, installing and upgrading software and the creation of user accounts.
All the system administrators are also involved in the development of software tools for use by SHARCNET researchers and staff. I recently completed work on a set of scripts that allow all SHARCNET sites to track their system usage. This data is stored in a common database that allows for reports of all kinds to be generated.
We need to track this usage data for a variety of reasons. One of the main reasons we need to track these statistics is because of SHARCNET’s involvement with C3.ca. Their website describes their organization as “a not-for-profit association formed by and on behalf of: users of High Performance Computing (HPC); resource providers (RP) of hardware, software and people services; and, other interested parties that wish to be associated with HPC for the benefit of science, engineering and the arts.” C3.ca requires that each resource provide make available up to 20% of their computing resources to external users. With all the usage data stored in a common database we are able to easily monitor SHARCNET’s external usage.
This task was complicated by the fact that we use two different schedulers on our systems. We use LSF from Platform Computing for accepting job requests and maintaining job queues and RMS from Quadrics for executing parallel jobs using the quadrics switch. These two systems both store usage data but in different formats, and they each track slightly different information.
The first step was to decide on what values we would track for each job. In order to help diagnose some hardware problems it helps to have all the statistics for each job. The problem with tracking everything is that the database would become large and most of the data would never be used. The decision was made to only track the basic usage statistics in the common database. This includes the user that submitted the program, date submitted, date completed, CPU seconds used, seconds suspended and the machine that executed the job. The scripts gather all the statistics available so that everything can be stored locally with only the minimum data being stored in the global database.
I wrote the scripts using Perl. These programs need to be able to parse the log files from the two schedulers and Perl is exceptional at text processing. The scripts will soon be run daily on each cluster with the usage data available on SHARCNET’s website (http://www.sharcnet.ca).
SHARCNET continues to build on the success of the Chairs program and has recruited five additional Chairs since December 2002.
Hermann Eberl joined the Department of Mathematics and Statistics at the University of Guelph effective January 1, 2003. For the past three years, he has held the post of Scientist at the Institute of Biomathematics and Biometry at the German National Research Center for Environment and Health, where his primary research has dealt with mass conversion and growth of spatially heterogeneous biofilms.
Matheus Grasselli joined the Department of Mathematics and Statistics at McMaster University as of May 1, 2003, and works in the area of Financial Mathematics. His research interests include utility based hedging in incomplete markets, classical information geometry and applications to fluid mechanics and quantum information geometry.
Bartek Protas, recruited to Canada from the University of California, will join the Department of Mathematics and Statistics at McMaster University on August 1, 2003. Dr. Protas' works in the area of HPC tools with scientific interest focused in Computational and Theoretical Fluid Dynamics.
James Wadsley's appointment in HPC Tools at McMaster University is effective July 1, 2003. Formerly an HPTC Consultant with SHARCNET, James brings to this position a breadth of high performance computing knowledge and experience, especially in the area of Computational Astrophysics.
Eugene Kim will join the Physics Department at the University of Windsor on July 1, 2003 in the research area of Computational Physics and Materials. Dr. Kim plans to initiate a theoretical research program investigating the electronic properties of condensed matter systems. Currently a collaborator with Dr. Erik Sorensen, also a SHARCNET Chair at McMaster University, Dr.Kim will be a valued addition to the SHARCNET Chairs Program.
The Chairs program, which represents SHARCNET’s long-term investment in HPC personnel, provides bridge funding for new faculty at SHARCNET institutions. Recruitment is targeted in key areas of computational materials, computational finance, bio-computing and HPC tools. Of the 15 Chair positions allocated, these appointments bring the recruitment total to ten. The other Chair positions are in various stages of recruitment and we expect to announce additional incumbents over the coming months.
and Xeon clusters added to the SHARCNET arsena
In 2002, the alpha-based SHARCNET clusters using the Quadrics interconnect showed high system usage and long waiting times within the queuing system for applications requiring large amount of parallel computing resources. Monitoring of the type of applications being run also demonstrated that there was a large percentage of latency tolerant applications being run on the systems designed for these clusters. These applications were better suited for an Ethernet-based interconnect environment, and based on the quantity of these application runs, there was a demonstrated need for new computational infrastructure.
Two new clusters were added to the SHARCNET computing capabilities: a 128 Itanium 2 processor cluster located at University of Western Ontario, and a 64 processor Xeon cluster located at McMaster University. The Itanium 2 cluster is composed of 64 dual processor HP rx2600 servers running at 900Mhz. Each system has 1 Gb of ram and a Gigabit Ethernet interconnect. The new Xeon cluster has 32 dual processor Xeons running at 2.4Ghz, with 1Gb or 2Gb of RAM per system and a 100 Mb Ethernet interconnect. Both clusters use Linux based OS for simple integration with existing SHARCNET clusters. The Itanium 2 cluster uses the LSF queuing system that is available on the alpha based clusters to allow for the multi-cluster environment and to duplicate the working environment for researchers.
With these two new clusters added to the arsenal of SHARCNET's computational
resources, queue wait times have been reduced for all applications, and
those researchers needing large amounts of parallel computation power
do not have to wait as long for resources as they would have had to wait
in the past. At the same time, researchers have the opportunity to migrate
their existing applications to the Itanium 2 architecture for those still
needing 64 bit processing, and to the Xeon architecture for 32 bit performance.
David joined SHARCNET in February from the Department of Computing and
Information Science at the University of Guelph. His background includes
massively parallel systems, software development, compiler optimizations
and teaching. He has significant experience with software and system
development in C, Java, C++ and a variety of scripting languages and
has taught undergraduate courses in compiler construction, software engineering
Doug joined SHARCNET in February from the Department of Mechanical Engineering at the University of Western Ontario. His research interests lie in the field of Computational Fluid Dynamics relating to Boundary Layer type flows. At SHARCNET, Doug is interested in assisting users migrate their codes to HPC clusters, making scientific packages more readily available, and devising efficient algorithms to best utilize SHARCNET systems
Lindsay joined SHARCNET in March from her former role as Marketing Communications
Coordinator at OneMind Inc, a London-based marketing and design consultancy.
She brings to this position her extensive experience in marketing communications,
public, stakeholder and media relations, web and information architecture
design, and the design of print and electronic publications.
The Shared Hierarchical Academic Research Computing