WINTER 2005, Vol. 3, Issue 1
in this issue:
Our vision of an integrated, multi-disciplinary and inter-institutional research institute, enabling globally-competitive research and discovery, continues to gain momentum.
I am pleased to have assumed the role of SHARCNET Board Chair, replacing Dr. Nils Petersen, a SHARCNET founder and long-time advocate, as he moved on to the position of Director General of the NRC’s National Institute for Nanotechnology (NINT) in Edmonton. On behalf of the Board and the entire SHARCNET community, I would like to express our immense appreciation and gratitude for Dr. Petersen’s pivotal role in both establishing SHARCNET and helping to develop it into a leading research centre.
SHARCNET is a dynamic research-driven organization that strives to support the best research and to promote applications of high performance computing to challenges and opportunities in society. We continue to grow and change to reflect this ethos. On November 1, the Board instituted a new organizational model which places the Scientific Director, Dr. Hugh Couchman, a renowned astrophysicist, in a leadership role for the organization.
I would like to take this opportunity to sincerely thank Carmen Gicante, former Executive Director, for his contributions and leadership over the past two and a half years as SHARCNET was being established as a premier HPC centre on both the national and international stage.
There are also many new faces on the Board itself. Dr. Mike Bauer, Chair of Computer Science at The University of Western Ontario, has joined the Board to act as a representative for the researcher community; Andrew Bjerring, President and CEO of CANARIE will act as the Board’s national representative, and Ian Hardgrove, President and General Manager of 3M Canada, will bring a new and unique perspective from the private sector environment.
I look forward to working with my fellow Board members, the SHARCNET researcher community, our staff, partners and funding agencies to ensure the continued success of the organization. We will also be striving to ensure that SHARCNET complements and strengthens the high performance computing capacity throughout Ontario and Canada.
|Message from the Chair of the Board|
|Scientific Director's Message|
SHARCNET and the science of the small
|SHARCS invade S2004|
than the sum of its parts
|SHARCNET-researcher partnership offers new horizons in hardware, computing capability|
SHARCNET to host HPCS 2005
Technology-driven, Research focused: SHARCNET and Fanshawe College
SHARCNET, and much of Canadian HPC, is undergoing an important transition. Like SHARCNET, many consortia are into their second round of CFI and provincial funding and moving from successful start-ups to growing and maturing organizations with track records and increased responsibilities to a burgeoning researcher base. The success of SHARCNET and others has led to increased attention by funding agencies and governments to the needs of the research community for HPC and to the challenges of sustaining an infrastructure that is dependent on a rapidly evolving technology.
This summer SHARCNET was awarded $19.3M by the Ontario government to match our CFI award and this allowed us to move forward with planning for the next installation of computer infrastructure. Largely because of significant technology evolution over the 18 months since our application, it was decided to go out to a competitive request for proposals. In parallel with vendor and equipment selection over the next quarter we are undertaking the required machine room renovations and coordinating planning across our 11 partners, all the while continuing to provide a high level of availability and support on existing systems.
The last two years has seen a growing awareness among the user community and provincial and federal funding agencies of the need to develop strategies for sustained funding for HPC infrastructure and personnel. HPCVL and SHARCNET are involved in developing strategies to ensure efficient and coordinated provision and use of resources in Ontario. Nationally, the C3.ca long-range plan in which SHARCNET has been closely involved is nearing completion. These initiatives are leading to a new understanding of the need for sustained support of HPC and new responsibilities for how large HPC use is coordinated.
Operationally, all of our systems have been running at full capacity for well over a year and demand continues to rise, highlighting the pressing need for new infrastructure. Contributions of equipment to SHARCNET by researchers with CFI New Opportunities and other funds have added over three hundred processors since the Spring and helped alleviate some of the most acute short-term demands.
Our key short-term technical thrust is to develop a framework for tight integration of all SHARCNET resources with the goal being seamless availability of any resource to all SHARCNET researchers independent of the location of either. We aim to anticipate the installation of the new hardware so as to quickly realize its full capability. The fundamental aim of SHARCNET remains, of course, to enable forefront research using HPC, and the opportunities promised by the new infrastructure are truly exciting.
Hugh M.P. Couchman
Nanotechnology, the science of creating new materials at the atomic and molecular level, is a hot topic these days. The potential to yield the discovery of new, previously-unknown materials would have significant applications to any number of industries, including computers, electronics, aviation, aerospace, automobiles, and more.
Using the Wobbegong (Wobbe) cluster at McMaster, an installation co-funded by SHARCNET and McMaster researchers Erik Sorensen, Catherine Kallin, James Wadsley, Walter Craig and Dmitry Pelinovsky, Sorensen and SHARCNET System Administrator Mark Hahn are bridging the gap between fundamental physics and future nanotechnological innovation.
According to Sorensen, a condensed matter physicist, future electronic circuits are likely to be “much, much smaller than today’s” and “could very well be made atom by atom or molecule by molecule”. Therefore, he says, quantum mechanics (a field of physics that deals with phenomena on extremely small scales) will play an increasingly prominent role in determining how a given circuit functions.
Sorensen’s research focuses on the way atoms interact using the principles of quantum mechanics, in particular how they are influenced by impurities. The presence of a magnetic impurity is capable of controlling whether or not a current can flow through a circuit. Though the question of how this effect takes place is of great interest (because of the potential to produce the smaller and yet much more powerful circuits of the future) current analytical calculations yield only an incomplete answer.
Before the introduction of the Wobbe cluster at McMaster, says Sorensen, completing this research was impossible.
“These calculations are exceedingly complicated and only now possible by exploiting the complete parallel capabilities of Wobbe. We can now include more atoms in our simulations than would have been imaginable using conventional computer systems.”
In order to determine the current flowing in a small circuit, Sorensen has to diagonalize a large so-called “sparse matrix”. Such a calculation requires not only an enormous amount of computing power, but also an immense amount of memory (211 Gigabytes), 200-400 times that of a regular desktop computer.
Using 64 of Wobbe’s 192 processors, Sorensen was able to diagonalize a matrix size of 3,929,717,484 in only 110 hours, just under 5 days.
The results of the calculation, which have been submitted to Physical Review Letters, are state-of-the-art: only a few physics groups in the world, including one in Switzerland (Drs. Andreas Läuchli and Matthias Troyer), have been able to achieve similarly-sized calculations, but they were completed on shared memory systems that cost many times more than the Wobbe cluster.
McMaster System Administrator Mark Hahn, who in large part modified Wobbe’s operating system to make this calculation possible, emphasizes not only the importance of the technology, but also the collaborative nature of the SHARCNET community itself.
“Much of the point of SHARCNET is resource pooling,” he explains. “Researcher contributions to existing SHARCNET infrastructure have served both individual research projects and the entire community in a very synergistic way. The Wobbe cluster wound up being much more powerful than a series of individual systems would have been, and thus any single researcher can now use more resources than he or she ever could have afforded alone.”
Nearly all aspects of the calculation were highly computationally challenging, says Sorensen, and, he adds, would have been nearly impossible without Hahn’s expertise and the collaborative nature of the SHARCNET support model.
“I would like to emphasize Mark’s great help in getting all this to come together. This project has only been possible due to his enormous expertise and availability at all strange hours – late nights and weekends. Having personnel like this is something that only SHARCNET offers.”
For more on the Wobbegong cluster, please see: SHARCNET-researcher
partnership offers new horizons in hardware, computing capability,
ancient Greek philosopher Heraclitus once wrote: “All things
come out of the one, and the one out of all things.”
Kim, whose interests are in the electronic properties of condensed matter systems, is currently focusing his energy on three main topics: molecular electronics, disordered electronic systems and quantum impurity problems, and quantum magnetism. He is most proud of work he has done on quantum impurity problems, especially his realization that exotic physics involving magnetic impurities – phenomena that have been sought after for over 20 years – could be observed in a carbon nanotube.
Because theoretical descriptions of condensed matter systems are often rather subtle and difficult, numerical simulations are crucial to understanding these systems. As a result, his group performs all of their numerical simulations using SHARCNET’s facilities. “It’s really a luxury to have access to some of Canada’s most powerful computing facilities,” Kim confirms.
Kim has also made contributions to quantum magnetism, and is particularly fond of one of his papers on the topic written with his collegues Solyom and Fath from Budapest. He describes this work as “the most enjoyable collaboration I ever had”.
Kim received his Bachelor of Science degree from the University of Illinois, his Ph.D. in physics from the University of California, Santa Barbara, and completed his post-doctoral work at McMaster University and the University of Toronto.
Eventually, Kim gained many frequent flyer miles traveling abroad to Budapest, Madrid and Seoul, Korea. He was able to experience life in diverse locations and associate with other great researchers across the world – some who became splendid friends.
While Kim values all of his experiences abroad, he recalls his interactions with Solyom from Budapest – a scientist who has made seminal contributions in his field – as having had considerable impact on his research, and more generally, his life.
“In hearing him describe his work, I realized it came out of a sincere fascination of nature,” Kim says. “More specifically, it arose from asking questions of nature, similar to how we used to ask questions when we were little.”
In 2003, SHARCNET gained a Chair and researcher with a passion for science, and someone who will never stop asking questions about the world humanity inhabits.
“The best things have happened,” he says, “when
I’ve been successful at freeing my mind and being a kid again.”
Imagine a more environmentally-friendly motor oil; one that not only protected your engine from wear but one that was also compatible with a lighter, more fuel-efficient engine material.
Imagine the development of lighter, yet more durable plastics for everything from medical devices to automobile instrument panels.
Imagine the applications of a computer chip that is a thousand times smaller than current electronic devices but with a million times more power and storage.
Tom Woo, an Assistant Professor of Chemistry at The University of Western Ontario is using SHARCNET to make tremendous leaps toward these exciting possibilities.
Woo’s research lies in the area of computational quantum chemistry and molecular modeling – the investigation of chemical systems at the atomic and electronic level. His work has far-reaching applications in the design of new materials and the development of new chemical processes to improve material composition, safety and utility.
Synthetic materials and chemicals have an enormous impact on our quality of life. Examples include the designer plastics used in joint replacements, Teflon on frying pans, and new blood pressure lowering drugs. Ultimately, the properties of a material can be traced back to how the atoms that make up the material are arranged and connected. For example, the reason diamonds are so strong is related to the 3-D network of chemical bonds that connect the carbon atoms of a diamond together. Thus, understanding how a material functions at this microscopic level can be enormously beneficial in the development of new materials.
The production of synthetic materials (i.e. plastics) involves chemical reactions that arrange the atoms in a material in a specific way. These reactions are often developed on a trial and error basis because experimental probing at the atomic level is often very difficult or impossible. Yet, it is precisely this sort of detailed knowledge that harbours the greatest potential to design new materials or develop more efficient chemical reactions to create these materials. This is why the marriage of chemistry and computational techniques has proved so lucrative. Just as computer simulations have been used to design new more aerodynamic vehicles, computer simulations at the atomic level are being used to design new materials and chemicals.
However, to accurately model systems at the atomic and electronic level the laws of quantum mechanics must be utilized, and such simulations are extremely computationally intensive. Using 48 processors simultaneously on SHARCNET’s Greatwhite Cluster, Woo’s group is able to simulate the high pressure compression of anti-wear materials that protect a car’s internal engine surfaces and understand how they function at a fundamental atomic level. Such simulations, which may lead to the development of anti-wear materials that allow the manufacturing of more fuel-efficient cars, would not have been possible in Canada without SHARCNET.
“SHARCNET has made possible new investigations that were simply not feasible before,” says Woo, a Premier’s Research Excellence Award (PREA) Winner. PREA’s are awarded annually by Ontario’s Ministry of Economic Development and Trade, but only to those who have a track record of excellence in their discipline and those who are engaged in research that has a demonstrated potential to make a difference.
General Motors, Nova Chemicals, Mitsui Chemicals and Defence Research and Development Canada have all taken advantage of Woo’s tremendous breakthroughs in computational quantum chemistry and molecular modelling. For example, Woo helped initiate a molecular modelling programme at Nova Chemicals, a leader in the commodity chemical industry, which has aided in the development of new catalysts that are now used commercially to produce plastics in mass quantities.
Woo’s work has been published in the widely cited
Journal of the American Chemical Society and other international
He is also a three-time SHARCNET Fellowship Program recipient.
a newly created position for a Dean of Applied Research and growing
research proposals, both the administration and students of Fanshawe
College are leveraging SHARCNET’s infrastructure to aid in their
Strybosch, also the SHARCNET site leader at Fanshawe, incorporates SHARCNET’s heterogeneous architecture and software development in the academic undertakings of his students. Although HPC equipment is not a necessity for the work of Strybosch’s students, its presence has furthered their ideas and potential to an exceptional level.
“Never, before SHARCNET, did any student choose topics like genetic algorithms. They’ve chosen to work with issues like problem heuristics, hybrid algorithms and now recognize the need to research papers in major publications.”
As Fanshawe’s applied environment becomes more research-oriented, students at Fanshawe benefit from the research-driven ethos of SHARCNET. Having experienced the research process both before and after the implementation of SHARCNET, Strybosch spoke to the precedent-setting advantages of having SHARCNET in the classroom.
“Our students are just beginning to realize that even without the more advanced math, there are opportunities to code scientific algorithms, and some have embraced the opportunity to explore techniques that they never would have thought of before SHARCNET.”
Essentially, Strybosch considers SHARCNET extremely important in its ability to “spark a vision.” From Strybosch’s personal experiences, it was his work with legitimate parallel machines – such as SHARCNET – that boosted his confidence regarding his own research practices. “My exposure was to a Sequent multiprocessor and the Myrias supercomputers. Today, systems like SHARCNET can offer students the same kind of inspiration.”
As well, Strybosch believes that the opportunities
presented by SHARCNET for students will have a cyclical influence
by continuing to drive
the imagination of current and future students at Fanshawe. Just as
SHARCNET has “sparked imagination in these students,” says
Strybosch, it “will make other students stretch their imaginations.”
From cells to crystals to winter coats, you might say that York University’s Dr. Huaxiong Huang has his hands full these days. An Associate Professor in the Department of Mathematics and Statistics, as well as York’s site-leader for SHARCNET, Dr. Huang is collaborating with researchers across a wide range of disciplines and industries to produce innovative mathematical models. In the process, the York researcher is demonstrating how mathematical modeling can be brought to bear on any number of real-world problems.
Take something as mundane as winter coats. Dr. Huang’s collaboration with researchers at City University of Hong Kong and Hong Kong Polytechnic University is yielding mathematical models that will help the textile industry test new fabrics that are less resistant to moisture removal.
Or take human cells—specifically, the cellular mechanisms that allow the mammalian brain to function. Until now, knowledge of how the brain functions through combinations of these mechanisms has remained largely elusive. But working with a number of researchers and using SHARCNET, Dr. Huang is looking to construct mathematical models to help researchers better understand the workings of the brain.
“SHARCNET will be essential to carrying out such complex research,” notes Dr. Huang. “You need a High Performance Computing network to process the large amounts of information that we deal with.”
But that’s not the only project of Dr. Huang’s that will require SHARCNET’s resources. Dr. Huang is also collaborating with Firebird Semiconductor Ltd.—a global leader in the production of semiconductor wafers for radiation detectors—to model crystal growth. This project, also supported by researchers at York, UBC and Penn State, will enable industry to detect how defects form in crystals during the growth process.
Dr. Huang’s work demonstrates not only the ability
of SHARCNET to facilitate interdisciplinary projects, but also the
real-world approach of York research. His collaborators at York include
Dr. Jianhong Wu, Dr. Dong Liang and Dr. Moshe Milevsky. Dr. Huang received
his PhD from UBC in Applied Mathematics in 1992, and was a postdoctoral
fellow at John Hopkins University and Simon Fraser University. He is
also one of the recipients of a PIMS 2000 Industrial Outreach Award,
and his work is funded by NSERC, Firebird, MITACS, and the Fields Institute
for Research in Mathematical Sciences.
(SC) is one of the leading conferences on high performance computing
held each year. SC aims to bring together individuals from industry,
academia and other areas of technical expertise to explore the latest
developments in the field, and incorporates an extensive exhibitor floor
where vendors and research organizations highlight their latest and greatest.
Perhaps the most well-known event at this annual conference is the unveiling of the November rankings of the Top500 Supercomputer sites (http://www.top500.org). It was IBM’s new BlueGene/L system with over 32,000 processors that came in at #1 this time around, performing at over 70TFlops, and it’s apparently only a quarter completed. NEC’s Earth Simulator, which had been atop the Top500 list for two years, has fallen to #3. There is little question that the face of HPC is evolving quite dramatically.
Other notable conference highlights included Hewlett Packard showcasing software developed on a pilot Sepia distributed visualization system at SHARCNET; a chance to get up close and personal with the BlueGene system from IBM, which promises a fundamental shift in how we approach scalable large-scale parallel computing; our first look at next generation technology from SGI and Cray, who are now both touting programmable FPGA units on their systems for customizable, application specific hardware support for increased performance; and a look at Sun’s latest developments in interval-based numeric computing, which have the potential to increase the reliability with which we can draw accurate conclusions from simulation.
partnership offers new horizons in hardware, computing capability
At the end of June, SHARCNET joined with a group of McMaster researchers: James Wadsley, Erik Sorensen, Walter Craig, Dmitry Pelinovsky and Catherine Kallin, in acquiring a new system, named Wobbe. The cluster was jointly funded by the SHARCNET Sustaining Fund and individual researcher contributions, including CFI new opportunities grants. The unusual name is short for Wobbegong, a species of shark (Orectolobus ornatus), indigenous to Australia and the coastal reefs of the Pacific Ocean.
Wobbe consists of 96 dual 1.8 GHz Opteron nodes, connected with Myrinet, running Linux, and it represents SHARCNET’s first foray into the world of 64 bit x86 computing. This support for 64 bit computing is important for several researchers because they need to access very large amounts of memory (2 to 4 GB per CPU), a level considerably higher than has been available heretofore on SHARCNET systems like Idra, Hammerhead or Greatwhite. The use of Myrinet is another new feature for SHARCNET, which until now has used the more expensive technology from Quadrics for the network interconnect. This should give researchers an opportunity to see how their codes scale using the different network technology, and if in fact this cheaper solution provides acceptable performance.
User jobs first began running on Wobbe in July, when it was initially configured as a 32 bit machine, using the Intel compilers and development tools. Beginning in late August, Wobbe started the transition to being a full 64 bit system, using a custom Linux kernel, which has necessitated a new set of development tools. The choice has been made to opt for compilers from PathScale, in conjunction with the AMD-supplied ACML library for BLAS/LAPACK support, and the queueing system is the standard set of sq* tools already in use on SHARCNET’s McMaster hardware.
The ideal jobs for Wobbe are large (requiring 32, 64 or more CPUs)
parallel jobs using MPI, that have relatively high internode communication
needs, and which use a fairly substantial amount of memory. The Opteron
architecture features a bus in which each CPU has its own distinct
connection to memory, and this should allow memory bandwidth-limited
jobs to perform very well.
HPCS2005: the 19th annual symposium
on high performance computing systems and applications, a multi-disciplinary
meeting to discuss new and exciting scientific and technical work involving
High Performance Computing, will be hosted by SHARCNET at the University
of Guelph, May 15-18, 2005. Researchers from all disciplines in the sciences,
engineering, mathematics and applied human sciences are invited to participate.
David H. Bailey
Jonathan M. Borwein, FRSC, CRC
Robert D. Moser, Professor
Visit http://www.sharcnet.ca/events/hpcs2005 for more details.
Zhenming (Jemmy) Hu holds a Ph.D. in Theoretical and Computational Chemistry from Kyoto University in Japan and a Masters degree in Computer Science from Dalhousie University. He held a faculty position at Kyoto University and has held post-doctoral and research associate positions at Dalhousie University, York University and the University of Toronto.
Jemmy’s background in ab initio computational
chemistry includes surface modeling, molecule-surface interaction,
and catalytic reaction
mechanisms. He has also worked on developing software systems for molecular
databases, data mining and bioinformatics. Jemmy joined SHARCNET as
an HPTC consultant in December of 2004.
© 2003 The
Shared Hierarchical Academic Research Computing Network.