Major Expansion of SSCC’s Computing Resources
We are thrilled to announce that the SSCC will soon be able to more than triple the amount of computing power available to SSCC researchers, thanks to a Research Core Revitalization Program grant from the Office of the Vice Chancellor for Research and Graduate Education and additional funding from the Department of Economics and the Wisconsin School of Business. We’ll also be adding powerful GPU servers. The new servers will be made available in the SSCC’s Slurm cluster, where they can be used most efficiently, and may be available as soon as this year’s holiday season (barring supply chain issues).
We encourage all SSCC researchers to ask themselves “What new things can I do because of this new computing power?” We are committed to making large-scale computing as accessible and easy to use as possible. That includes installing easy to use tools like RStudio Server and JupyterLab, teaching multiple workshops on how to use the SSCC’s computing resources, and creating a new Guide to Research Computing at the SSCC. The SSCC’s Statistical Consultants will be happy to answer questions and help you navigate using the new servers. Please don’t let the idea of using Linux servers intimidate you: let us show you how easy it is.
As an example, we’ve recently written three new articles on how to take an ordinary loop written in R, Stata, or Python and modify it so that instead of carrying out repeated tasks sequentially, it submits them to the Slurm cluster so they can all run at the same time. The process is straightforward, leveraging what you already know about R, Stata, or Python.
Please Welcome Ryan St. Peters
Please welcome Ryan St. Peters as our newest user support specialist at the SSCC.
Ryan is joining us as part of our Help Desk staff and will also take over SSCC’s instructional support duties. Ryan joins us with a bachelor’s degree in computer science. He moved from West Texas to Madison to be with his fiancé. Outside of work, Ryan enjoys traveling around the country and hopes to soon be able to travel outside of the country in the future.
The SSCC will be undergoing our annual summer tech update from 10am-8pm on Sunday August 14th. All SSCC services will be unavailable during that time. Any ongoing projects or work should be saved to prevent any data loss. Please contact the SSCC Help Desk with any questions or concerns.
Some of the new features included in the summer tech update are:
- We are setting aside designated Winstat for Big Jobs servers so big jobs don’t slow down regular Winstat.
- JupyterLab is now available on Linstat and LinSilo, and configured such that Stata, R, Julia, and SAS users can use Jupyter Notebooks as well as Python users.
- You can now run RStudio Server on Linstat by going to https://ssc.wisc.edu/rstudio-server and SAS Studio on Linstat by going to https://ssc.wisc.edu/sas-studio. Both will run on the least busy Linstat server.
- After logging into silo.ssc.wisc.edu you’ll have the option to start a session directly on LinSilo rather than first starting a session on WinSilo.
- The SSCC’s Slurm cluster is now considered “in production” and Linstat1 through Linstat4 will be moved into Slurm.
- We are replacing CondorSilo with a separate Slurm cluster, SlurmSilo. It will contain the servers currently in CondorSilo plus one LinSilo server and one LinSiloBig server.
- Linstat and LinSilo will now focus on interactive work and development. If you’re still running long jobs on Linstat it’s time to learn how to send them to Slurm.
- You may now use up to 500GB of memory on Linstat, depending on how much others are using. (Keep in mind that in Slurm you can reserve the memory you need and be guaranteed to get it.)
- ArcGIS will upgraded to ArcGIS Pro, and all other software will be updated to their current versions.
Changing Roles for Linstat and Slurm
The SSCC’s Slurm cluster is now considered “in production.” Slurm makes it easy to submit and manage jobs, take advantage of multiple servers, and ensure jobs get the resources they need. This makes the Slurm cluster the place to run big research computing jobs at the SSCC. Accordingly, Linstat1 through Linstat4 will be moved into Slurm, where you can reserve up to 750GB of memory for a job rather than sharing it with others, and new servers will primarily go to the Slurm cluster.
We are also creating a Slurm cluster in Silo, SlurmSilo, which is replacing CondorSilo. It will contain the servers currently in CondorSilo plus one LinSilo server and one LinSiloBig server.
If you’re still running big jobs on Linstat, it’s time to start sending them to Slurm instead. That’s easy to do. For example:
ssubmit --cores=32 --mem=50g --email=bbadger@wisc.edu "stata -b do my_do_file"
will tell Slurm to run a Stata do file, reserving 32 cores and 50GB of memory, and send an email when it’s finished.
Linstat and LinSilo will now focus on interactive work. This includes running easy to use tools like RStudio Server, JupyterLab, Stata and Matlab’s graphical user interfaces, and SAS Studio. But it’s also where you can run future Slurm jobs briefly and see how many cores and how much memory they use by monitoring them with top. Then you’ll know what resources to reserve when you submit them to Slurm.
These changes will make it easier than ever to run large research computing jobs at the SSCC, and use the available servers more efficiently.
Introducing Winstat for Big Jobs
In this weekend’s update we will set aside designated “Winstat for Big Jobs” servers. Regular Winstat will consist of small servers intended for interactive work and short statistical jobs, while Winstat for Big Jobs is intended for jobs that will run for a long time or need more cores or memory (though they won’t have nearly as much of either as the Linux servers). On Winstat for Big Jobs, you can start a job and disconnect from your session, and it will keep running for as long as necessary. See Using Winstat for more details.
Moving big jobs to designated servers will ensure they don’t slow down the regular Winstat servers for the many people who use Winstat for interactive work.
We are re-introducing full time in-person support for the SSCC help desk. For our new members or anyone who needs a reminder, the SSCC help desk is in room 4226 in the Sewell Social Science Building. Remote support is still available for all SSCC members alongside in-person support. We will describe any changes to in-person support on the Get Help page on our website should any unexpected circumstances require us to transition to primarily remote support.
The SSCC has a busy fall training schedule. Some highlights:
- “Introduction to…” and “Data Wrangling in…” workshops will teach you the basics of R, Stata, or Python and how to work with real-world data in it. Great for new graduate students!
- Regression Review and Regression Diagnostics courses in R or Stata will help you apply what you’ve learned in your statistics classes to research.
- Running Big Research Computing Jobs at the SSCC and Using the SSCC Slurm Cluster will teach you how to use the SSCC’s computing resources, current and future. Bring a big job and you’ll have it running in Slurm by the end. If you’re a Linux veteran go straight to the Slurm class; if this is all new to you sign up for both. Note that the 9/23 sections will be taught online.
- Other topics, like Mediation Models and creating publication-quality tables in Stata.
Visit our training page for details and to register.
Update Highlight: JupyterLab and Jupyter Notebooks
Jupyter Notebooks have been popular among Python users for years, but with JupyterLab it’s now possible to write notebooks in other languages. As part of the Summer Tech Update, the SSCC has set up JupyterLab on Linstat such that Stata, R, Julia, and SAS users can use Jupyter Notebooks as well as Python users.
JupyterLab is an easy-to-use programming environment. The user interface runs in a browser on your computer so it’s not affected by network lag, but the code is run on Linstat so you have lots of computing power. The only Linux you need to know is how to specify a directory. We think many SSCC researchers will be interested in JupyterLab purely for writing code, especially Julia and Python users where the alternatives are lacking.
Jupyter Notebooks can contain text (optionally formatted using Markdown), code, and the results of running that code, all in a single convenient file. They’re great for collaboration, informal communication of results, and especially teaching. Tools like Quarto can easily convert Notebooks into web pages, PDF files, LaTeX, and more.
We’ll be teaching a workshop specifically on JupyterLab and Jupyter Notebook on September 6th. It will mostly cover how to use JupyterLab, but also discuss some of the ways Notebooks can be used in research and teaching.