Skip to content. | Skip to navigation

Personal tools

Navigation

You are here: Home / BioFrontiers Workshops / Workshops in the Series / Short-read 2016 Course Materials / Day 3: Cluster Usage and Data Transfer

Day 3: Cluster Usage and Data Transfer

Day 3: Cluster Usage and Data Transfer Videos

Welcome to Day 3 of the Short-Read Sequencing Analysis Workshop. The videos for Day 3 will serve as an introduction to compute clusters and how to work on them. Jobs are submitted to the cluster, jobscripts to perform this will be introduced, as well as calling modules to work with specific programs. Transferring files to/from the cluster, and downloading files from the internet will also be discussed. For the in-class portion of Day 3 we will walk through some practical examples of submitting jobs on the cluster and transferring/downloading data. Additionally we will review the error and log files and how to use them for troubleshooting failed jobs. After this day you should have the basic skills necessary to perform the different analysis pipelines we will be performing throughout the remainder of the workshop.


Videos for Day 3

Video 1: Introduction to the BioFrontiers Compute Clusters (3:53)  This first video gives a brief introduction to the Vieques and Pando compute clusters available at BioFrontiers, as well as how these resources are managed through a scheduler and job scripts.

Video 2: Running Jobs on the Compute Cluster (8:27)  This video explains how to set up PBS job scripts as well as how to submit, monitor, and terminate jobs on the cluster.

Video 3 : Modules (4:54)  This short video from 2015 explains what modules are and why and how we use them on the cluster to call programs we will be using.

Video 3b: More on Modules More on using modules. 

Video 4: Troubleshooting Failed Jobs (8:37)  Video 4 review how to use the error and log files to troubleshoot failed jobs, how to interpret error messages, and how to revise job scripts to fix errors.

Video 5: Transferring Data (7:23)  The last video shows how to securely copy files between a server and your local machine, as well as how to download data from the internet directly to a server.

 

Files for Day 3

Day 3 Cheat sheet Contains useful information about PBS headers, Torque commands, commands for modules, and common commands for data transfer.

Homework for Day 3

Day 3 Homework Answers

Day 3 In-class slides

 

Additional Resources

Access Request for BFICores website  If you cannot access the links below, follow this link to request access to the internal website (this is separate from the account that was set up for the workshop)

Cluster Resources at BioFrontiers  This page has a lot of information about using the BioFrontiers compute clusters

Vieques and Pando Descriptions  This page has the details about both the Vieques and Pando clusters at BioFrontiers

Vieques Account Request  For attendees that do not have an existing Vieques account you can request access here (the account for the workshop is temporary)

Pando Account Request  For attendees that do not have existing Pando account you can request access here

This is Pacific Theme