Status: draft. Verified steps and screenshots to come.
Link your scratch space into your home directory so the OnDemand apps can see it, then put a dataset there. Time: 10 minutes · Program: the OnDemand Files app and a terminal
| Before this | Data | You'll need |
|---|---|---|
| Nothing | Your own files. Step 3 downloads a public dataset as an example | Your account on the cluster |
Note
Cluster details. Highlighted values are placeholders. AMA27's queue names, module versions, paths and addresses go in when the cluster is deployed.
Why this matters
OnDemand apps open in your home directory, which is small. Work usually belongs in scratch or project space, and an app can only browse there if a path leads to it from your home directory. A symlink solves that. Knowing which space to use also keeps your files from being deleted by the purge.
Key idea
The cluster has three kinds of storage, and each one trades something away. Home is small but backed up; keep code and notes there. Scratch is large and fast, and nothing in it is backed up: files older than a set age are deleted automatically, which is the purge. Project space is shared with your group, and its size is set on request. Work in scratch, and copy anything you want to keep somewhere else before the purge takes it. → Glossary: scratch
Steps
Step 1: Know the three places
| Place | Path | Size | Backed up? | Use it for |
|---|---|---|---|---|
| Home | AMA27 home path | AMA27 home quota | Yes | Scripts, notes, small results you must not lose |
| Scratch | AMA27 scratch path | AMA27 scratch quota | No, and old files are deleted | Data you're working on right now |
| Project | AMA27 project path | Set per group | No | Data your group shares |
Working data goes in scratch.
Step 2: Make scratch visible from home
Open a terminal: the shell on the dashboard's Clusters menu, or the terminal in a desktop session.
Run on: a login node, or the OnDemand desktop terminal. Put AMA27's scratch path in place of /path/to/scratch.
ln -s /path/to/scratch/$USER ~/scratch
You should see: no output. In the Files app, your home directory now has a scratch folder that leads to your scratch space. An app that opens in your home directory can now reach scratch through it.
Screenshot: the Files app, with the scratch folder in your home
Screenshot to come: The Files app showing your home directory, with the new scratch folder marked
Step 3: Download a dataset
Compute nodes may not be able to reach the internet, but login nodes can. Run the download on a login node, into scratch. The example here is ParaView's tutorial data.
Run on: a login node
cd ~/scratch
curl -LO https://www.paraview.org/files/data/ParaViewTutorialData-20220629.tar.gz
tar -xzf ParaViewTutorialData-20220629.tar.gz
ls ParaViewTutorialData-20220629
You should see: a folder of example datasets, disk_out_ref.ex2, can.ex2 and headsq.vti among them. Your own data gets there the same way, by download or upload.
Step 4: Upload a file of your own
In the Files app, open scratch, click Upload and choose a small file from your computer.
You should see: your file listed in scratch. For anything over a few gigabytes, use Globus instead of the browser (see module B3).
Screenshot: the Files app's Upload button
Screenshot to come: The Files app open on your scratch folder, with the Upload button marked
Step 5: Check how much space you have
Run on: a login node. Put AMA27's quota command in place of quota-command.
quota-command
quota-command $USER-scratch
You should see: your usage and limit for home, and then for scratch.
Check your understanding
Your simulation writes 5 GB of results you'll visualize next week and want to keep for a paper. Where should they live?
Write them to scratch, which is fast and large, and visualize them from there. Then copy the results you need to keep to project space, or somewhere backed up, before the purge can delete them.
If something goes wrong
| What you see | Why | What to do |
|---|---|---|
ln: failed to create symbolic link … File exists |
You already made the link | Nothing to do: open ~/scratch |
curl: (6) Could not resolve host |
You're on a compute node without internet | Run the download on a login node instead |
Permission denied in a project folder |
Your group hasn't given you write access | Ask your group's owner (see module B2) |
Next
Credits
ParaView example data: Kitware, BSD-3-Clause.