Repository navigation
Small improvements and fixes - #9
Conversation
| You can also poll `nvidia-smi` on a tight interval: | ||
|
|
||
| ```bash | ||
| watch -n 0.5 nvidia-smi |
There was a problem hiding this comment.
Having this below a second caused the timestamp in the nvidia-smi output to update in a weird way which I found distracting. Changing it to 1 fixed it.
| Reach for `watch nvidia-smi` when you want to see memory change over time. Reach for `nvtop` when you want to see whether a running process is keeping the GPU busy. | ||
| Reach for `watch nvidia-smi` when you want to see snapshots of the current GPU state. Use `nvtop` when you want to see a timeline to check whether a running process is keeping the GPU busy. | ||
|
|
||
| For timeline profiling, install [Nsight Systems](https://developer.nvidia.com/nsight-systems). Reach for it when the GPU looks busy but the workload is still slow: it lays out CPU/GPU memory transfers, CUDA API calls, kernel launches, and synchronization points on a single timeline. We install it now and use it in detail later in this section. On a fresh VM you first need the NVIDIA CUDA apt repository, since the Nsight Systems package lives there: |
There was a problem hiding this comment.
The phrase "reach for" is a little overused in this guide, I took a few out.
| ```bash | ||
| python -m cProfile -o profile-xarray-cpu.prof xarray_cpu_baseline.py | ||
| python -m snakeviz profile-xarray-cpu.prof | ||
| python -m snakeviz -s profile-xarray-cpu.prof |
There was a problem hiding this comment.
When I ran the command as written it tried to open a browser on the Brev VM and errored. I needed the -s flag to disable this.
There was a problem hiding this comment.
Did this error even with the port forwarding set up?
@jayavenkatesh19 IIRC you didn't have this issue, do you recall what was the order in which you run this and opened it in browser?
There was a problem hiding this comment.
Yep Jacob is right, we need this flag to open this in server only mode which does not open a browser on remote by default.
Co-authored-by: Jaya Venkatesh <jjayabaskar@nvidia.com>
3 participants
I ran through the material in this repo today. Overall it looks great. It'll be good to see it collected all together with the other material, it seems to reference some Python setup stuff that's stored elsewhere.
I suggested a few tweaks in here and fixed a couple of issues. Take most of these as nitpick suggestions.