Skip to content

Issues with pbs.py for PBS Pro 2022.1.7 and a patch we are currently using as a workaround #3579

Closed

Description

@colleeneb

TLDR Summary: We have been trying to use the develop branch of reframe-hpc on our system (which is using PBS Pro 2022.1.7), but ran into some issues with pbs.py, possibly due to differences in the PBS version pbs.py was originally developed for and our version of PBS. We wrote a patch for pbs.py to make it work with PBS Pro 2022.1.7 on our system: https://github.com/reframe-hpc/reframe/compare/develop...colleeneb:reframe:develop?expand=1 . Is this useful to contribute back to reframe as a PR and if so, what should I tag it as? A new feature "PBS Pro Scheduler" or something else?

Details:
The main changes we needed to make to get it to work were in poll in pbs.py. A comparison is here: https://github.com/reframe-hpc/reframe/compare/develop...colleeneb:reframe:develop?expand=1 shows the differences we made to it to get it to work reliably on our system. We would like to contribute back what we did if they are useful for you – either in pbs.py or a new "PBS Pro" scheduler type, whatever would be preferred by reframe. Let us know and we can submit this as a PR however you want--I'm not sure if it is a new feature of pbs pro or bugfix.

To add more details, a few of the issues we had were:

  • The call to "qstat -f {' '.join(job.jobid for job in jobs)}" (
    f'qstat -f {" ".join(job.jobid for job in jobs)}'
    ) will return 35 if any of the jobs in the joblist are not in the queue (i.e. if they have finished running). Thus if one job is out of the queue but others are still running, this call will still return 35, and all jobs are marked as completed when they are not actually done.
  • output_ready (
    def output_ready(job):
    ) is used as a check on if the job is done as well, but "output_ready" is true as soon as the output and error files exist, and for PBS Pro at least, these files exist when the job begins to run, not when it ends, so it's true whenever the job starts running, and can't be used to know when the job is done. [0]
  • In PBS Pro, the state that jobs go into when they are complete is "F" for Finished, not "C". (Table 2-25: Job States in https://help.altair.com/2022.1.0/PBS%20Professional/PBS2022.1.pdf#M54.9.38036.PBSHeading1.151.qstat). Thus
    if job.state == 'COMPLETED':
    is never true for us since there's no "C" state.

With that in mind, we modified poll to a poll function for PBS Pro where we keep the same structure, but query "qstat -xf F json" to get the output of all the jobids. Since we use "-x" this list includes the jobids that have already finished, and we asked for it in json format so it's a bit easier to parse. Then we loop over the jobs and we can rely on just checking the state to determine if it's finished.
Thanks!
Credit also to @TApplencourt , @thilinarmtb, @bensallen !

[0] @bensallen has clarified that the files are written as soon as the job runs for our system is because of how it's set up (we're using $usecp in the pbs_mom configuration, so PBS knows we have shared filesystems and thus will write directly to them instead of copying the files back at the end of the job). However, the most clear way of knowing a job is finished is with checking the status via qstat as it works regardless of the state of the files.

Activity

  1. vkarak commented on Nov 7, 2025

    @vkarak
    Contributor

    @colleeneb Thanks for your issue and for sharing your changes. The original pbs backend was developed to bit generic and this is why the torque backend derives directly from it. I think your changes would most likely make sense as a different, more focused backend called pbspro.

    I will let two other contributors to the current plugin chime in with their opinion: @giordano @smoors iirc you are using PBS. Which flavor are you using and what do you think of these changes?

  2. colleeneb commented on Nov 7, 2025

    @colleeneb
    ContributorAuthor

    Thanks! I can indeed switch it to a new pbspro backend instead, that seems clean and is not a problem! (But will wait for the other contributors opinions.)

  3. colleeneb commented on Nov 14, 2025

    @colleeneb
    ContributorAuthor

    I updated the code so that now it does not edit the poll function directly but it registers a new scheduler pbspro which can be used without affecting the pbs scheduler. The diff is now: https://github.com/reframe-hpc/reframe/compare/develop...colleeneb:reframe:develop?expand=1 . Let me know if this is ok for PR!

  4. smoors commented on Nov 15, 2025

    @smoors
    Contributor

    we don't use Torque anymore, but i agree that pbspro is sufficiently different to warrant a new backend.

  5. colleeneb commented on Nov 16, 2025

    @colleeneb
    ContributorAuthor

    I opened #3584 since that seems to be the agreed upon path. I tried to follow the contributing guidelines, but let me know if I should change anything.

  6. colleeneb commented on Dec 1, 2025

    @colleeneb
    ContributorAuthor

    Closing since #3584 is merged in, thanks @vkarak !

  7. added this to the ReFrame 4.10 milestone on Dec 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions