Skip to content

Implement most of CompletionQueueActor (#4714) - #4714

Closed
samlurye wants to merge 2 commits into
meta-pytorch:mainfrom
samlurye:export-D116806594
Closed

Implement most of CompletionQueueActor (#4714)#4714
samlurye wants to merge 2 commits into
meta-pytorch:mainfrom
samlurye:export-D116806594

Conversation

@samlurye

@samlurye samlurye commented Aug 20, 2026 •
edited by meta-codesync Bot
Loading

Copy link
Copy Markdown
Contributor

Summary:

Add CompletionQueueActor: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an Attach message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending Posted notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Reviewed By: zdevito

Differential Revision: D116806594

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 20, 2026
@meta-codesync

meta-codesync Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

@samlurye has exported this pull request. If you are a Meta employee, you can view the originating Diff in D116806594.

@meta-codesync meta-codesync Bot changed the title Implement most of CompletionQueueActor Implement most of CompletionQueueActor (#4714) Aug 20, 2026
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 20, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
bf9f007 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Differential Revision: D116806594
@samlurye
samlurye force-pushed the export-D116806594 branch 2 times, most recently from 65abb87 to ad38e89 Compare August 24, 2026 19:11
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 24, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
13ad43e 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Differential Revision: D116806594
@samlurye
samlurye force-pushed the export-D116806594 branch 2 times, most recently from 13ad43e to a54bca2 Compare August 24, 2026 22:45
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 25, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
674f73e 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 25, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
5ea8f94 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 25, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
f6bc15c 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 25, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
a05edca 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 26, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
142342e 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Reviewed By: zdevito

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 26, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
c4b4ce4 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Reviewed By: zdevito

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 26, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
cbbaf22 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Reviewed By: zdevito

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 26, 2026
@samlurye @meta-codesync
Implement most of CompletionQueueActor (meta-pytorch#4714)
fcf024a 
Summary:
Pull Request resolved: meta-pytorch#4714

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Differential Revision: D116806594
@samlurye @facebook-github-bot
Introduce the IbvCompletionQueue trait (meta-pytorch#4695)
afe0838 
Summary:

Introduce the `IbvCompletionQueue` trait, which will wrap real hardware-backed completion queue. The upcoming `CompletionQueueActor` will be generic over `IbvCompletionQueue`, allowing the batched polling logic to be tested without real hardware.

Reviewed By: zdevito

Differential Revision: D116689368
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
3659093 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Reviewed By: zdevito

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 26, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
a217d1d 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Reviewed By: zdevito

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 26, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
d1f141b 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Reviewed By: zdevito

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 26, 2026
@samlurye @facebook-github-bot
Implement most of CompletionQueueActor (meta-pytorch#4714)
b07cb19 
Summary:

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Reviewed By: zdevito

Differential Revision: D116806594
samlurye added a commit to samlurye/monarch-1 that referenced this pull request Aug 26, 2026
@samlurye @meta-codesync
Implement most of CompletionQueueActor (meta-pytorch#4714)
1282066 
Summary:
Pull Request resolved: meta-pytorch#4714

Add `CompletionQueueActor`: one poller for many CQs, holding a route from each queue pair's completions back to that queue pair.

Each CQ actor starts empty. When a new QP starts up, the manager actor will (eventually, not yet implemented) assign a CQ and, independently, a CQ actor to that QP according to some user configured policy. This design allows us to try different concurrency models (e.g., one CQ actor polls every CQ in the process vs. one CQ actor for every NIC polls every CQ for that specific NIC).

When the QP is ready to start posting, it sends an `Attach` message to the assigned CQ actor with: the CQ to poll; an atomic counter indicating how many WRs that QP has posted so far; and a port for sending batches of completions back to the QP.

The CQ actor does not poll unless there is work to be done. QPs wake it by sending `Posted` notifications when they have posted new WRs. Between yields, the CQ actor loop polls every CQ that it tracks, requesting 64 CQEs at a time from each CQ. When it consumes CQEs, it routes them back to the relevant QP so that it can update its credit tracking and send replies to the original caller.

QP detaching logic and other failure handling is left for future commits. Nothing uses this code yet.

Differential Revision: D116806594
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Meta Open Source bot. meta-exported

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@samlurye