Skip to content

feat: Add ETL Indexing for mpt_issuance_history#3104

Open
BryanJ1ang wants to merge 12 commits into
XRPLF:developfrom
BryanJ1ang:brjiang-mpt-rpc-pr2
Open

feat: Add ETL Indexing for mpt_issuance_history#3104
BryanJ1ang wants to merge 12 commits into
XRPLF:developfrom
BryanJ1ang:brjiang-mpt-rpc-pr2

Conversation

@BryanJ1ang

@BryanJ1ang BryanJ1ang commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds live ETL indexing for mptoken_issuance_history by extracting MPT issuance references from transaction metadata and transaction fields, then writing both MPT transaction index shapes during the existing MPT ETL pass. Successful transactions are indexed from affected MPT ledger objects; failed transactions are indexed only when they carry an attached MPT issuance reference.

Changes

  • Adds getMPTokenIssuanceTxsFromTx to extract issuance transaction index records from tesSUCCESS transactions.
  • Scans AffectedNodes generically for MPToken and MPTokenIssuance ledger objects instead of relying on a transaction-type allowlist.
  • Reconstructs issuance IDs for MPTokenIssuance objects with makeMptID(sequence, issuer), avoiding the hashed ledger-key trap.
  • Deduplicates distinct issuances per transaction and attaches the transaction’s affected accounts for account-level fanout.
  • Extends MPTExt so live and initial ETL data writes:
    • existing MPT holder index data
    • mptoken_issuance_transactions
    • account_mptoken_issuance_transactions
  • Adds a Prometheus counter for MPT issuance transaction index rows written by ETL.
  • Logs unexpectedly large per-transaction fanout.

Tests

Adds unit coverage for:

  • failed transactions producing no records
  • MPTokenIssuanceCreate, Destroy, Set, and MPTokenAuthorize
  • generic transaction-type coverage through Payment, Clawback, OfferCreate, and AMMDeposit
  • ID reconstruction from issuance nodes
  • multi-issuance fanout and deduplication
  • affected-account propagation
  • ETL extension writes to both backend index paths while preserving existing holder indexing

@BryanJ1ang BryanJ1ang force-pushed the brjiang-mpt-rpc-pr2 branch 2 times, most recently from 6e3d1b9 to 8031c6b Compare June 17, 2026 14:41
@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.78082% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.66%. Comparing base (0b20dc7) to head (fc73c0e).
⚠️ Report is 13 commits behind head on develop.

Files with missing lines Patch % Lines
src/etl/impl/ext/MPT.cpp 80.95% 3 Missing and 1 partial ⚠️
src/etl/MPTHelpers.cpp 96.15% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3104      +/-   ##
===========================================
+ Coverage    82.60%   82.66%   +0.06%     
===========================================
  Files          397      397              
  Lines        16024    16104      +80     
  Branches      8395     8439      +44     
===========================================
+ Hits         13236    13312      +76     
- Misses        1667     1670       +3     
- Partials      1121     1122       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@BryanJ1ang BryanJ1ang changed the title Feat: Add ETL Indexing for mpt_issuance_history feat: Add ETL Indexing for mpt_issuance_history Jun 18, 2026
@PeterChen13579 PeterChen13579 self-requested a review June 25, 2026 19:43
@BryanJ1ang BryanJ1ang force-pushed the brjiang-mpt-rpc-pr2 branch from 9af68a0 to a5c11bf Compare June 25, 2026 21:09

@PeterChen13579 PeterChen13579 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also Codecov that shows you some lines from source file that are missing tests. Please cover them.

Comment thread src/etl/MPTHelpers.cpp Outdated
Comment thread src/etl/MPTHelpers.cpp Outdated
Comment thread src/etl/MPTHelpers.cpp Outdated
Comment thread src/etl/impl/ext/MPT.cpp
auto const mptHolders = getMPTHolderFromTx(tx.meta, tx.sttx);
holders.append_range(mptHolders);

auto txIndexData = getMPTokenIssuanceTxsFromTx(tx.meta, tx.sttx);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understood correctly, line 57 will scan through every txn on every incoming ledger number, if the txn's related ledger_entry is not mpt, it will return {}. But line 60, you scan the same data again for mpt fields, which won't exist if it already returned {} prior. Therefore, I would have a way to exit early instead of doing another scan

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite. Line 57 only extracts newly created ltMPTOKEN holder records from successful metadata. I believe returning empty does not mean the transaction is unrelated to MPT.

Line 60 builds the MPT issuance transaction-history lookup, so it also needs to catch ltMPTOKEN_ISSUANCE metadata and MPT references in the submitted transaction body, such as sfMPTokenIssuanceID, MPT Amount, or MPT Issue. Early exit after line 57 would miss these valid MPT-related transactions.

Comment thread src/etl/MPTHelpers.cpp Outdated
Comment thread src/etl/MPTHelpers.cpp
return {};

auto const& fieldsName =
node.getFName() == xrpl::sfCreatedNode ? xrpl::sfNewFields : xrpl::sfFinalFields;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we confirm the affected-node field selection here is always valid?
CreatedNode uses sfNewFields, while ModifiedNode/DeletedNode use sfFinalFields. (I'm pretty sure this is right checking rippled code, but what is the mechanism that allows it to change? 🤔 ). (cc @yinyiqian1 @shawnxie999 )

Comment thread tests/unit/etl/ext/MPTTests.cpp Outdated
Comment thread src/etl/MPTHelpers.cpp Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants