From e4eaddec4ebb13559ee592fa01400817688782ed Mon Sep 17 00:00:00 2001 From: thloyi Date: Mon, 18 May 2026 14:22:48 +0800 Subject: [PATCH] reademe --- README.md | 249 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 249 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..f985c17 --- /dev/null +++ b/README.md @@ -0,0 +1,249 @@ +# pump-parser + +Solana transaction parser focused on swap, liquidity, migration, platform, MEV, compute budget, and compact binary persistence workflows. + +The package works with a normalized `RawTx` representation, parses it into `Tx`, and emits one or more `Swap` records when a supported protocol action is found. + +## Features + +- Parse Solana RPC / Yellowstone transactions into local `RawTx`. +- Extract swap and liquidity events into `Tx.Swaps`. +- Preserve transaction metadata such as slot, block time, signer, fee, CU limit, CU consumed, token balances, platform fees, and MEV agent hints. +- Encode/decode parsed transactions with the `PTXB` / `PTXS` binary formats. +- Encode/decode raw transactions with the `PRTX` / `PRTS` / `PRBS` binary formats. +- Stream decode large `PTXS` / `PRTS` payloads without loading every transaction into memory. +- Merge `PTXS` batches while remapping address tables. + +## Supported Parsers + +Default parser initialization enables the hot-path Pump parsers only: + +- Pump +- Pump AMM + +`EnableAllParsers()` additionally enables: + +- Meteora DLMM +- Meteora Pools +- Meteora DAMM v2 +- Meteora Bonding Curve +- Orca Whirlpool +- Raydium AMM v4 +- Raydium CLMM +- Raydium CPMM +- Raydium LaunchLab + +Use `InitParser(WithMeteoraDlmm())` when only Meteora DLMM should be added to the default parser set. + +## Installation + +```bash +go get github.com/thloyi/pump-parser +``` + +This module currently declares `go 1.25.1` in `go.mod`. + +## Basic Usage + +```go +package main + +import ( + "fmt" + "log" + + pump_parser "github.com/thloyi/pump-parser" +) + +func main() { + // Enable all known parser programs. Omit this call to keep the default + // Pump + Pump AMM parser set. + pump_parser.EnableAllParsers() + + var rawTx *pump_parser.RawTx + // Fill rawTx from RPC, Yellowstone, JSON, or RawTx binary decoding. + + tx, err := pump_parser.ParseRawTx(rawTx) + if err != nil { + log.Fatal(err) + } + + fmt.Println("tx:", tx.GetTxHash()) + for _, swap := range tx.Swaps { + fmt.Printf("%s %s base=%s quote=%s pool=%s user=%s\n", + swap.Program, + swap.Event, + swap.BaseAmount, + swap.QuoteAmount, + swap.Pool, + swap.User, + ) + } +} +``` + +## Swap Result + +`ParseRawTx` returns a `Tx`. A transaction can contain zero, one, or multiple parsed swap records in `Tx.Swaps`. + +Each `Swap` describes one protocol-level swap, liquidity, migration, or pool operation that the parser can normalize. The most important fields are: + +| Field | Meaning | +| --- | --- | +| `Program` | Normalized protocol name, such as `Pump`, `PumpAMM`, `MeteoraDLMM`, `RaydiumV4`, or `OrcaWhirPool`. | +| `Event` | Normalized action, such as `buy`, `sell`, `add_liquidity`, `remove_liquidity`, `create`, `complete`, or `migrate`. | +| `TxIndex` | Approximate execution order across outer and inner instructions. | +| `InstrIdx` / `InnerIdx` | Outer instruction index and inner instruction index where the swap was found. | +| `Pool` | Pool, pair, bonding curve, or market account for the parsed action. | +| `BaseMint` / `QuoteMint` | Base and quote mint accounts after parser normalization. | +| `BaseTokenProgram` / `QuoteTokenProgram` | Token program for each side, useful when Token-2022 is involved. | +| `BaseMintDecimals` / `QuoteMintDecimals` | Decimals used to interpret raw token amounts. | +| `User` | User or effective owner account for the action. If the parsed user is not on-curve, the parser may fall back to the transaction signer. | +| `BaseAmount` / `QuoteAmount` | Actual parsed base-side and quote-side amounts, stored as `decimal.Decimal`. | +| `BaseReserve` / `QuoteReserve` | Pool reserves when the protocol event or accounts expose them. | +| `UserBaseBalance` / `UserQuoteBalance` | User token balances after the transaction when available from token balance metadata. | +| `AfterSOLBalance` | User or signer SOL balance after the transaction. | +| `EntryContract` | Known router / entry contract account when detected. | +| `Mayhem` / `Cashback` | Protocol or platform-specific labels detected from the transaction path. | + +Swap amount and slippage fields normalize instruction intent: + +| Field | Meaning | +| --- | --- | +| `SwapMode` | `exact_in`, `exact_out`, or empty when unknown. | +| `FixedAmount` / `FixedAmountSide` / `FixedMint` | The user-specified fixed side of the swap. For `exact_in`, this is the input amount. For `exact_out`, this is the target output amount. | +| `LimitAmountType` | `min_out` for `exact_in`, `max_in` for `exact_out`, or empty when unknown. | +| `LimitAmount` / `LimitAmountSide` / `LimitMint` | User-specified limit on the opposite side. | +| `ActualLimitAmount` / `ActualLimitAmountSide` | Actual executed amount on the limited side. | +| `SlippageBps` | Remaining headroom to the user's limit in basis points. See `SLIPPAGE_MAPPING.md` for protocol-specific derivation rules. | + +Liquidity, migration, and DLMM-specific fields are populated only for protocols that expose them: + +| Field | Meaning | +| --- | --- | +| `Creator` | Creator account when exposed by pool or launch events. | +| `MigrateToPool` / `MigrateTopProgram` | Destination pool and program for migration events. | +| `LpMint` | LP token mint for liquidity operations when available. | +| `ActiveBinId` / `StartBinId` / `EndBinId` | Meteora DLMM bin identifiers. | +| `RemoveBp` | DLMM remove-liquidity basis points. | +| `PositionAccount` | DLMM position account. | +| `FeeAmount` / `LpFeeAmount` | Parsed fee amounts when the protocol exposes fee breakdowns. | +| `FeeSide` / `FeeMint` / `FeeTokenProgram` / `FeeMintDecimals` | Fee side and mint metadata. | +| `ConsumeUnit` | Per-swap compute unit value when available. | + +Amounts are normalized into decimals in parser output, but the exact scale depends on the source event and parser path. For persisted binary output, `tx_binary.go` defines the conversion rules and schema version. + +## Convert Transactions + +RPC transactions can be converted with `FromRpcTransactionWithMeta`: + +```go +rawTx, err := pump_parser.FromRpcTransactionWithMeta( + txWithMeta, + blockTime, + slot, + indexWithinBlock, +) +``` + +Yellowstone transactions can be converted with `ConvertYellowstoneGrpcTransactionToSolanaTransaction`: + +```go +rawTx, err := pump_parser.ConvertYellowstoneGrpcTransactionToSolanaTransaction( + update, + createdUnix, +) +``` + +Both conversion paths accept optional `RawTxConvertOptions`: + +```go +rawTx, err := pump_parser.FromRpcTransactionWithMeta( + txWithMeta, + blockTime, + slot, + indexWithinBlock, + pump_parser.RawTxConvertOptions{ + ParseLogEvents: true, + }, +) +``` + +`ParseLogEvents` attaches decoded program log events to matching instructions. `IgnoreLogMessages` skips log-message retention. + +## Binary Formats + +Parsed transaction binary: + +- `EncodeTxBinary` / `DecodeTxBinary` for one parsed `Tx`. +- `EncodeTxsBinary` / `DecodeTxsBinary` for a batch of parsed `Tx`. +- `DecodeTxsBinaryReader` for streaming `PTXS` reads. +- `MergeTxsBinaryBytes` and `MergeTxsBinarySourcesToWriter` for merging `PTXS` batches. + +Raw transaction binary: + +- `EncodeRawTxBinary` / `DecodeRawTxBinary` for one `RawTx`. +- `EncodeRawTxsBinary` / `DecodeRawTxsBinary` for a batch of `RawTx`. +- `EncodeRawTxBlocksBinary` / `DecodeRawTxBlocksBinary` for grouped block data. +- `DecodeRawTxsBinaryReader` for streaming `PRTS` reads. + +Format magic values: + +- `PTXB`: one parsed `Tx`. +- `PTXS`: parsed `Tx` batch. +- `PRTX`: one raw `RawTx`. +- `PRTS`: raw `RawTx` batch. +- `PRBS`: raw block-grouped `RawTx` batch. + +When adding or renaming transaction-facing enum values, update `tx_binary.go` enum tables by appending new values only. Reordering existing values changes persisted numeric IDs. + +## Commands + +Parse a live transaction by signature: + +```bash +TX_HASH= go run ./cmd/rpc_parse +``` + +Collect Yellowstone transactions into a `.prbs` raw-block binary file: + +```bash +YELLOWSTONE_X_TOKEN= go run ./cmd/collect_yellowstone_rawtx_binary \ + -endpoint ams.rpc.orbitflare.com:10000 \ + -duration 5m \ + -output testdata/rawtx-binary/sample.prbs +``` + +Measure parsed `Tx` binary size from a `getBlock` JSON payload: + +```bash +go run ./cmd/measure_tx_binary_block \ + -file /path/to/block.json \ + -slot 413539056 \ + -swaps-only +``` + +Analyze raw transaction binary size distribution: + +```bash +go run ./cmd/analyze_rawtx_binary_size \ + -file testdata/rawtx-binary/sample.prbs +``` + +## Development + +Run the full test suite: + +```bash +go test ./... +``` + +Useful focused checks: + +```bash +go test -run 'TestTxBinary|TestRawTxBinary' . +go test -run 'TestMetaoraPoolSwapEventFromLogsUsesMatchingInvocation|TestMetaoraPoolSwapInstructionOccurrenceIncludesInnerInstructions|TestAttachLogEventsToInstructions' . +TX_HASH= go run ./cmd/rpc_parse +``` + +For parser regressions, reproduce with the exact transaction signature first, then add a targeted unit test or fixture around the corrected parser path.