reademe
This commit is contained in:
thloyi
249
README.md
Normal file
249
README.md
Normal file
@@ -0,0 +1,249 @@
|
||||
# pump-parser
|
||||
|
||||
Solana transaction parser focused on swap, liquidity, migration, platform, MEV, compute budget, and compact binary persistence workflows.
|
||||
|
||||
The package works with a normalized `RawTx` representation, parses it into `Tx`, and emits one or more `Swap` records when a supported protocol action is found.
|
||||
|
||||
## Features
|
||||
|
||||
- Parse Solana RPC / Yellowstone transactions into local `RawTx`.
|
||||
- Extract swap and liquidity events into `Tx.Swaps`.
|
||||
- Preserve transaction metadata such as slot, block time, signer, fee, CU limit, CU consumed, token balances, platform fees, and MEV agent hints.
|
||||
- Encode/decode parsed transactions with the `PTXB` / `PTXS` binary formats.
|
||||
- Encode/decode raw transactions with the `PRTX` / `PRTS` / `PRBS` binary formats.
|
||||
- Stream decode large `PTXS` / `PRTS` payloads without loading every transaction into memory.
|
||||
- Merge `PTXS` batches while remapping address tables.
|
||||
|
||||
## Supported Parsers
|
||||
|
||||
Default parser initialization enables the hot-path Pump parsers only:
|
||||
|
||||
- Pump
|
||||
- Pump AMM
|
||||
|
||||
`EnableAllParsers()` additionally enables:
|
||||
|
||||
- Meteora DLMM
|
||||
- Meteora Pools
|
||||
- Meteora DAMM v2
|
||||
- Meteora Bonding Curve
|
||||
- Orca Whirlpool
|
||||
- Raydium AMM v4
|
||||
- Raydium CLMM
|
||||
- Raydium CPMM
|
||||
- Raydium LaunchLab
|
||||
|
||||
Use `InitParser(WithMeteoraDlmm())` when only Meteora DLMM should be added to the default parser set.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
go get github.com/thloyi/pump-parser
|
||||
```
|
||||
|
||||
This module currently declares `go 1.25.1` in `go.mod`.
|
||||
|
||||
## Basic Usage
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
pump_parser "github.com/thloyi/pump-parser"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Enable all known parser programs. Omit this call to keep the default
|
||||
// Pump + Pump AMM parser set.
|
||||
pump_parser.EnableAllParsers()
|
||||
|
||||
var rawTx *pump_parser.RawTx
|
||||
// Fill rawTx from RPC, Yellowstone, JSON, or RawTx binary decoding.
|
||||
|
||||
tx, err := pump_parser.ParseRawTx(rawTx)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
fmt.Println("tx:", tx.GetTxHash())
|
||||
for _, swap := range tx.Swaps {
|
||||
fmt.Printf("%s %s base=%s quote=%s pool=%s user=%s\n",
|
||||
swap.Program,
|
||||
swap.Event,
|
||||
swap.BaseAmount,
|
||||
swap.QuoteAmount,
|
||||
swap.Pool,
|
||||
swap.User,
|
||||
)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Swap Result
|
||||
|
||||
`ParseRawTx` returns a `Tx`. A transaction can contain zero, one, or multiple parsed swap records in `Tx.Swaps`.
|
||||
|
||||
Each `Swap` describes one protocol-level swap, liquidity, migration, or pool operation that the parser can normalize. The most important fields are:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `Program` | Normalized protocol name, such as `Pump`, `PumpAMM`, `MeteoraDLMM`, `RaydiumV4`, or `OrcaWhirPool`. |
|
||||
| `Event` | Normalized action, such as `buy`, `sell`, `add_liquidity`, `remove_liquidity`, `create`, `complete`, or `migrate`. |
|
||||
| `TxIndex` | Approximate execution order across outer and inner instructions. |
|
||||
| `InstrIdx` / `InnerIdx` | Outer instruction index and inner instruction index where the swap was found. |
|
||||
| `Pool` | Pool, pair, bonding curve, or market account for the parsed action. |
|
||||
| `BaseMint` / `QuoteMint` | Base and quote mint accounts after parser normalization. |
|
||||
| `BaseTokenProgram` / `QuoteTokenProgram` | Token program for each side, useful when Token-2022 is involved. |
|
||||
| `BaseMintDecimals` / `QuoteMintDecimals` | Decimals used to interpret raw token amounts. |
|
||||
| `User` | User or effective owner account for the action. If the parsed user is not on-curve, the parser may fall back to the transaction signer. |
|
||||
| `BaseAmount` / `QuoteAmount` | Actual parsed base-side and quote-side amounts, stored as `decimal.Decimal`. |
|
||||
| `BaseReserve` / `QuoteReserve` | Pool reserves when the protocol event or accounts expose them. |
|
||||
| `UserBaseBalance` / `UserQuoteBalance` | User token balances after the transaction when available from token balance metadata. |
|
||||
| `AfterSOLBalance` | User or signer SOL balance after the transaction. |
|
||||
| `EntryContract` | Known router / entry contract account when detected. |
|
||||
| `Mayhem` / `Cashback` | Protocol or platform-specific labels detected from the transaction path. |
|
||||
|
||||
Swap amount and slippage fields normalize instruction intent:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `SwapMode` | `exact_in`, `exact_out`, or empty when unknown. |
|
||||
| `FixedAmount` / `FixedAmountSide` / `FixedMint` | The user-specified fixed side of the swap. For `exact_in`, this is the input amount. For `exact_out`, this is the target output amount. |
|
||||
| `LimitAmountType` | `min_out` for `exact_in`, `max_in` for `exact_out`, or empty when unknown. |
|
||||
| `LimitAmount` / `LimitAmountSide` / `LimitMint` | User-specified limit on the opposite side. |
|
||||
| `ActualLimitAmount` / `ActualLimitAmountSide` | Actual executed amount on the limited side. |
|
||||
| `SlippageBps` | Remaining headroom to the user's limit in basis points. See `SLIPPAGE_MAPPING.md` for protocol-specific derivation rules. |
|
||||
|
||||
Liquidity, migration, and DLMM-specific fields are populated only for protocols that expose them:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `Creator` | Creator account when exposed by pool or launch events. |
|
||||
| `MigrateToPool` / `MigrateTopProgram` | Destination pool and program for migration events. |
|
||||
| `LpMint` | LP token mint for liquidity operations when available. |
|
||||
| `ActiveBinId` / `StartBinId` / `EndBinId` | Meteora DLMM bin identifiers. |
|
||||
| `RemoveBp` | DLMM remove-liquidity basis points. |
|
||||
| `PositionAccount` | DLMM position account. |
|
||||
| `FeeAmount` / `LpFeeAmount` | Parsed fee amounts when the protocol exposes fee breakdowns. |
|
||||
| `FeeSide` / `FeeMint` / `FeeTokenProgram` / `FeeMintDecimals` | Fee side and mint metadata. |
|
||||
| `ConsumeUnit` | Per-swap compute unit value when available. |
|
||||
|
||||
Amounts are normalized into decimals in parser output, but the exact scale depends on the source event and parser path. For persisted binary output, `tx_binary.go` defines the conversion rules and schema version.
|
||||
|
||||
## Convert Transactions
|
||||
|
||||
RPC transactions can be converted with `FromRpcTransactionWithMeta`:
|
||||
|
||||
```go
|
||||
rawTx, err := pump_parser.FromRpcTransactionWithMeta(
|
||||
txWithMeta,
|
||||
blockTime,
|
||||
slot,
|
||||
indexWithinBlock,
|
||||
)
|
||||
```
|
||||
|
||||
Yellowstone transactions can be converted with `ConvertYellowstoneGrpcTransactionToSolanaTransaction`:
|
||||
|
||||
```go
|
||||
rawTx, err := pump_parser.ConvertYellowstoneGrpcTransactionToSolanaTransaction(
|
||||
update,
|
||||
createdUnix,
|
||||
)
|
||||
```
|
||||
|
||||
Both conversion paths accept optional `RawTxConvertOptions`:
|
||||
|
||||
```go
|
||||
rawTx, err := pump_parser.FromRpcTransactionWithMeta(
|
||||
txWithMeta,
|
||||
blockTime,
|
||||
slot,
|
||||
indexWithinBlock,
|
||||
pump_parser.RawTxConvertOptions{
|
||||
ParseLogEvents: true,
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
`ParseLogEvents` attaches decoded program log events to matching instructions. `IgnoreLogMessages` skips log-message retention.
|
||||
|
||||
## Binary Formats
|
||||
|
||||
Parsed transaction binary:
|
||||
|
||||
- `EncodeTxBinary` / `DecodeTxBinary` for one parsed `Tx`.
|
||||
- `EncodeTxsBinary` / `DecodeTxsBinary` for a batch of parsed `Tx`.
|
||||
- `DecodeTxsBinaryReader` for streaming `PTXS` reads.
|
||||
- `MergeTxsBinaryBytes` and `MergeTxsBinarySourcesToWriter` for merging `PTXS` batches.
|
||||
|
||||
Raw transaction binary:
|
||||
|
||||
- `EncodeRawTxBinary` / `DecodeRawTxBinary` for one `RawTx`.
|
||||
- `EncodeRawTxsBinary` / `DecodeRawTxsBinary` for a batch of `RawTx`.
|
||||
- `EncodeRawTxBlocksBinary` / `DecodeRawTxBlocksBinary` for grouped block data.
|
||||
- `DecodeRawTxsBinaryReader` for streaming `PRTS` reads.
|
||||
|
||||
Format magic values:
|
||||
|
||||
- `PTXB`: one parsed `Tx`.
|
||||
- `PTXS`: parsed `Tx` batch.
|
||||
- `PRTX`: one raw `RawTx`.
|
||||
- `PRTS`: raw `RawTx` batch.
|
||||
- `PRBS`: raw block-grouped `RawTx` batch.
|
||||
|
||||
When adding or renaming transaction-facing enum values, update `tx_binary.go` enum tables by appending new values only. Reordering existing values changes persisted numeric IDs.
|
||||
|
||||
## Commands
|
||||
|
||||
Parse a live transaction by signature:
|
||||
|
||||
```bash
|
||||
TX_HASH=<signature> go run ./cmd/rpc_parse
|
||||
```
|
||||
|
||||
Collect Yellowstone transactions into a `.prbs` raw-block binary file:
|
||||
|
||||
```bash
|
||||
YELLOWSTONE_X_TOKEN=<token> go run ./cmd/collect_yellowstone_rawtx_binary \
|
||||
-endpoint ams.rpc.orbitflare.com:10000 \
|
||||
-duration 5m \
|
||||
-output testdata/rawtx-binary/sample.prbs
|
||||
```
|
||||
|
||||
Measure parsed `Tx` binary size from a `getBlock` JSON payload:
|
||||
|
||||
```bash
|
||||
go run ./cmd/measure_tx_binary_block \
|
||||
-file /path/to/block.json \
|
||||
-slot 413539056 \
|
||||
-swaps-only
|
||||
```
|
||||
|
||||
Analyze raw transaction binary size distribution:
|
||||
|
||||
```bash
|
||||
go run ./cmd/analyze_rawtx_binary_size \
|
||||
-file testdata/rawtx-binary/sample.prbs
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
Run the full test suite:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Useful focused checks:
|
||||
|
||||
```bash
|
||||
go test -run 'TestTxBinary|TestRawTxBinary' .
|
||||
go test -run 'TestMetaoraPoolSwapEventFromLogsUsesMatchingInvocation|TestMetaoraPoolSwapInstructionOccurrenceIncludesInnerInstructions|TestAttachLogEventsToInstructions' .
|
||||
TX_HASH=<signature> go run ./cmd/rpc_parse
|
||||
```
|
||||
|
||||
For parser regressions, reproduce with the exact transaction signature first, then add a targeted unit test or fixture around the corrected parser path.
|
||||
Reference in New Issue
Block a user