This prints one line for each tx output: 1) the output's txid, 2) its index, 3) its age in units of txos since genesis before being spent, 9999999999 means a long time or currently unspent

txo-history.py

#!/usr/bin/env python3
import time, sys

from bitcoinrpc.authproxy import AuthServiceProxy
api = AuthServiceProxy("http://lmr:[email protected]:8332")

# infinite retry
def api_retry(*args):
    global api
    while True:
        try:
            return getattr(api, args[0])(*args[1:])
        except:
            # sleeping is probably unneeded, but give the server
            # a chance to recover, and also allows double-control-C
            time.sleep(4)
            api = AuthServiceProxy("http://lmr:[email protected]:8332")
            continue

# any utxos beyond this age (in txos seen) are considered this age
# (so we don't get an accurate age for these, they're just "old")
maxage = int(sys.argv[1]) if len(sys.argv) > 1 else 20_000_000

info = api_retry('getblockchaininfo')
tip_height = info['blocks']

# utxo_set[(txid,outindex)] = height
g_txo_count = 0
g_utxo_set = dict()

def cleanup():
    global g_txo_count, g_utxo_set
    to_delete = list()
    for outpoint, txo_count in g_utxo_set.items():
        if g_txo_count - txo_count > maxage:
            to_delete.append(outpoint)
            (txid, output_index) = outpoint
            print(format(txid, '064x'), output_index, 9_999_999_999, sep=',')

    for outpoint in to_delete:
        del g_utxo_set[outpoint]

for height in range(0, tip_height+1):
    # if we don't do this cleanup periodically, the system runs out of memory
    if height % 20_000 == 0:
        cleanup()

    blockhash = api_retry('getblockhash', height)
    b = api_retry('getblock', blockhash, 2)
    for tx in b['tx']:
        txid = int(tx['txid'], 16)
        for txi in tx['vin']:
            if 'coinbase' in txi: continue
            txi_txid = int(txi['txid'], 16)
            output_index = txi['vout']
            outpoint = (txi_txid, output_index)
            if outpoint in g_utxo_set:
                txo_count = g_utxo_set[outpoint]
                del g_utxo_set[outpoint]
                print(format(txi_txid, '064x'), output_index, g_txo_count - txo_count, sep=',')

        for txo in tx['vout']:
            g_utxo_set[txid, txo['n']] = g_txo_count
            g_txo_count += 1

cleanup()

Note, this ran the system out of memory after running for 7 days (mainnet), reaching block height 600k, output file 86G. The utxo_set dict becomes too large. So it's not practical to run this to completion.

This could be memory-optimized at least two ways:

• store the dict key as binary, not a string (reducing memory by almost 50%)
• use a shortened hash (txid) as the dict key (only the first 4 bytes)
  • when adding to the dict, see if there's a short-hash conflict, and if so, store it in an alternate dict with the full txid key
  • when looking up, first check the full-hash dict, and if not there, then check the short-hash dict (it must be there)
  • we can still print out the full txid, because it's in the outpoint

But if that isn't efficient enough, it probably needs to reimplement in a compiled language.

Several updates:

• track a UTXOs lifespan (time between creation and being spent) in units of transaction outputs (txos), not blocks
• prevent memory exhaustion by deleting UTXOs from the map when they become more than (rather arbitrarily) 20m txos old
• store the txid as an integer instead of a hex string; requires half as much memory

I think we want to construct the filter to contain long-life UTXOs, and the boundary between long-life and non-long-life will probably be less than 20m txos, I think, at least for the default cache size. If that's so, then we don't really need the exact lifetime when it's above 20m. When we do delete a long-running UTXO from the map, we print its age as 99999...9 (just so we know that's what happened).