Free-threaded (no-GIL): dumps() crashes when another thread mutates the dict being serialized (PyDict_Next iterator invalidation) #234

"""python-rapidjson 1.23 — iterator-invalidation in dumps() over a shared dict.

`rapidjson.cpp:2320` all_keys_are_string() and dumps_internal() (2553/2593) walk
the user dict with `PyDict_Next`, no copy and no lock:

    while (PyDict_Next(dict, &pos, &key, NULL))   // 2320
        if (!PyUnicode_Check(key)) return false;

`PyDict_Next` is documented unsafe against concurrent mutation: `pos` indexes the
dict's internal table, and a structural mutation (insert that triggers a resize,
or a delete) reallocates or reorders that table, so a `pos` captured before the
mutation now points into freed/relocated memory.

Under the GIL, `dumps()` never yields mid-walk, so it is safe. Under
`Py_GIL_DISABLED`, another thread structurally mutating the same dict while
`dumps()` walks it invalidates the iteration → crash or garbage read.

This is the iterator-invalidation class, not borrow-then-use: the corrupted thing
is the *walk's cursor over the container*, not a single borrowed item.

Run with PYTHON_GIL=0 (the module does not declare Py_MOD_GIL_NOT_USED, so import
re-enables the GIL otherwise).
"""
import os
import sys
import threading
import time

import rapidjson

N_DUMPERS = int(os.environ.get("N_DUMPERS", 6))
N_MUTATORS = int(os.environ.get("N_MUTATORS", 3))
SIZE = int(os.environ.get("SIZE", 64))
SECONDS = float(os.environ.get("SECONDS", 6))
MUTATE = os.environ.get("MUTATE", "1") == "1"
DECOY = os.environ.get("DECOY", "0") == "1"


def fresh(n):
    return {f"k{i}": i for i in range(n)}


def main():
    gil = getattr(sys, "_is_gil_enabled", lambda: True)()
    print(f"py={sys.version.split()[0]} gil={gil} "
          f"dumpers={N_DUMPERS} mutators={N_MUTATORS if MUTATE else 0} "
          f"decoy={DECOY}", flush=True)

    shared = fresh(SIZE)
    decoy = fresh(SIZE)
    stop = threading.Event()
    barrier = threading.Barrier(N_DUMPERS + (N_MUTATORS if MUTATE else 0) + 1)
    dumped = [0] * N_DUMPERS

    def dumper(t):
        barrier.wait()
        n = 0
        while not stop.is_set():
            try:
                rapidjson.dumps(shared)
            except Exception:
                pass          # value/type errors from racing data are irrelevant
            n += 1
        dumped[t] = n

    def mutator():
        barrier.wait()
        target = decoy if DECOY else shared
        i = 0
        while not stop.is_set():
            # STRUCTURAL mutation: grow (insert → resize) and shrink (delete),
            # which is what invalidates the PyDict_Next cursor.
            target[f"x{i}"] = i
            if i % 3 == 0 and len(target) > SIZE:
                try:
                    target.pop(next(iter(target)))
                except (KeyError, StopIteration, RuntimeError):
                    pass
            i += 1

    threads = [threading.Thread(target=dumper, args=(t,)) for t in range(N_DUMPERS)]
    if MUTATE:
        threads += [threading.Thread(target=mutator) for _ in range(N_MUTATORS)]
    for th in threads:
        th.start()
    barrier.wait()
    time.sleep(SECONDS)
    stop.set()
    for th in threads:
        th.join()
    print(f"clean: {sum(dumped)} dumps survived", flush=True)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())