pacing_rate had been computed on every ACK since the controller was written
and never read by anything: admission was decided purely by cwnd. But cwnd
bounds how much may be UNACKNOWLEDGED, not how fast it reaches the wire, so a
full window went out back-to-back. On a bottleneck with a deep buffer that
burst is absorbed rather than dropped, and it lands as standing queue — the
mechanism behind the multi-second RTT this protocol has been showing on
mobile. It is also why BBR could not simply be dropped in: BBR's whole model
is "send at the estimated bottleneck rate", which is meaningless without a
pacer underneath it.
Adds a token bucket to CongestionController, charged in on_send so every byte
that reaches the wire pays exactly once — retransmits included, since those
are precisely what must not bypass the limit and pile into a full queue.
Burst allowance is 10ms-at-rate rather than one packet. Pacing intervals here
are fractions of a millisecond, so strict per-packet release would need a
sub-millisecond timer per packet; sizing the burst to the loop's existing
~10ms wakeup lets the configured rate still be saturated. There is a floor of
4 MTU so a cold or collapsed rate estimate can never wedge sending entirely.
Wired into both directions. The client gates its proxy-event branch on it,
and the server's per-session backpressure snapshot reports zero headroom when
the bucket is empty. The download path matters most here — that is the one
carrying video — and it was also still clamped to the old 16384-packet
ceiling (~20 MB outstanding), now aligned with MAX_CWND_PACKETS.
Tested that the bucket denies once drained and refills over time; the second
is what keeps a stalled bucket from wedging sending permanently.
Matches a real user report after 2 weeks on this version: bandwidth is low,
the client reconnects every 10-20 minutes, sites randomly stop loading or
crawl, and ping visibly jitters. Root cause: the per-target-connection
reader task (handle_relay_message's Connect handler) read from the
upstream target as fast as it would send and forwarded every chunk
straight to send_relay_to_stream -> an immediate UDP datagram, with ZERO
awareness of the client-facing OSTP session's actual congestion window.
The client already gates its own uplink on cwnd (bridge.rs's proxy_ev
select arm); the server's download direction had no equivalent.
On a real (lossy/jittery mobile or Wi-Fi) client path, a fast target (a
CDN, say) gets blasted at the client far beyond what the path can sustain.
That's a self-inflicted loss burst: it wrecks the RTT/RTO estimate (the
"ping jitters" symptom), can push the session into a stall bad enough that
the client's 25s/180s keepalive stall-detection gives up and reconnects
(the "every 10-20 minutes" symptom), and produces exactly the "randomly
stops loading or crawls" experience while it's happening.
Fix: Dispatcher::snapshot_backpressure() computes each session's headroom
(clamped cwnd - in_flight, same clamp(16,16384) the client uses) from the
existing 10ms retransmit tick - no new polling loop. Published to a
lock-free-reader Arc<AtomicI64> per session in a shared map so relay reader
tasks (which don't have access to Dispatcher; it lives on the main loop
task) can check it without touching a lock/mutex on every read. Before
each read, a reader task waits (capped at 2s, so a wedged read can't stall
forever) while headroom is <= 0. New sessions default to a healthy 32
packets until their first snapshot lands (worst case 10ms), so this can't
stall the very first bytes of a fresh connection.
Scoped to the primary TCP CONNECT/proxy path (the one actual web browsing
uses); UDP relay/TURN is unaffected. No load test against real network
jitter was possible in this environment - the mechanism directly targets
the identified cause, but real-world confirmation is still needed.
- Karn's algorithm: drop_acked_frames no longer samples RTT from frames
that were retransmitted (last_sent is bumped on each retransmit, so an
ACK for the original transmission would measure a spuriously small RTT
and drag SRTT/RTO down). Added CongestionController::on_ack_no_rtt for
the case where every acked frame was ambiguous, so the window still
advances without polluting the RTT estimator. Refactored the shared
window-growth into grow_window.
- Frame decode: header+payload+pad length now uses checked_add. payload_len
is a u32 from the header and on 32-bit targets (MIPS/ARMv7 routers are
supported) the sum could wrap usize and slip past the truncation check.
- Replay cache: a full cache used to reject ALL new handshakes globally
until the next tick, letting one flooding key-holder deny service to
everyone. Now it reclaims expired entries and, if still full, evicts the
single oldest — new handshakes always get in. Fixed the mislabelled
"100000" log (cap is 50000) and named it REPLAY_CACHE_MAX.
Two classes of issue:
- Hot-path/attacker-triggerable events logged at info/error with internal
detail: a per-handshake info! byte dump (raw_vec[0..6]) and a per-packet
error! on session-id mismatch that dumped expected/got session ids.
Both are log-flood + info-leak surfaces; downgraded to debug and
stripped of the sensitive detail. Close/Resume frame handling likewise
moved from info to debug.
- The access key (a shared secret) was written to logs verbatim in three
places (session drop, key-created UI event, API create-user) and as an
8-char prefix in one. Added key_fp() — a short SHA-256 fingerprint — and
routed all key logging through it so operators can still correlate
events without the secret ever hitting the log.
Every datagram from an unrecognized source ran the full key-trial loop:
for each registered access key, an HKDF (derive_all_secrets) plus two
HMACs (junk markers) plus a Noise read. A garbage flood from spoofed
sources could therefore force unbounded O(N_keys) crypto per packet — a
CPU-amplification DoS with no throttle (the existing token bucket only
guarded the roaming path, not this one).
Two mitigations:
- Memoize the per-key derived secrets (pure function of key+version) and
the per-window junk markers, so the trial loop is now cheap comparisons
plus one Noise read per key instead of HKDF+2*HMAC per key per packet.
Also speeds up every legitimate new connection. Caches are pruned in
on_tick when keys are deleted.
- Gate the trial path behind a global token bucket (TRIAL_RATE=100/s,
same burst). The established-session fast path and roaming are not
gated, so live sessions are unaffected; only unknown-datagram trials
are bounded. Over-budget datagrams are dropped silently.
The junk marker was a per-key CONSTANT sent in plaintext at a fixed offset in
junk frames. Junk is meant to look like random noise (zapret-style), but a
constant prefix is a recognizable per-user structure: an on-path observer
watching one user sees the same 4 bytes on every junk packet, i.e. an OSTP
fingerprint. (The earlier fix only removed the GLOBAL constant.)
Now the marker rotates every 60s window: junk_marker = HKDF(key, ver, 0x04 ||
window). To an observer the prefix changes each window (no fixed signature),
and a captured marker is only valid for ~1 window — the "bit of protection"
against a leaked marker. Only a key holder can compute it, so an outsider still
can't forge a silently-dropped junk packet (and silent-drop is cheaper than
normal processing anyway, so junk spam was never a DoS lever to begin with).
- core: derive_junk_marker(key, window) + current_junk_window() (60s window),
same version-gated HKDF scheme; junk_marker dropped from DerivedSecrets.
- client: stamps junk with the current window's marker.
- server: checks current AND previous window per key (absorbs ~1 window of
clock skew) before falling through to unauthorized-probe handling.
- Not a wire break: only junk framing changes; real handshake/data untouched.
During mixed rollout, unmatched junk merely logs as a probe (cosmetic).
security / protocol:
- Derive a PER-KEY junk marker (obfuscation.rs, info byte 0x04) instead of the
global constant [0x88,0x1A,0x93,0x5D]. A fixed marker was a universal DPI
signature identifying ALL OSTP users at once — exactly what the HKDF version
gate avoids for the handshake. Server drops junk via a new DispatchOutcome::Junk
inside the existing key-trial loop (secrets already derived → zero extra cost);
client stamps its own key's marker.
- §E: configurable junk/fragmentation params (junk_pc / junk_ps / frag_chunk / frag_sleep).
GUI (desktop):
- Light theme + toggle, GUI version footer in Settings.
- Fix mouse-wheel scroll on Settings (flex child needed min-height: 0).
- Drop the false "process exclusions unsupported in TUN mode" warning — they DO
work (native_handler maps port->process via GetExtendedTcpTable).
release / infra:
- build.ps1: add -PreRelease (tag CURRENT version as v<ver>-beta.N, no bump, no
master commit); guard the panel build when ostp-control ships no source; bump
the real ostp-gui/package.json instead of the nonexistent ostp-control one.
- release.yml: mark hyphenated tags as GitHub pre-releases; don't hard-fail the
web-panel step when there is no source (use committed dist/).
- Versions aligned to 0.4.1; README license badge BSL 1.1 -> AGPL v3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Applied Kerckhoffs's principle: the protocol's security and obfuscation
now depend SOLELY on the access key. An adversary who reverse-engineers
the binary cannot build a DPI filter without knowing the key.
Changes:
- Replaced hardcoded salt string ('-ostp-psk-salt') with HKDF-SHA256.
The salt is now derived from the key hash itself — no protocol-specific
strings remain in the binary.
- Unified all secret derivation into derive_all_secrets() which produces
PSK, obfuscation key, and handshake padding range from a single HKDF
invocation.
- Handshake padding range is now key-derived: different access keys
produce different size distributions (min: 16-79, max: +48..+175).
A universal size-based filter is impossible without the key.
- HKDF-SHA256 (RFC 5869) implemented inline using existing hmac+sha2
dependencies — no new crate required.
What remains identifiable in the binary:
- 'Noise_NNpsk0_25519_ChaChaPoly_BLAKE2s' — standard Noise pattern
string, shared with many other projects, NOT OSTP-specific.
- Generic HMAC/SHA-256/ChaCha20-Poly1305 code — standard crypto
primitives used by millions of applications.
Previously handshake obfuscation used a FIXED mask derived from
HMAC(obf_key, u64::MAX). This meant bytes [4..6] (noise_len XOR
fixed_mask) produced the SAME 2-byte value on every handshake from
the same access key — a correlation fingerprint for DPI.
Now BOTH data and handshake packets use the same payload-sampling
approach:
mask = HMAC-SHA256(obf_key, payload_sample[0..32])
For data packets: payload_sample = AEAD ciphertext (random per packet)
For handshake packets: payload_sample = Noise ephemeral key (random per connection)
Result: every single byte on the wire is cryptographically independent
across packets. No fixed patterns, no correlation between connections.
Wire analysis after this change:
- Packet sizes: random (84-182 for handshake, variable for data)
- All header bytes: unique per packet (XOR with unique HMAC mask)
- Payload bytes: AEAD ciphertext / Noise handshake (indistinguishable from random)
- No protocol signatures, no version fields, no magic bytes visible on wire
The previous commit added random padding after Noise handshake payloads
but the receiver passed the entire raw buffer (including padding) to
snow::read_handshake(), which cannot handle trailing bytes.
New wire format:
[session_id:4][noise_len:2][noise_payload:N][random_padding:32-128]
Changes:
- wrap_datagram_handshake: puts noise_len (u16 BE) at bytes [4..6]
before the Noise payload, followed by 32-128 random padding bytes
- handle_inbound: reads noise_len from [4..6], passes only
raw_vec[6..6+noise_len] to snow, ignoring trailing padding
- obfuscation: handshake mask extended from 4 to 6 bytes to also
cover the noise_len field (prevents DPI from seeing constant u16)
- dispatcher: key-trial loop updated to deobfuscate 6-byte header
Both client and server now produce/consume the same padded format.
Critical fixes (6):
- protocol.rs: in_flight_count() now counts only retransmittable Data frames,
not Ack/Nack control frames — eliminates false backpressure under load
- protocol.rs: NACK is now rate-limited to once per 30ms — prevents
retransmission storm during normal UDP jitter
- protocol.rs: zombie frames exceeding max_retries+4 are evicted each tick —
prevents unbounded memory growth and stale retransmits
- protocol.rs: Closing state now processes final in-flight packets instead
of silently dropping them — prevents data loss at session teardown
- server/lib.rs: stream_tx changed from bounded(10000) to unbounded_channel —
prevents TCP-reader collapse during Speedtest with 50+ streams
- bridge.rs: liveness timeout raised from 30s to 60s — prevents false
reconnect during heavy Speedtest load
Medium fixes (8):
- protocol.rs: ACK range truncation preserves cumulative range (index 0)
- bridge.rs: Ping now uses send_datagram() for correct TURN wrapping
- dispatcher.rs: replay_cache hard-capped at 100k entries (DoS protection)
- dispatcher.rs: old addr cleaned from addr_to_session on roaming
- server/lib.rs: TCP connect_target() now has 10s timeout
- config.rs: TURN section parsed during hot-reload
- proxy.rs: HTTP header parsing uses 512-byte chunks instead of 1-byte reads
- proxy.rs: stream_id wrap-around skips active IDs to prevent collision
- runner.rs: is_essential_log matches actual log strings from bridge.rs
Other:
- kex.rs: clearly marked as dead PQ stub (not used by protocol)
- README.md + README.ru.md: complete rewrite with architecture diagram
- docs/en/specification.md: updated ARQ section with all new semantics
ospab
ospab